A multi-modal entity linking method based on double encoders and hybrid expert mechanism

Abstract

A multimodal entity linking method based on dual encoders and a hybrid expert mechanism is proposed. This invention relates to multimodal entity linking technology at the intersection of natural language processing and computer vision. Addressing the problems of low inference efficiency, insufficient cross-modal interaction, and shallow modal fusion in existing methods, this invention proposes a multimodal entity linking method based on dual encoders and a hybrid expert mechanism. A dual-tower architecture is used to independently encode mentions and entities. Entity embeddings can be pre-computed offline and indexed, and linking is completed during inference through fast vector retrieval. A hybrid expert mechanism is introduced to achieve adaptive feature transformation of samples, and a gating network dynamically selects expert combinations. Bidirectional cross-modal attention is used to establish fine-grained alignment at the word-image block granularity. A channel attention mechanism dynamically balances the contributions of textual and visual modalities. The model is jointly optimized by multiple constraints, including load balancing loss. This invention achieves efficient retrieval while maintaining deep inference capabilities, simplifies inference time complexity, and is suitable for scenarios such as knowledge graph construction and intelligent question answering systems.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of natural language processing and computer vision, specifically to a multimodal entity linking method based on dual encoders and a hybrid expert mechanism. This method is applicable to scenarios such as knowledge graph construction, intelligent question answering systems, and multimedia content understanding, achieving efficient entity linking tasks by fusing textual semantic information and visual perception information. Background Technology

[0002] With the explosive growth of multimedia content on the internet, entity linking technology has expanded from traditional pure text scenarios to mixed text and image scenarios, requiring the simultaneous processing of textual descriptions and visual image information. Multimodal entity linking aims to map mentions containing both text and images to standard entities in a knowledge base, and is a core task in building multimodal knowledge graphs and enhancing information retrieval systems.

[0003] Existing technologies suffer from the following problems: First, while end-to-end joint encoding methods can fully capture cross-modal semantic correspondences, they must compute interactions with all candidate entities for each mention, leading to computational bottlenecks on large-scale entity databases. The inference time complexity increases linearly with the entity size, making it difficult to apply to scenarios with millions of entities. Second, retrieval enhancement methods improve efficiency through a two-stage architecture, but performance heavily depends on the quality of the candidate set. Errors in the recall stage accumulate later, and the retrieval stage typically uses shallow semantic similarity, making it difficult to fully utilize multimodal information. Third, the independent encoding characteristics of existing dual-tower architectures make it difficult for models to capture complex semantic correspondences and lack explicit interactions between mentions and entities, resulting in insufficient capabilities in fine-grained semantic matching and cross-modal inference. Fourth, existing methods typically employ simple feature concatenation or shallow attention mechanisms during multimodal fusion, failing to establish deep semantic mapping relationships between text and vision, leading to insufficient fusion of heterogeneous modal information. Fifth, existing training strategies easily cause models to over-rely on one modality while neglecting others, resulting in severe performance degradation and a lack of robustness in scenarios with insufficient single-modal information or uneven modal information quality. Summary of the Invention

[0004] 3.1 Purpose of the Invention

[0005] This invention aims to address the problems of low inference efficiency, insufficient cross-modal interaction, shallow modal fusion, and unbalanced modal dependencies in existing multimodal entity linking technologies, and provides a multimodal entity linking method based on dual encoders and a hybrid expert mechanism.

[0006] 3.2 Overview of Technical Solution

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a multimodal entity linking method based on dual encoders and a hybrid expert mechanism, comprising the following steps:

[0008] S1: Receive the mention to be linked, its text context, and associated images; obtain a set of candidate entities from the knowledge base, where each entity contains a text description and a set of images;

[0009] S2: It adopts a dual encoder architecture to independently encode mentions and entities. The mention encoding function encodes the text and image of the mention into an embedding vector, and the entity encoding function encodes the text and image of the entity into an embedding vector. This allows entity embeddings to be pre-computed and stored offline, and entity links are simplified to nearest neighbor search during inference.

[0010] S3: During the encoding process, the CLIP pre-trained model is used as the basic encoder. The CLIP text encoder generates word-level sequences for mentioned text, the CLIP visual encoder generates image block-level sequences for mentioned images, and the same encoding method is used to obtain corresponding sequences for entities.

[0011] S4: Construct a multimodal fusion tower to project text sequences and visual sequences to a unified dimension using a learnable projection matrix;

[0012] S5: A hybrid expert mechanism is used to perform sample adaptive processing on the projected sequence. K expert networks are defined. For each word, an expert activation weight is generated through a gating network. A top-p sparse activation strategy is used to select p experts. The output is a normalized weighted combination of the selected experts. The same hybrid expert mechanism is applied to the visual sequence.

[0013] S6: Fine-grained alignment between text and vision is established through bidirectional cross-modal attention. A multi-head attention mechanism is adopted to enable text lexical queries to visual blocks to generate text-to-image attention output, while visual blocks query text lexical queries to generate image-to-text attention output. Modality-specific global representations are generated through mean pooling.

[0014] S7: Employs a channel attention mechanism to dynamically fuse text and visual representations. The spliced ​​representation is used to generate modal weights through an attention network. Element-level weighting is applied to the fused representation. The final embedding is generated through the fusion layer and then L2 normalized.

[0015] S8: Train a dual encoder model using a contrastive learning objective function, compute the cosine similarity matrix in a batch containing B mention-entity pairs, define the InfoNCE loss from mention to entity and the symmetric loss from entity to mention, and simultaneously apply load balancing loss, expert diversity loss and channel attention regularization constraints to jointly optimize all learnable parameters of the encoder and fusion tower.

[0016] S9: During the inference phase, the embedding vector of the reference to be linked is calculated, and the nearest neighbor search is performed in the pre-computed entity embedding library by cosine similarity. The entity with the highest similarity is output as the link result.

[0017] 3.3 Optional Technical Features

[0018] Optionally, the data receiving in S1 specifically includes: receiving the mention m to be linked and extracting the text context of the mention. The context is the complete sentence or paragraph containing the reference; extract the associated image of the reference. The image is a scene image or context image associated with the mention; from the knowledge base. Obtain the candidate entity set, each entity Includes text description and image collection The text description includes the entity name, a list of attributes, and a text description, while the image set contains 1 to 10 images related to the entity.

[0019] Optionally, the dual encoder architecture in S2 adopts a parameter sharing strategy or independent parameterization configuration. In parameter sharing mode, the encoding function is mentioned. and entity encoding function Using the same CLIP base encoder, projection layer, hybrid expert network, bidirectional attention module, and channel attention fusion layer parameters reduces model complexity and facilitates knowledge transfer; in independent parameterization mode, the two encoder towers use their own independent parameters to adapt to different input distribution characteristics.

[0020] Optionally, the CLIP text encoder in S3 processes the text input by: encoding the text input using a Transformer architecture, concatenating entity names and entity attributes using a special delimiter [SEP] to construct a unified input sequence, and extracting the word-level representations of all hidden layers as sequence features after encoder processing. Sequence length The original feature dimension is determined by the number of lexical units after text segmentation. The same encoding process is used for the mentioned text to extract word-level sequence representations.

[0021] Optionally, the processing method of the CLIP visual encoder in S3 includes: dividing the image input into visual blocks of fixed size and linearly projecting them; after processing by the Vision Transformer, extracting all block-level representations as sequence features. Sequence length Determined by the image patching strategy, the original feature dimension is For entity image sets During the training phase, an image is randomly selected for encoding to enhance the model's generalization ability. During the inference phase, the first image is selected to ensure the reproducibility of the results. Alternatively, a multi-image aggregation strategy can be used to average or weightedly fuse the features of multiple images.

[0022] Optionally, the projection operation in S4 is performed using a learnable projection matrix. and Project the text sequence and the visual sequence onto a unified dimension. The projected sequence is obtained. and Unified Dimension Typically set to 512 or 768, this allows heterogeneous textual and visual features to be used for subsequent interactive computations within the same representation space.

[0023] Optionally, the hybrid expert mechanism in S5 defines K expert networks, each consisting of two layers of feedforward neural networks. ,in and Let be the parameters of the k-th expert. For the hidden layer dimension; the gated network generates expert activation weights through linear transformation and a temperature-scaled softmax function. ,in For gating parameters, The temperature parameter is learnable; the top-p sparse activation strategy selects the p experts with the highest gating scores for activation, typically p is set to 2 and K is set to 4 to 8.

[0024] Optionally, S5 includes multiple diversity enhancement strategies: employing an expert dropout mechanism during training, with probability... Randomly disabled One expert set its gating score to... This forces the model to learn using all experts, with dropout probabilities typically set to 0.1 to 0.3; in the previous In the warm-up phase of each training step, ignoring the gating score, p experts are randomly selected for each word and assigned equal weights. To ensure all experts receive sufficient gradient updates during early training, the warm-up steps are typically set to 1000 to 5000 steps; after the warm-up phase, Gumbel noise is added to the gated logits. Enhanced exploration, among which The noise intensity gradually decays to zero in an exponential or linear manner during training.

[0025] Optionally, the bidirectional cross-modal attention in S6 employs a multi-head attention mechanism. Each attention head is defined as The linear transformation of query, key, and value is computed in parallel using H attention heads, with the number of attention heads H typically set to 8 or 16; text-to-image attention treats text terms as queries and visual blocks as keys and values, and computes... Image-to-text attention treats visual blocks as queries and text terms as keys and values, and computes... Residual connections preserve the original modal information while injecting cross-modal context.

[0026] Optionally, in step S6, a modality-specific global representation is generated through average pooling, and average pooling is performed along the sequence length dimension on the text-to-image attention output to calculate... The dimension is obtained as The text global representation; perform the same operation on the image-to-text attention output, and calculate The dimension is obtained as The two global representations encode the text semantic enhancement visual information and the visual perception enhancement text information, respectively, providing a foundation for subsequent channel attention fusion.

[0027] Optionally, the channel attention mechanism in S7 adaptively learns the relative importance of text and visual modalities through a multi-layer perceptual network. The attention network contains three fully connected layers. , and splicing representation The dimension is The nonlinear transformation is performed sequentially through three fully connected layers and the GELU activation function to calculate... This yields normalized weights for both text and visual data; temperature parameters can be learned. To control the sharpness of the weight distribution, the initial value is usually set to 1.0 to 2.0.

[0028] Optionally, in S7, element-level weighting is applied to the fused representation to calculate... and The dot product is an element-wise multiplication operation; the weighted text and visual representation are concatenated and the final embedding is generated through a fusion layer. ,in and The parameters are for the fusion layer; finally, the embedding vectors are subjected to L2 normalization. This ensures that the magnitude of all embedded vectors is 1, which facilitates subsequent cosine similarity calculation.

[0029] Optionally, channel attention regularization loss is applied in S7. To prevent the model from completely ignoring a certain mode, where The maximum allowable weight difference threshold is typically set between 0.3 and 0.5. The loss weight coefficient is typically set to 0.1 to 0.5; this regularization constraint encourages moderate modality balance, penalizes when the weight gap exceeds a threshold, and allows sample-specific adaptive recalibration, avoiding performance degradation caused by forced equal allocation.

[0030] Optionally, the contrastive learning loss function in S8 computes a cosine similarity matrix in a batch containing B mention-entity pairs, and computes the inner product of the normalized embedding vectors for the i-th mention and j-th entity in the batch. ,in The temperature parameter is typically set between 0.05 and 0.1; smaller temperature values ​​result in a sharper similarity distribution. The definition mentions the InfoNCE loss for entities. This loss prompts positive samples to... The similarity is higher than that of all negative sample pairs; define the loss for symmetric entities to mentions. Through weighted combination Comprising the main search targets, weight coefficients and It is typically set to 0.5 to 1.0.

[0031] Optionally, the load balancing loss in S8 includes several components: Switch Transformer loss. ,in The frequency of expert k's use in the batch. Its importance is defined as the average gating score of all token features processed by expert k; variance penalty Directly constrain the uniformity of the distribution; use insufficient penalty. and overuse of punishment To prevent extreme imbalances; entropy regularization This encourages uncertainty in gating distribution; the combined load balancing loss is... The weight coefficients for each item were determined through optimization using the validation set.

[0032] Optionally, the expert diversity loss in S8 Minimize the output signatures of different experts Cosine similarity between experts facilitates expert learning of orthogonal feature transformations; expert signature is defined as the average output vector of that expert on a batch. ,in The number of activations for expert k; diversity loss weights. It is typically set to 0.05 to 0.2.

[0033] Optionally, the final training objective in S8 The algorithm jointly optimizes retrieval performance, representation diversity, and balance. It calculates the gradient of the objective function with respect to all model parameters using backpropagation, and synchronously updates the CLIP encoder parameters of the encoding layer and the parameters of all projection layers, hybrid expert networks, attention mechanisms, and normalization layers in the multimodal fusion tower using the Adam or AdamW optimizer. The learning rate is set between 1e-5 and 5e-4. Pre-trained CLIP parameters are fine-tuned using a smaller learning rate (e.g., 1e-6 to 1e-5), while newly added fusion tower parameters are fine-tuned using a larger learning rate (e.g., 1e-4 to 5e-4). Gradient clipping is applied to limit the gradient norm to within 1.0 to 5.0.

[0034] Optionally, the reasoning stage in S9 includes two steps: offline pre-computation of entity embeddings and online nearest neighbor search. In the offline stage, all entities in the knowledge base are pre-computed. Embedding vectors are calculated sequentially using entity encoding functions. All entity embeddings can be pre-computed and persistently stored, avoiding online coding overhead; during the online phase, only one forward propagation is needed to compute the embedding vector for each new mention. The time complexity is Nearest neighbor search is performed in a pre-computed entity embedding library using cosine similarity. ,in The inner product of normalized vectors; an approximate nearest neighbor search algorithm such as FAISS is used for efficient retrieval in the entity embedding library, compared to joint coding methods. The complexity is significantly accelerated.

[0035] Optionally, the inference stage supports dynamic updates of the entity database. When a new entity is added to the knowledge base, only the embedding vector of the new entity needs to be calculated and added to the index, without retraining the model. It supports batch query optimization, allowing multiple mentions to have their embedding vectors calculated in parallel, leveraging the parallel computing capabilities of GPUs to improve throughput. It supports multiple candidate outputs, returning the top-K candidate entities with the highest similarity and their corresponding confidence scores for further processing by downstream applications. K is typically set between 3 and 10. It supports threshold filtering, marking link results with similarity below a preset threshold as "uncertain," suggesting manual intervention or further verification.

[0036] 3.4 Beneficial Effects

[0037] This invention employs a dual-encoder architecture to achieve independent encoding of mentions and entities. This allows entity embeddings to be pre-computed offline and indexed, simplifying entity linking during inference into an efficient vector retrieval operation. Compared to end-to-end joint encoding methods that require computation and interaction with all candidate entities for each mention, the inference time complexity of this invention is significantly reduced. Reduced to approximate Significant efficiency improvements were achieved on a database of millions of entities. Meanwhile, the dual-tower architecture supports dynamic updates to the entity database; adding a new entity only requires calculating its embedding and adding it to the index, eliminating the need to retrain the model and significantly reducing system maintenance costs.

[0038] This invention achieves sample-adaptive feature transformation by introducing a hybrid expert mechanism during the encoding stage. Different mention-entity pairs require different inference modes. The hybrid expert mechanism dynamically selects the most suitable expert combination for each sample through a gating network, enabling the model to adaptively adjust its processing strategy based on sample characteristics. The introduced multiple diversity enhancement strategies, including expert dropout, forced exploration during the warm-up period, and Gumbel noise enhancement, effectively solve the expert collapse problem in standard hybrid expert mechanisms, ensuring that all experts can learn fully and play their roles. Experimental analysis shows that different experts exhibit clear functional division patterns in text and visual modalities. Visual experts adopt a cooperative mode with balanced activation, while text experts adopt a functional differentiation mode with selective activation. This heterogeneous collaborative architecture allows the model to flexibly adapt to different entity types and scene complexity.

[0039] This invention establishes fine-grained alignment between text and visual representations at the token-patch level through a bidirectional cross-modal attention mechanism. Text-to-image attention enables text units to focus on relevant image regions, while image-to-text attention enables visual blocks to focus on relevant text units. This bidirectional mechanism ensures the symmetry and comprehensiveness of cross-modal feature matching. Compared to simple feature concatenation or global-level modal fusion, the fine-grained attention mechanism can establish precise correspondences in local semantics, such as accurately matching "red sports car" in text with a red car region in an image. The multi-head attention mechanism enables text units and visual blocks to establish correspondences in different representation subspaces, capturing diverse semantic association patterns.

[0040] This invention dynamically learns the relative importance of text and visual modalities through a channel attention mechanism, adaptively assigning weights to the two modalities. Experimental analysis shows that the model learns a weight distribution of text:visual = 1.82:1, accurately capturing the modal dependencies in entity linking tasks. This asymmetric design not only aligns with the semantic-dominated nature of the task but also achieves efficient integration of multimodal information through adaptive difficulty adjustment and modal collaboration mechanisms. The introduced channel attention regularization constraint prevents the model from completely ignoring any modality, maintaining good performance even in scenarios with insufficient or unevenly distributed single-modal information, significantly improving the model's robustness.

[0041] This invention systematically promotes balanced model learning through a combination of multiple regularization constraints. Load balancing loss, through a combination of Switch Transformer loss, variance penalty, underutilization and overutilization penalties, and entropy regularization, ensures that all experts are fully utilized. Expert diversity loss, by minimizing the cosine similarity between the output signatures of different experts, promotes the learning of orthogonal feature transformations by experts, avoiding functional redundancy. Channel attention regularization constraint prevents modality degradation, ensuring that both textual and visual information are effectively utilized. These regularization constraints work synergistically during training, optimizing retrieval performance while ensuring the diversity and balance of representations, enabling the model to maintain stable performance across different data distributions and task scenarios. Attached Figure Description

[0042] The accompanying drawings, which form part of this invention, are used to explain the technical solutions of this invention and do not constitute an undue limitation of this invention. For those skilled in the art, other drawings can be obtained from these drawings without any inventive effort.

[0043] Appendix Figure 1 This is a flowchart illustrating the overall architecture of the present invention, showing the complete processing flow from data reception, CLIP encoding, multimodal fusion towers (including projection layers, hybrid expert layers, bidirectional cross-modal attention layers, and channel attention fusion layers) to final embedding generation. The diagram includes two independent encoding towers in the dual-encoder architecture, the contrastive learning training module, and the nearest neighbor search process during the inference phase. Detailed Implementation

[0044] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0045] S1: Receive the mentions to be linked and their multimodal information, and obtain the knowledge base entity set, specifically:

[0046] S11: Receiving Linked Mentions. Receives the linked mention m, which is an explicitly stated entity reference in a text document, social media post, or multimedia content. This reference may be the entity's standard name, alias, abbreviation, or a variant of the spelling. Extracts the textual context of the mention. This context refers to a complete sentence or paragraph containing the reference, ranging in length from 10 to 100 words, providing contextual information about the reference. Extract the associated image of the reference. The image is a scene or context image associated with the mention, in JPEG or PNG format, with a resolution of at least 224×224 pixels. For complex samples containing multiple mentions, multiple mentions and their corresponding text contexts and associated images can be received simultaneously.

[0047] S12: Knowledge base entity information retrieval. From the knowledge base... The candidate entity set is obtained from the database. The knowledge base uses a structured storage method, and each entity has a unique identifier. For each entity... Extract its text description This description includes the entity name, attribute list, and text description. The entity name is a string and is usually the entity's standard or common name. The attribute list is a collection of key-value pairs containing structured information such as the entity's type, category, and key features. For example, the attributes of a person entity might include information such as "occupation: scientist" and "nationality: United States." The text description is an unstructured natural language description, usually derived from description fields in encyclopedias or knowledge bases, and is between 50 and 500 words in length. Extracting the entity's image set... The collection contains 1 to 10 images related to the entity, in JPEG or PNG format, with a resolution of at least 224×224 pixels.

[0048] S13: Candidate Entity Set Construction. Different candidate entity selection strategies can be adopted based on task requirements and computing resources. For offline scenarios or situations with sufficient computing resources, all entities in the knowledge base can be used as the candidate set to achieve full-base retrieval. For online scenarios or large-scale knowledge bases, pre-screening strategies can be used to narrow down the candidate range: string matching based on entity names, including exact matching, fuzzy matching, and edit distance matching; querying based on an entity alias database to retrieve entities corresponding to all aliases similar to the mentioned name; filtering based on prior probability to select entities with high co-occurrence frequency with the mentioned name; and using a lightweight vector retrieval method with a simplified encoder to quickly filter the top-K similar entities. The number of candidate entities is set according to the specific application scenario, typically between 10 and 1000.

[0049] S2: Employs a dual-encoder architecture to independently encode mentions and entities, specifically:

[0050] S21: Dual Encoder Architecture Design and Initialization. A dual-tower architecture consisting of a mention encoder and a entity encoder is constructed. The two encoders are structurally symmetrical, each containing a CLIP base coding layer and a multimodal fusion tower. In parameter-sharing mode, the mention encoding function... and entity encoding function Using the same set of parameters, including CLIP text encoder parameters, CLIP visual encoder parameters, projection layer parameters, hybrid expert network parameters, bidirectional attention module parameters, and channel attention fusion layer parameters, parameter sharing reduces model complexity and facilitates knowledge transfer between mentions and entities. The total number of parameters is approximately 1.2 to 1.5 times that of the original CLIP model. In independent parameterization mode, the two encoders use their own independent set of parameters, which is suitable for scenarios where mentions and entities have significantly different input distribution characteristics, but increases the number of model parameters and training difficulty.

[0051] S22: Encoder input / output interface definition. This refers to the encoder receiving the mentioned text context. and associated images As input, the normalized embedding vector is output after multi-layer processing. Where d is the embedding dimension, typically set to 512 or 768. The entity encoder receives the text description of the entity. and image collection As input, the output is a normalized embedding vector. The output dimensions of the two encoders must be consistent for subsequent similarity calculations. The embedding vectors are L2 normalized so that the magnitude of all vectors is 1, facilitating matching using cosine similarity.

[0052] S23: The inference advantages of the dual-tower architecture. During the inference phase, the entity encoder can pre-compute the embedding vectors of all entities in the knowledge base offline. And establish efficient index structures, such as FAISS's IVF or HNSW indexes. For a new mention to be linked, its embedding vector only needs to be computed through a single forward propagation performed by the mention encoder. Then, a fast nearest neighbor search is performed in the pre-built entity embedding index. This decoupled design reduces the inference time complexity from that of joint encoding methods. Reduce to or approximate It achieves speedups of hundreds to thousands of times on large-scale entity databases. Furthermore, when new or updated entities are added to the knowledge base, only the embedding vectors of the affected entities need to be recalculated and the indexes updated; the entire model does not need to be retrained, significantly reducing system maintenance costs.

[0053] S3: The CLIP pre-trained model is used as the base encoder, specifically:

[0054] S31: CLIP Model Selection and Loading. A pre-trained CLIP model is selected as the base encoder for both text and vision. CLIP learns aligned text-image representations through contrastive learning pre-training on large-scale network data. Common CLIP model variants include CLIP-ViT-Base-Patch32, CLIP-ViT-Large-Patch14, and CLIP-ResNet50, selected based on computational resources and performance requirements. The pre-trained weights are loaded, and the parameters of the text and vision encoders are initialized. The text encoder is based on a Transformer architecture, typically containing 12 or 24 layers of self-attention modules, with a maximum sequence length of 77 tokens and an output dimension of... The value is 512 or 768. The visual encoder, based on the VisionTransformer or ResNet architecture, divides the image into fixed-size patches, processes them through multiple layers of self-attention or convolutional networks, and outputs the dimension. It is 512 or 768.

[0055] S32: Sequence generation by the text encoder. For mentioned text... First, word segmentation is performed. The CLIP-compatible tokenizer converts the text into a token sequence, adding a start marker [BOS] and an end marker [EOS]. Parts shorter than 77 characters are padded with padding markers [PAD], while parts longer than 77 characters are truncated. The token sequence is then converted into a token ID sequence and mapped to initial word embedding vectors through an embedding layer. Positional encoding is added, using either learnable positional embeddings or fixed sine / cosine positional encoding. The word embedding sequence is input into a Transformer encoder, processed through multiple layers of self-attention modules and a feedforward neural network. Each layer includes multi-head self-attention, layer normalization, and residual connections. The word-level representations of all hidden layers are extracted as sequence features. ,in This represents the actual number of tokens. This is the output dimension of the CLIP text encoder. This sequence feature preserves the semantic information of each token position in the text, providing a foundation for subsequent fine-grained cross-modal alignment.

[0056] S33: Encoding processing of entity text. This involves encoding entity text. The same encoding process as for the mentioned text is adopted. Entity names and entity attributes are concatenated using a special delimiter [SEP] to construct a unified input sequence. Specifically, the concatenation method is as follows: first, a start marker [BOS] is added; then, the entity name text is concatenated; next, the [SEP] marker is added; finally, a list of attribute strings is concatenated, with attributes separated by commas or semicolons; and finally, an end marker [EOS] is added to the end of the sequence. This is processed by the CLIP text encoder to generate word-level sequence features of the entities. ,in This represents the actual number of tokens in the entity text. The concatenation strategy allows the model to utilize both the entity's standard name and structured attribute information, enhancing the text's semantic expressiveness.

[0057] S34: Sequence generation by the visual encoder. (Referring to the mentioned image) First, preprocessing is performed. The Lanczos interpolation algorithm is used to scale the image to a preset width and height. The Lanczos algorithm uses the sinc function for interpolation, which can maintain high image quality and edge sharpness when the image size is changed. The scaled image is then normalized by subtracting the mean of the ImageNet dataset [0.485, 0.456, 0.406] from each channel and dividing by the standard deviation [0.229, 0.224, 0.225]. The preprocessed image is then input into the CLIP visual encoder. The encoder first divides the image into fixed-size square patches, typically 16×16 or 32×32 pixels. Therefore, a 224×224 resolution image is divided into 196 or 49 patches. Each patch is flattened into a one-dimensional vector, and the patch vector is mapped to a linear projection layer. Dimensionality. A learnable [CLS] classification label is added to the beginning of all patch feature sequences. Positional encoding is added to each patch feature, using learnable parameters or fixed 2D sine / cosine encoding to preserve spatial location information. The positionally encoded patch feature sequences are then input into a multi-layer Transformer encoder or convolutional network to generate the output feature sequence. ,in Increment the patch count by 1 (including the [CLS] flag). This represents the output dimension of the visual encoder.

[0058] S35: Encoding processing of entity images. For the entity image set... During the training phase, an image is randomly selected for encoding. This random selection strategy enhances the model's adaptability to different image perspectives and avoids overfitting to specific images. During the inference phase, the first image is consistently selected to ensure reproducibility and stability of the results. The same preprocessing and encoding process as the mentioned image is used, employing Lanczos interpolation scaling, normalization, patch partitioning, linear projection, positional encoding, and Transformer encoding to generate visual sequence features of entities. For entities with multiple images, possible implementations include: randomly selecting different images for each batch during training to enhance diversity; encoding multiple images separately and then aggregating features, such as average pooling or max pooling, during inference to generate more robust entity representations; and introducing an image attention mechanism to dynamically select the most relevant entity image for matching based on the features of the mentioned images.

[0059] S36: Normalization and Caching of Encoded Features. Dimensional checks and format normalization are performed on the generated text sequence features and visual sequence features to ensure that the sequence length and feature dimensions meet expectations. For variable-length sequences, the actual length information is recorded for masking operations in subsequent processing. Encoded features are stored in an in-memory cache, and a mapping index between mention IDs, entity IDs, and feature tensors is established. For entity features in the knowledge base, offline pre-computation can be performed and persistently stored on disk or in a database to reduce online computation overhead. For mention features, real-time encoding is performed upon receiving task samples. The caching strategy uses the LRU (Least Recently Used) algorithm, prioritizing the eviction of the least recently used feature data when cache space is insufficient. For scenarios with limited GPU memory, a block processing strategy can be adopted, dividing large batches of data into smaller batches for sequential processing.

[0060] S4: Construct a multimodal fusion tower and project it onto a unified dimension, specifically:

[0061] S41: Projection Layer Design and Initialization. Design a learnable linear projection layer to map heterogeneous textual and visual features to a unified representation space. Text projection matrix. Dimension is Text features mapped to a unified dimension Visual projection matrix Dimension is Visual features are mapped to the same uniform dimension Unified Dimension Typically, the value is set to 512 or 768 to ensure sufficient representational power while controlling computational complexity. The projection matrix uses Xavier or He initialization methods to ensure gradient stability in the early stages of training. Optionally, a bias term can be added after the projection layer, but in practice, good results can be achieved without it. The projection operation can be understood as a linear transformation of the feature space, adaptively adjusting the representation of different modal features through learnable parameters to make them comparable in a unified space.

[0062] S43: Projective transformation of visual sequences. Regarding the mentioned visual sequence features... Multiply the feature vector at each patch location by the visual projection matrix to calculate... , to obtain the projected visual sequence The visual sequence features of entities are transformed using the same projection matrix to obtain... The projected visual sequence and the projected text sequence have the same feature dimensions. This provides a dimensional consistency guarantee for subsequent cross-modal interactions.

[0063] S43: Projective transformation of visual sequences. Regarding the mentioned visual sequence features... Multiply the feature vector at each patch location by the visual projection matrix to calculate... , to obtain the projected visual sequence The visual sequence features of entities are transformed using the same projection matrix to obtain... The projected visual sequence and the projected text sequence have the same feature dimensions. This provides a dimensional consistency guarantee for subsequent cross-modal interactions.

[0064] S44: Parameter Learning Strategy for the Projection Layer. The parameters of the projection layer, as part of the multimodal fusion tower, are jointly optimized with other parameters during end-to-end training. In the early stages of training, the projection layer quickly learns to map features from different modalities to a suitable unified space. In the later stages of training, the parameter changes of the projection layer gradually decrease, primarily fine-tuned by subsequent hybrid expert layers and attention layers. Different learning rate strategies can be employed: using the same learning rate for the projection layer as for subsequent layers, or using a slightly larger learning rate to accelerate convergence. The gradients of the projection layer are typically relatively stable and do not require special gradient clipping. The number of parameters introduced by the projection operation is relatively small, approximately [missing information]. It will not significantly increase the parameter size of the model.

[0065] S5: A hybrid expert mechanism is used for adaptive sample processing, specifically:

[0066] S51: Expert Network Structure Design. Define K expert networks, each consisting of two layers of feedforward neural networks with the same network structure but independent parameters. The formula for calculating the k-th expert is: ,in This is the first layer weight matrix. This is the weight matrix for the second layer. The hidden layer dimension is usually set to... It is 2 to 4 times faster than ReLU. The GELU activation function provides a smoother non-linear transformation compared to ReLU and is widely used in Transformer-type models. LayerNorm layer normalization stabilizes the training process and accelerates convergence. Each expert is essentially a feature mapping function with non-linear transformation capabilities, learning different feature transformation patterns through different parameter configurations. The number of experts K is usually set to 4 to 8; too few experts will limit the model's expressive power, while too many will increase computational overhead and may lead to training difficulties.

[0067] S52: Weight generation in the gating network. The gating network determines which experts are activated for each token, achieving sample-adaptive expert selection. This applies to the projected text sequence. Each token is processed by a gating network to compute expert activation weights. The gating network contains a linear transformation layer with parameters of... , with dimensions as The input features are mapped to K-dimensional gated logits. Normalization is performed using a temperature-scaled softmax function, calculated as follows: ,in The temperature parameter is a learnable parameter, typically set to an initial value between 1.0 and 2.0. The temperature parameter controls the sharpness of the weight distribution; smaller temperature values ​​result in a more concentrated weight distribution, while larger temperature values ​​result in a smoother distribution. (Gated weights) It is a K-dimensional probability distribution vector, the sum of all elements is 1, and each element represents the probability of the corresponding expert being selected.

[0068] S53: Sparse Activation Strategy. A top-p sparse activation strategy is adopted, activating only the p experts with the highest gating scores for each token; typically, p is set to 2. Sparse activation significantly reduces computational complexity compared to full activation; when K=4, activating 2 experts reduces computation by approximately 50%.

[0069] S54: Weighted combination of expert output. For the selected... Each expert calculates their transformation result for the current token features. The input features... Enter each selected expert The expert outputs are obtained. The outputs of all selected experts are then weighted and summed according to normalized weights. The calculation formula is as follows:

[0070]

[0071] The final output of the hybrid expert layer is obtained, and the dimension remains the same. The weighted combination operation allows the model to integrate the feature transformation results of multiple experts, with each expert contributing their area of ​​expertise. This process is repeated for all token positions in the text sequence to obtain the text sequence processed by the hybrid experts. .

[0072] S55: Hybrid expert processing of visual sequences. This involves processing the projected visual sequences. The exact same hybrid expert mechanism is applied. Independent gating network parameters are used. Calculate the expert activation weights for visual tokens, or share gating network parameters with text tokens. The structure of the visual expert network is the same as that of the text expert network, but the parameters are independent, denoted as... For each visual patch, weights are generated using a gating network and top-weighted algorithms are applied. Select and calculate the weighted combination output of the selected experts.

[0073]

[0074] Obtain the visual sequence after hybrid expert processing The text and visual models employ hybrid expert mechanisms, enabling the model to perform specific feature transformations based on the characteristics of different modalities.

[0075] S56: Expert Dropout Mechanism. During training, an expert dropout strategy is employed to enhance expert diversity and prevent a few experts from dominating the training process. This is done using probability... Randomly disabled For each expert, the dropout probability is typically set to 0.1 to 0.3. In each training batch, a set of experts to be disabled is randomly selected. The gating score for disabling experts is set to... ,Right now for This way, after softmax normalization, the weights of disabled experts automatically become 0 and will not be selected. Expert dropout forces the model to learn to use all experts, preventing some experts from being completely ignored. During the evaluation phase, dropout is not used, and all experts can be selected normally.

[0076] S57: Forced exploration during the warm-up period. Before training... Each step is designated as a warm-up phase, typically consisting of 1000 to 5000 steps. During this phase, the output of the gating network is ignored, and p experts are randomly selected for each token, with all selected experts assigned equal weights. The calculation formula is: ,in This means randomly selecting p experts from K experts. The forced exploration strategy ensures that all experts have an equal chance of being selected and updated in the early stages of training, preventing the gating network from prematurely solidifying its expert selection pattern due to improper initialization. After the warm-up period, the network gradually transitions to a normal gating selection mechanism, which can be smoothly achieved using linear interpolation.

[0077] S58: Gumbel Noise Enhancement. After the warm-up phase, Gumbel noise is added to the gated logits to enhance exploration. Gumbel noise is defined as... ,in The numbers are uniformly distributed random numbers. Noise is added to the gated logits, resulting in... ,in Here are the noise intensity coefficients. Softmax is applied to the logits after adding noise to obtain the gating weights. The noise intensity gradually decays exponentially or linearly during training, for example... or , where t is the current training step number and T is the total training step number. The noise enhancement mechanism provides moderate randomness in the middle of training to prevent the gated network from converging to a local optimum too early, and the noise decays to zero in the later stage of training, so that the model stabilizes near the optimal solution.

[0078] S6: Fine-grained alignment is established through bidirectional cross-modal attention, specifically:

[0079] S61: Structural Design of the Multi-Head Attention Mechanism. A multi-head attention mechanism is employed to achieve cross-modal interaction; the number of attention heads H is typically set to 8 or 16. Multi-head attention allows the model to focus on different semantic association patterns in parallel across different representation subspaces. Projection parameters for query, key, and value are defined. For the h-th attention head, the query projection matrix... Key projection matrix Value projection matrix ,in For each head dimension, output the projection matrix. Used to fuse the outputs of all heads. The standard attention calculation formula is: , of which It is a scaling factor to prevent the dot product result from becoming too large.

[0080] S62: Text-to-Image Attention Calculation. This involves establishing a text-to-image attention mapping, using text terms as queries and visual blocks as keys and values. The attention is applied to the mentioned text sequences after hybrid expert processing. and visual sequences Calculate each attention head Output: The text sequence is obtained through query projection. The visual sequence is obtained by key projection. Obtained through value projection Calculate the attention score matrix. , dimension Elements in the matrix This represents the attention weight of the i-th text token to the j-th visual patch. Calculate the attention output. , dimension Concatenate the outputs of all H heads to obtain... , dimension Transformation via the output projection matrix .

[0081] S63: Residual Connections and Layer Normalization. The attention output is residually connected to the original text sequence, and layer normalization is calculated. Residual connections preserve information from the original text features, preventing information loss due to excessive transformations introduced by the attention mechanism. Layer normalization stabilizes the feature distribution and accelerates training convergence. The normalized sequence... Dimensions The feature vector at each token location integrates the original text semantics and the contextual information of visual perception.

[0082] S64: Image-to-Text Attention Calculation. A symmetric mechanism is employed, using visual blocks as queries and text terms as keys and values ​​to establish an image-to-text attention mapping. The calculation process is similar to text-to-image attention, using visual sequences... As the query source, the text sequence As a key-value source. For each attention head ,calculate , , Note that the projection matrix here can be shared with the text-to-image method or use independent parameters. Calculate the attention score matrix. , dimension Calculate the attention output and concatenate all heads to obtain... By applying residual connectivity and layer normalization, the calculation is performed. , dimension .

[0083] S65: The Synergistic Effect of Bidirectional Attention. Text-to-image attention allows each text token to focus on image regions that are semantically relevant to it; for example, the phrase "red sports car" will focus on the red car region in the image. Image-to-text attention allows each visual patch to focus on text terms describing that region; for example, a patch containing "car" will focus on related words like "sports car." The bidirectional attention mechanism ensures the symmetry and comprehensiveness of cross-modal alignment, with text and vision mutually guiding each other to focus on the most relevant information. Compared to unidirectional attention, the bidirectional mechanism can establish a more robust cross-modal semantic mapping; even if alignment fails in one direction, the other direction can still provide effective information.

[0084] S66: Pooling generation of global representations. Attention output sequence from text to image. Perform mean pooling along the sequence length dimension to calculate The dimension is obtained as The text global representation is generated. This representation aggregates features from all text token locations, and the features at each location have been fused with visual information through an attention mechanism. Therefore, the global representation encodes the visual information that enhances the text semantics. The image-to-text attention output sequence is then processed. Perform the same mean pooling operation to calculate The dimension is obtained as The two global representations encode the visually enhanced textual information. These two global representations provide input for subsequent channel attention fusion.

[0085] S67: Bidirectional attention processing on the entity side. Text sequences for entities. and visual sequences It employs the exact same bidirectional cross-modal attention mechanism. It computes the text-to-image attention output for entities. Image-to-text attention output Generate a global text representation of entities using mean pooling. and visual global representation Mentions and entities employ the same bidirectional attention process to ensure that their features reside in the same semantic space, facilitating subsequent similarity calculations.

[0086] S7: Employs a channel attention mechanism to dynamically fuse text and visual representations, specifically:

[0087] S71: Feature concatenation and attention network input. This involves globally representing the mentioned text. and visual global representation By concatenating along the feature dimensions, we obtain the concatenated representation. , dimension The concatenation operation preserves the independent information of the two modalities, providing complete input for the subsequent attention network. Layer normalization is performed on the concatenated representation, calculating the mean and variance of the vector elements and then normalizing them to obtain... Layer normalization stabilizes the numerical range of features, which is beneficial for subsequent network training.

[0088] S72: Weight Generation in a Multilayer Perceptron. The channel attention network consists of three fully connected layers, forming a multilayer perceptron with non-linear transformation capabilities. The first layer transforms the input dimension from... Dimensional reduction The parameters are ,calculate The GELU activation function introduces nonlinearity, enabling the network to learn complex modal relationships. The second layer further reduces the dimensionality to... The parameters are ,calculate Through two dimensionality reduction and nonlinear transformations, the network extracts the key features required for modality fusion. The third layer output has a dimension of 2, corresponding to both text and visual channels, with parameters as follows. Calculate logits .

[0089] S73: Temperature-scaled softmax normalization. Apply the temperature-scaled softmax function to the output logits to calculate... ,in The temperature parameter is a learnable parameter, typically initialized to 1.0 to 2.0. The temperature parameter controls the sharpness of the weight distribution; smaller temperature values ​​result in a sharper weight distribution, causing the model to tend to explicitly select a particular mode; larger temperature values ​​result in a smoother weight distribution, with relatively balanced weights across the two modes. (Normalized weights) and satisfy and This indicates the relative importance of textual and visual modalities.

[0090] S74: Element-wise weighted fusion. Applicable weights are applied to the text global representation and the visual global representation, and then element-wise multiplication is performed to calculate... and The dot product here is an element-wise multiplication of a scalar and a vector, with weights... and A uniform scaling is applied to all dimensions of the entire feature vector. The weighting operation adjusts the contribution of each modality according to its relative importance; important modalities receive greater weights, and their features are amplified; unimportant modalities receive smaller weights, and their features are suppressed.

[0091] S75: Feature transformation of the fusion layer. The weighted text and visual representations are concatenated to obtain... , dimension The transformation is achieved through a two-layer feedforward neural network. The first layer calculates... ,in The output dimension is Second-level calculation ,in The output dimension remains the same. Application layer normalization yields... The fusion layer integrates the weighted multimodal features into a unified representation through nonlinear transformation.

[0092] S76: L2 normalization generates the final embedding. The output of the fusion layer is L2 normalized, and the embedding is calculated. ,in Let L2 be the L2 norm of the vector. The normalized embedding vector has a magnitude of 1, i.e. L2 normalization ensures that all embedding vectors lie on the unit hypersphere, at which point the vector dot product is equivalent to the cosine similarity. ,in Let be the angle between the two vectors. The cosine similarity value ranges from 1 to 2. A larger value indicates greater similarity between the vectors. L2 normalization also prevents differences in vector magnitude from affecting similarity calculations, making the matching more equitable. The final embedding vectors... As a reference, it is used for subsequent similarity calculations and entity linking.

[0093] S77: Channel attention fusion on the entity side. Global text representation of entities. and visual global representation The exact same channel attention mechanism is employed. The representations of the two modalities are concatenated, modal weights are generated through a three-layer perceptual network, element-wise weighting is applied to the representations, a fusion layer transformation is performed, and finally L2 normalization is applied to obtain the final embedding vector of the entity. The same fusion process and parameters are used for mentions and entities to ensure that their embedding vectors can be directly compared in the same metric space.

[0094] S78: Channel attention regularization loss. This applies channel attention regularization constraints during training to prevent the model from completely ignoring a particular modality. Define the regularization loss. ,in This means taking the expected value for all samples in the batch. The larger of the two weights. For the smaller value, the difference Indicates the degree of imbalance in weights. Threshold The maximum allowable weight difference is typically set to 0.3 to 0.5. When the weight difference exceeds this threshold, The function produces positive values, which penalize the model; the penalty is zero when the weight difference is within a threshold. Loss weights The value is typically set to 0.1 to 0.5 to control the strength of regularization. This constraint encourages moderate modality balance, avoids the complete neglect of any particular modality, and allows for sample-specific adaptive adjustments without forcing equal distribution.

[0095] S8: The dual encoder model is trained using a contrastive learning objective function, specifically as follows:

[0096] S81: Embedding computation for batch data. During the training phase, a batch containing B mention-entity pairs is constructed, with each sample pair... middle for The correct target entity. All mentions in the batch. The embedding vectors are calculated by referring to the encoder. Each embedding vector has a dimension of 1. And it has undergone L2 normalization. For all entities in the batch. The embedding vectors are calculated using the entity encoder. The same dimension is It is also L2 normalized. In parameter-sharing mode, the mention encoder and the physical encoder use the same parameters, but the input data is different. The batch size B is set according to the GPU memory capacity, typically between 16 and 128.

[0097] S82: Calculation of the similarity matrix. Calculate the similarity between all mentions and all entities within a batch, and construct the similarity matrix. Matrix elements Let represent the similarity between the i-th mention and the j-th entity, calculated using the formula: The inner product Equivalent to cosine similarity, This is a temperature parameter, typically set to 0.05 to 0.1. The temperature parameter scales the similarity; smaller temperature values ​​result in a sharper similarity distribution, which is beneficial for distinguishing between positive and negative samples. (Diagonal elements) Similarity of positive sample pairs, off-diagonal elements ( The similarity matrix is ​​the similarity between negative sample pairs. The similarity matrix can be efficiently calculated using matrix multiplication. ,in To mention the embedding matrix, This is an entity embedding matrix.

[0098] S83: InfoNCE loss for mentioning entities. Define the contrastive learning loss for mentioning entities. For each mention... Its positive samples are the corresponding entities. Negative samples are other entities in the same batch. . set up Indicates mention With entity The similarity between samples is calculated by the index value of positive sample similarity. The sum of the similarity indices of all candidate entities is The InfoNCE loss in the entity direction is defined as follows:

[0099]

[0100] This loss function maximizes the similarity of positive sample pairs while minimizing the similarity of negative sample pairs using a negative log-likelihood approach. When the similarity of positive samples is significantly higher than that of negative samples, the corresponding probability is approximately... The loss is close to When the similarity between positive and negative samples is close, the corresponding probability is small and the loss value is large.

[0101] S84: Symmetric Loss from Entity to Mention. Defines a contrastive learning loss for the symmetric entity-to-mention direction, for each entity... Its positive samples are Negative samples are other mentions in the batch. The calculation formula is: Symmetric loss ensures that the matching from entity to mention direction is also optimized, enabling effective training for both bidirectional similarity calculations. The use of bidirectional contrastive loss is a common practice in contrastive learning, improving the quality of representation learning.

[0102] S85: Weighted combination of primary search targets. The entity mention loss and entity-to-mention loss are weighted and combined to form the primary search targets. ,in and This is the weighting coefficient, typically set to 0.5 to 1.0, and can be set to equal values, such as... Alternatively, the model can be tuned based on validation set performance. Weighted combination balances the loss contributions from both directions, enabling the model to achieve good performance in both bidirectional matching. InfoNCE loss is the core objective function of contrastive learning, efficiently constructing positive and negative sample pairs through in-batch negative sampling, avoiding the computational overhead of full-database negative sampling.

[0103] S86: Definition of Load Balancing Loss. To facilitate the balanced use of all experts in a hybrid expert mechanism, load balancing loss is defined as comprising several components. Switch Transformer Loss ,in The frequency of use of expert k. Let L be the total number of times expert k is activated in the batch, and L be the average sequence length. Let k be the average importance of expert k, i.e., the average gating score when selected. Variance penalty. Directly constraining the variance of usage frequency promotes uniform distribution. Insufficient usage is penalized. A penalty is imposed when the expert uses the technology less than half the expected frequency. Overuse is penalized. A penalty is imposed when experts use the technology more than twice the expected frequency. Entropy regularization. Encouraging uncertainty in gating distribution, among which Let be the gating score of the l-th token for expert k. The combined load balancing loss is... The weight coefficients for each item are determined through validation set experiments, typically arrive Set between 0.1 and 1.0. Set it to 0.01 to 0.1.

[0104] S87: Definition of Expert Diversity Loss. To promote the learning of orthogonal feature transformations by different experts and avoid functional redundancy, expert diversity loss is defined. The output signature of each expert k on a batch is calculated. ,in Given the number of activations for expert k, sum the token positions for all activated experts k. The output signature is the average output vector of the expert over the batch, with dimension 1. This characterizes the feature transformation patterns of experts. The cosine similarity between different expert signatures is calculated, and the diversity loss is defined as... ,in The cosine similarity is calculated by averaging the absolute values. This loss minimizes the similarity between the outputs of different experts, encouraging them to learn different feature transformations. Loss weights It is typically set to 0.05 to 0.2.

[0105] S88: Construction of the final training objective. The InfoNCE loss, load balancing loss, expert diversity loss, and channel attention regularization loss are weighted and summed to form the final training objective function. The objective function jointly optimizes four aspects: retrieval performance, expert balance, expert diversity, and modality balance. Through the synergistic effect of multiple constraints, the model can learn high-quality multimodal representations while avoiding various degradation problems during training.

[0106] S89: Backpropagation and parameter update. The batch-average loss is obtained by averaging the loss of all samples in the batch. The gradient of the loss function with respect to all model parameters is calculated using an automatic differentiation mechanism. This includes CLIP text encoder parameters, CLIP visual encoder parameters, projection layer parameters, hybrid expert network parameters (including expert parameters and gating parameters), bidirectional attention module parameters (including query, key-value projection parameters, and output projection parameters), channel attention fusion layer parameters (including three-layer perceptual network parameters and fusion layer parameters), and various learnable temperature parameters. The model parameters are updated using either the Adam or AdamW optimizer. The Adam optimizer includes first-order moment estimation and second-order moment estimation, enabling adaptive adjustment of the learning rate for each parameter. The learning rate is set between 1e-5 and 5e-4. For pre-trained CLIP encoder parameters, a smaller learning rate, such as 1e-6 to 1e-5, is used for fine-tuning to avoid disrupting pre-trained knowledge; for newly added fusion tower parameters, a larger learning rate, such as 1e-4 to 5e-4, is used to accelerate convergence. Gradient clipping is applied; when the gradient norm exceeds a threshold (e.g., 1.0 or 5.0), the gradient is scaled down to within the threshold. The calculation formula is as follows: if This is to prevent gradient explosion from causing training instability.

[0107] S810: Training Strategy and Hyperparameter Configuration. Set the training batch size between 16 and 128, adjusting according to GPU memory capacity and model size. Larger batches provide more negative samples, which is beneficial for comparative learning. Set the total number of training epochs between 10 and 50, employing an early stopping strategy. Terminate training prematurely if validation set performance does not improve for several consecutive epochs (e.g., 3 to 5 epochs). Use a warmup learning rate scheduling strategy, linearly increasing the learning rate from 0 to the target learning rate in the early stages of training, with 1000 to 5000 warmup steps. Subsequently, use cosine annealing or linear decay strategies to gradually decrease the learning rate. Set the weight decay coefficient between 1e-4 and 1e-2 to prevent overfitting. Employ dropout regularization, adding a dropout layer after the attention layer and fully connected layer, with a dropout rate set between 0.1 and 0.3. Periodically evaluate model performance on the validation set, calculating metrics such as accuracy, recall@K, and mean reciprocal rank (MRR), and save the model parameters with the best overall performance.

[0108] S9: Perform efficient entity linking during the inference phase, specifically:

[0109] S91: Offline pre-computation of entity embeddings. Before the inference phase begins, all entities in the knowledge base are pre-computed. Perform offline pre-computation. For each entity in turn... Extract its text description and image collection The embedding vector is calculated by the entity encoder. The embedding vectors are L2 normalized to a magnitude of 1. All entity embeddings are organized into an embedding matrix. The data is stored on disk or in a database. For large-scale knowledge bases, a chunked processing strategy can be used to embed entity calculations in batches, avoiding memory overflow.

[0110] S92: Constructing an Efficient Index Structure. To accelerate nearest neighbor search, an approximate nearest neighbor search algorithm such as FAISS is used to construct the index structure. FAISS provides various index types, including the Flat index for exact search, the IVF index based on inverted indexes, and the HNSW index based on graphs. For small to medium-sized knowledge bases (tens of thousands to hundreds of thousands of entities), the Flat index can be used for exact search, with a time complexity of O(log n). However, the constant term is very small. For large-scale knowledge bases (millions of entities), IVF or HNSW indexes are recommended, reducing the time complexity to [missing value]. or approximate The search speed is improved by hundreds to thousands of times. The index building process includes: inputting the embedding matrix into FAISS, selecting the appropriate index type and parameters, performing index training and addition operations, and saving the built index to disk.

[0111] S93: Online mention of embedded computing. For new mentions to be linked. Extract its text context and associated images By mentioning that the encoder performs one forward propagation, its embedding vector is computed. The embedded vectors are L2 normalized. Forward propagation includes: CLIP text encoding to generate word-level sequences, CLIP visual encoding to generate patch-level sequences, projection to a unified dimension, hybrid expert mechanism processing, bidirectional cross-modal attention calculation, channel attention fusion, and L2 normalization. The entire computation process has a time complexity of O(n log n). (Relative to the size of the entity library), it depends only on the sequence length of the mentions and the network depth of the model. For batch queries, multiple mentions can be organized into batches, leveraging the parallel computing power of GPUs to compute their embeddings simultaneously, thereby improving throughput.

[0112] S94: Nearest Neighbor Search and Entity Ranking. (Embedding will be mentioned.) As the query vector, a nearest neighbor search is performed in the pre-built entity embedding index, calculated using the following formula: ,in The similarity is calculated using the cosine similarity (inner product). For exact searches, the similarity is calculated across all entity embeddings, and the maximum value is selected. For approximate searches, the FAISS index quickly returns the top-K most similar entities and their similarity scores, where K is typically set between 1 and 10. Candidate entities are then sorted in descending order based on their similarity scores, with the entity with the highest similarity being considered the most likely link target.

[0113] S95: Link Results Output and Confidence Assessment. Select the entity ranked highest. As the final entity linking result, the output includes complete entity information, such as entity ID, entity name, and entity type. The confidence score is defined as the similarity between the entity and the mentioned entity. The range of values ​​is Typically, positive samples have a similarity score above 0.5, while negative samples have a similarity score below 0.3. Optionally, the top-N candidate entities and their corresponding confidence scores can be output for downstream applications or manual review; N is typically set between 3 and 10. Links with low confidence scores (e.g., similarity scores below a preset threshold of 0.3) are marked as "uncertain," suggesting manual intervention or further verification. For mentions without suitable candidates (all candidate entities have very low similarity), "NIL" is output, indicating that the mention does not link to any entity in the knowledge base.

[0114] S96: Dynamic Updates and Online Maintenance. When a new entity is added to the knowledge base, only its embedding vector needs to be calculated and added to the index; retraining the model is unnecessary. The specific process is as follows: extract the text and image information of the new entity, calculate the embedding using an entity encoder, and add the embedding vector to the embedding matrix and index structure. For FAISS's Flat index, the addition operation is a simple vector append, with a time complexity of O(log n). For IVF or HNSW indexes, an incremental update operation needs to be performed, with a time complexity of O(n log n). When entity information in the knowledge base is modified or deleted, the embeddings of the affected entities are recalculated and the index is updated, or the corresponding vectors are deleted from the index. This dynamic update capability allows the system to adapt to the continuous evolution of the knowledge base without the need for expensive model retraining, significantly reducing maintenance costs.

[0115] S97: Batch Processing and Performance Optimization. For samples containing multiple mentions, all mentions can be processed in batches, leveraging the parallel computing capabilities of GPUs to improve efficiency. All mentioned text and images are organized into batches, and their embedding vectors are computed in parallel using a mention encoder. Then, nearest neighbor search is performed on each mention separately. For high-concurrency online service scenarios, asynchronous processing and request queuing mechanisms can be used to aggregate multiple user requests into batches, improving throughput. Model quantization and acceleration techniques, such as INT8 quantization, model pruning, and knowledge distillation, are employed to reduce model size and computational overhead, enabling the system to be deployed in resource-constrained environments. For ultra-large-scale knowledge bases, a distributed indexing and retrieval architecture can be used, sharding the entity database across multiple nodes to execute search operations in parallel.

[0116] This invention implements a multimodal entity linking method based on dual encoders and a hybrid expert mechanism through the aforementioned complete processing flow. This method achieves independent encoding of mentions and entities through a dual-tower architecture, enabling offline pre-computation of entity embeddings and the establishment of efficient indexes. During inference, entity linking is completed through fast vector retrieval, achieving speedups of hundreds to thousands of times on large-scale knowledge bases. The hybrid expert mechanism achieves adaptive feature transformation of samples, with different experts learning different processing modes, and a gating network dynamically selecting the most suitable expert combination. Bidirectional cross-modal attention establishes fine-grained alignment between text and vision at the token-patch granularity, enabling the model to accurately match local semantic correspondences. Channel attention dynamically balances the contributions of text and visual modalities, adaptively adjusting modal weights. Multiple regularization constraints, including load balancing loss, expert diversity loss, and channel attention regularization, systematically promote balanced expert learning, feature diversification, and modal balance, significantly improving the model's robustness and generalization ability.

[0117] This invention is not limited to the technical solutions described in the specific embodiments above. Those skilled in the art, inspired by the technical solutions of this invention, can make various equivalent substitutions or obvious modifications, all of which are within the scope of protection of this invention. For example, the CLIP encoder can be replaced with other pre-trained multimodal models such as ALIGN, BLIP, CoCa, etc.; the number of experts, activations, and network structure in the hybrid expert mechanism can be adjusted according to specific application requirements; bidirectional cross-modal attention can adopt other variations such as cross-attention, a combination of self-attention and cross-attention, etc.; the channel attention mechanism can be extended to support the fusion of more modalities, such as the trimodal fusion of text, image, and audio; the loss function can be combined with triplet loss, other variations of contrastive loss, knowledge distillation loss, etc.; the inference stage can be combined with a re-ranking model to finely rank candidate entities, further improving accuracy; these equivalent solutions are all within the scope of protection of this invention.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal entity linking method based on dual encoders and a hybrid expert mechanism, characterized in that, Includes the following steps: S1: Receive the link to mention m and its text context. and associated images From the knowledge base Obtain the candidate entity set, where each entity Includes text description and image collection ; S2: Employs a dual-encoder architecture to independently encode mentions and entities, through a mention encoding function. Encode the mentioned text and images into embedding vectors. Through entity encoding function Encode the entity's text and image into embedding vectors. ,in This allows entity embeddings to be pre-computed and stored offline, and entity linking during inference is simplified to nearest neighbor search. S3: During the encoding process, the CLIP pre-trained model is used as the basic encoder for the mentioned text. Generate word-level sequences using CLIP text encoder Regarding the image mentioned Image block-level sequences are generated using the CLIP visual encoder. ,in and The lengths of the text and visual sequences are respectively. and Using the original feature dimension, the same encoding method is applied to the entities to obtain the corresponding sequences; S4: Construct a multimodal fusion tower to combine text and visual sequences using a learnable projection matrix. and Projected onto a unified dimension The projected sequence is obtained. and ; S5: A hybrid expert mechanism is used to perform sample adaptive processing on the projected sequence, defining K expert networks. ,in and For the parameters of the k-th expert, each word is processed through a gating network. Generate expert activation weights, select p experts using a top-p sparse activation strategy, and output the normalized weighted combination of the selected experts. The same hybrid expert mechanism is applied to visual sequences; S6: Fine-grained alignment between text and vision is established through bidirectional cross-modal attention, and a multi-head attention mechanism is adopted to generate text-to-image attention output by querying visual blocks using text lexical terms. Simultaneously, it enables visual block query text lexical generation to produce images and text attention output. Modality-specific global representations are generated through mean pooling. and ; S7: Employs a channel attention mechanism to dynamically fuse text and visual representations, combining the spliced ​​representations. Modal weights are generated using an attention network. Apply element-level weighting to the fusion representation and The final embedding is generated through the fusion layer. And perform L2 normalization; S8: Train a dual encoder model using a contrastive learning objective function, and compute the cosine similarity matrix in a batch containing B mention-entity pairs. The definition mentions the InfoNCE loss of entities. Symmetric loss from entity to mention Through weighted combination The main retrieval target is formed, and load balancing loss, expert diversity loss and channel attention regularization constraints are applied to jointly optimize all learnable parameters of the encoder and fusion tower. S9: During the inference phase, the embedding vector of the link mention is calculated, and a nearest neighbor search is performed in the pre-computed entity embedding library using cosine similarity. The entity with the highest similarity is output as the link result.

2. The multimodal entity linking method based on dual encoders and hybrid expert mechanisms according to claim 1, characterized in that, The hybrid expert mechanism in S5 incorporates multiple diversity enhancement strategies: During training, an expert dropout mechanism is employed, based on probability. Randomly disabled One expert set its gating score to... Force the model to learn using all experts; in front In the warm-up phase of each training step, ignoring the gating score, p experts are randomly selected for each word and assigned equal weights. This ensures that all experts receive sufficient gradient updates during early training. Gumbel noise is added to the gated logits after the warm-up phase. Enhanced exploration, among which The noise intensity gradually decreases during training.

3. The multimodal entity linking method based on dual encoders and hybrid expert mechanisms according to claim 1, characterized in that, The bidirectional cross-modal attention in S6 employs a multi-head attention mechanism. Each attention head is defined as By using H attention heads to compute linear transformations of queries, keys, and values ​​in parallel, fine-grained correspondences can be established between text terms and visual blocks in different representation subspaces. Residual connections preserve the original modal information while injecting cross-modal context.

4. The multimodal entity linking method based on dual encoders and hybrid expert mechanisms according to claim 1, characterized in that, The channel attention mechanism in S7 adaptively learns the relative importance of text and visual modalities through a multilayer perceptual network, wherein the attention network contains three fully connected layers. , and Temperature parameters can be learned through nonlinear transformation using the GELU activation function and layer normalization. Control the sharpness of the weight distribution while applying channel attention regularization loss. To prevent the model from completely ignoring a certain mode, where This is the threshold for the maximum allowed weight difference.

5. The multimodal entity linking method based on dual encoders and hybrid expert mechanisms according to claim 1, characterized in that, The contrastive learning objective function in S8 incorporates multiple regularization constraints, including: Load balancing losses Loss due to Switch Transformer Variance penalty Penalties for underuse and overuse and and entropy regularization Composition, in which The frequency of expert k's use in the batch. Its importance; Loss of expert diversity Minimize the output signatures of different experts Cosine similarity between them facilitates expert learning of orthogonal feature transformations; Ultimate training goal Jointly optimize retrieval performance with representation diversity and balance.

6. The multimodal entity linking method based on dual encoders and hybrid expert mechanisms according to claim 1, characterized in that, The dual encoder architecture mentions the encoding function. and entity encoding function By adopting a parameter-sharing strategy, the two encoder towers use the same CLIP base encoder, projection layer, hybrid expert network, bidirectional attention module, and channel attention fusion layer parameters, which reduces model complexity and promotes knowledge transfer, while supporting independent parameterized configuration to adapt to different application scenarios.

7. The multimodal entity linking method based on dual encoders and hybrid expert mechanisms according to claim 1, characterized in that, The processing method of the CLIP encoder in S3 includes: The text input is encoded using the CLIP text encoder's Transformer architecture, and the word-level representations of all hidden layers are extracted as sequence features. The sequence length is... Determined by the number of lexical units after text segmentation; The image input is divided into fixed-size visual blocks by the CLIP visual encoder and linearly projected. After processing by the Vision Transformer, all block-level representations are extracted as sequence features. The sequence length is... Determined by the image block partitioning strategy; For entity image set During the training phase, an image is randomly selected for encoding, and during the inference phase, the first image is selected or a multi-image aggregation strategy is adopted.

8. The multimodal entity linking method based on dual encoders and hybrid expert mechanisms according to claim 1, characterized in that, The entity linking process in the inference phase supports large-scale, efficient retrieval: Embedding vectors of all knowledge base entities It can be pre-calculated and stored offline, avoiding online coding overhead; For a new mention, only one forward propagation is needed to compute its embedding vector. The time complexity is ; Using approximate nearest neighbor search algorithms such as FAISS for efficient retrieval in entity embedding libraries, compared to joint coding methods... Significantly accelerated in terms of complexity; It supports dynamic updates to the entity library. When adding a new entity, you only need to calculate its embedding and add it to the index, without retraining the model.

9. A multimodal entity linking system based on dual encoders and a hybrid expert mechanism, employing the multimodal entity linking method as described in any one of claims 1 to 8, characterized in that, The system includes: CLIP Basic Encoding Module: Employs a pre-trained CLIP model to independently encode the text and images of mentions and entities, generating word-level and image block-level sequence representations; Multimodal projection module: Maps text and visual features of different dimensions to a unified semantic space through a learnable linear projection matrix; Hybrid expert processing module: It includes K expert networks and a gated routing mechanism to achieve adaptive feature transformation of samples, and is equipped with diverse guarantee strategies such as expert dropout, forced exploration and Gumbel noise enhancement; Bidirectional cross-modal attention module: Establishes bidirectional fine-grained alignment between text and image and image and text through a multi-head attention mechanism, generating a global representation that encodes cross-modal correspondences; Adaptive Channel Fusion Module: Employs an attention network to dynamically learn the relative weights of text and visual modalities, and is equipped with channel regularization constraints to prevent modal degradation; The contrastive learning training module implements bidirectional InfoNCE loss calculation for in-batch negative sampling, and performs end-to-end optimization by combining multiple regularization constraints such as load balancing, expert diversity, and channel attention. Efficient retrieval reasoning module: Maintains a pre-computed entity embedding index and performs fast nearest neighbor search on new mentions to achieve scalable entity linking.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the multimodal entity linking method based on dual encoders and hybrid expert mechanisms as described in any one of claims 1 to 8.

Images