Cross-view reflection ranking distillation method, device and medium for entity knowledge index

Abstract

The present application relates to a kind of entity knowledge index cross-view introspection ordering distillation method, equipment and medium, the method is through ordering knowledge transfer to improve the robustness of entity linking under visual absence.Based on complete multi-modal data, teacher model is trained, and pure text data is input into student model (inherit teacher architecture and parameter).Specifically, it includes: ordering alignment loss: through the differentiable ordering alignment teacher (complete modality) and the entity ordering distribution of student (absence modality), the semantic distinguishability is retained;Cross-view distillation: utilize teacher entity-mention reverse ordering to enhance multi-modal representation understanding;Self-consistency constraint: force student mention-entity ordering and reverse entity-mention ordering consistent, break through one-way search limitation.Three-loss joint optimization, and student model only needs text input is deployed.Compared with traditional probability distillation, through ordering transmission and cross-view introspection mechanism, the stability in absence scene is significantly improved, and there is no additional parameter.

Description

Technical Field

[0001] This invention relates to the fields of multimodal machine learning and knowledge graphs, specifically to a cross-view reflective sorting distillation method, apparatus, and medium for entity knowledge indexing. Background Technology

[0002] Multimodal entity linking (MEL) is an important technology in the field of information retrieval, aiming to associate ambiguous references containing textual and visual information with corresponding entities in a multimodal knowledge base. With the rapid development of social media and knowledge-intensive applications (such as search engines and question-answering systems), MEL effectively alleviates the ambiguity problem of traditional text entity linking by integrating multimodal information, becoming a key technology for improving content understanding and reasoning capabilities.

[0003] However, existing MEL methods are typically based on the modality integrity assumption, which assumes that the input data contains complete textual and visual information during both the training and inference phases. This assumption has significant limitations in real-world applications. Due to factors such as user privacy protection, device acquisition limitations, or the complexity of the operating environment, input mentions in real-world scenarios may frequently lack visual modalities, leading to input asymmetry problems in MEL systems (i.e., mentions that are only textual need to match multimodal entities). Experiments show that when the visual modality is completely missing, the retrieval performance (such as Top-1 hit rate) of state-of-the-art MEL models can drop by more than 20%, seriously threatening their practicality in open environments.

[0004] To address the problem of missing modalities, various solutions have been proposed in the field of multimodal learning, including data completion based on generative models, designing missing-resistant network architectures, or using knowledge distillation to transfer multimodal knowledge. However, these methods are mainly geared towards classification tasks (such as sentiment analysis and medical diagnosis), and have the following shortcomings in retrieval-based MEL tasks: 1) Generative methods rely on additional model parameters and have low reliability in data completion; 2) Dedicated architectures are difficult to adapt to different base models and have poor generalization; 3) Traditional knowledge distillation only constrains the output distribution through KL divergence, failing to effectively convey the key contribution of visual modalities to semantic discriminability, resulting in insufficient robustness of student models in missing modalities.

[0005] Therefore, there is an urgent need for an innovative approach for MEL tasks that can improve the semantic discrimination ability and retrieval stability of the model in the case of missing visual modalities through an efficient knowledge transfer mechanism, without the need to fill in missing data or modify the model architecture. Summary of the Invention

[0006] This invention provides a cross-view reflective sorting distillation method, device, and medium for entity knowledge indexing, which aims to solve the problem of decreased retrieval performance caused by insufficient semantic distinguishability in multimodal entity links when visual modalities are missing.

[0007] To achieve the above objectives, the first aspect of the present invention provides a cross-view reflective ranking distillation method for entity knowledge indexing, comprising the following steps:

[0008] The teacher model is trained based on complete multimodal data, which includes mention-entity pairs in both textual and visual modalities;

[0009] By masking the visual modalities in the complete multimodal data, incomplete mention data containing only the text modalities is generated;

[0010] Construct a student model, which uses the same network architecture as the teacher model and is initialized based on the parameters of the teacher model;

[0011] The complete multimodal data is input into the teacher model, while the incomplete mention data is input into the student model;

[0012] Based on the mention-entity similarity output by the teacher model, the first ranking distribution of entities in the full modality is generated;

[0013] Based on the mention-entity similarity output by the student model, a second ranking distribution of entities under the incomplete modality is generated;

[0014] Define a sort alignment loss function, and minimize the difference between the first sort distribution and the second sort distribution using a differentiable strategy;

[0015] Based on the entity-mention similarity of the teacher model, a cross-view ranking distribution is generated, and a cross-view distillation loss function is defined to align the cross-view ranking distribution of the student model.

[0016] Define a self-consistency loss function to force the mention-entity ranking distribution of the student model to be consistent with its reverse-generated entity-mention ranking distribution;

[0017] The sorting alignment loss function, cross-view distillation loss function, and self-consistency loss function are jointly optimized to update the parameters of the student model, thus obtaining the trained student model.

[0018] Deploy the trained student model to handle mention-entity linking tasks with missing visual modalities.

[0019] Furthermore, the method for minimizing the difference between the first ranking distribution and the second ranking distribution using the differentiable strategy includes:

[0020] The pairwise similarity difference between the teacher model and the student model is approximated as a continuous value using the hyperbolic tangent function to generate a differentiable comparison matrix.

[0021] Based on the comparison matrix, the normalized difference between the first sorting distribution and the second sorting distribution is calculated;

[0022] The parameters of the student model are updated by minimizing the normalized difference value.

[0023] Furthermore, the alignment method for the cross-view sorting distribution includes:

[0024] Based on the entity-mention similarity output by the teacher model, the cross-view ranking distribution of the entity-mention in the full modality is generated;

[0025] Based on the entity-mention similarity output by the student model, a cross-view ranking distribution of students from entities to mentions is generated under incomplete modality;

[0026] Calculate the cross-entropy loss between the teacher cross-view ranking distribution and the student cross-view ranking distribution, and generate the cross-view distillation loss function;

[0027] The parameters of the student model are adjusted to align with the cross-view ranking distribution of the student model by minimizing the cross-view distillation loss function using the backpropagation algorithm.

[0028] Furthermore, the calculation of the self-consistency loss function includes the following operations:

[0029] Extract the mention-entity similarity ranking distribution and the entity-mention similarity ranking distribution of the student model;

[0030] Calculate the absolute difference between each entity in the two sorting distributions;

[0031] The average of the absolute differences among all entities is used to generate the self-consistency loss value.

[0032] Furthermore, the specific operations of the joint optimization include:

[0033] The sorting alignment loss function, cross-view distillation loss function, and self-consistency loss function are weighted and summed according to a preset weight ratio to generate the total loss function;

[0034] The gradient of the total loss function with respect to the student model parameters is calculated using the backpropagation algorithm.

[0035] The parameters of the student model are updated using an adaptive optimizer, with the optimization objective being to minimize the total loss function.

[0036] Furthermore, the method for generating incomplete mention data containing only text modalities by masking the visual modalities in the complete multimodal data includes replacing the visual modal features in the complete multimodal data with zero vectors.

[0037] Furthermore, the training method for the teacher model includes:

[0038] Define a contrastive learning loss function, which includes:

[0039] Maximize the similarity score of correct mentions-entity pairs;

[0040] Apply a negative sample penalty term to the similarity score of the incorrect mention-entity pair, making it approach zero;

[0041] The parameters of the teacher model are iteratively optimized using the stochastic gradient descent algorithm until the contrastive learning loss function converges.

[0042] Furthermore, the parameter initialization of the student model includes: copying the network weights of the teacher model and freezing the parameters of the teacher model during training.

[0043] To achieve the above objectives, a second aspect of the present invention provides an electronic device including a memory and a processor, the memory being used to store a program supporting the processor in executing a cross-view reflective sorting distillation method for entity knowledge indexing, the processor being configured to execute the program stored in the memory.

[0044] To achieve the above objectives, a third aspect of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, performs the steps of the cross-view reflective sorting distillation method for entity knowledge indexing.

[0045] The beneficial effects of this invention are:

[0046] Compared with existing technologies, this invention provides a cross-view reflective ranking distillation method, device, and medium for entity knowledge indexing. By constructing a teacher-student distillation framework, it transfers the semantic discriminative ability of a teacher model trained on complete multimodal data to a student model that only processes text modality. Unlike traditional knowledge distillation, which only aligns the output probability distribution, this cross-view reflective ranking distillation method for entity knowledge indexing introduces missing-aware ranking distillation. It dynamically aligns the similarity ranking distribution of candidate entities between the teacher model (complete modality) and the student model (missing modality) through a differentiable strategy. This process no longer forcibly constrains absolute similarity values ​​but focuses on the relative ranking relationship between entities within a batch, thereby effectively conveying the key contribution of visual modality to semantic discriminability and avoiding the semantic distribution collapse problem caused by modality missingness. Secondly, addressing the challenge of modal asymmetry between mention-side and entity-side in open environments, the cross-view reflective ranking distillation method for entity knowledge indexing proposes cross-view distillation. This method not only distills ranking knowledge from mention to entity but also constructs cross-view supervision through reverse retrieval from entity to mention. This forces the student model to still bidirectionally understand multimodal semantic associations even when visual information is missing, significantly alleviating the structural bias between single-modal input and multimodal entity matching. Furthermore, through a self-consistent loss function, the student model's mention-to-entity ranking distribution is forced to maintain intrinsic consistency with the mention-to-entity ranking distribution, further constraining the logical consistency of the model's reasoning under missing modalities and enhancing its robustness. Finally, by jointly optimizing ranking alignment, cross-view distillation, and self-consistent loss, a multi-level supervision is formed, enabling the student model to fully inherit the multimodal semantic discrimination capability of the teacher model without relying on visual completion or modifying the architecture. Compared with existing methods, the cross-view reflective ranking distillation method for entity knowledge indexing solves the problem that traditional knowledge distillation cannot transmit semantic discriminability through a ranking-driven rather than probabilistic matching knowledge transfer mechanism. Through cross-view and introspection mechanisms, it breaks through the limitations of unidirectional retrieval, significantly improves the robustness and generalization of the model in open environments, and maintains model independence and computational efficiency. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0048] Figure 1 This is a flowchart of a cross-view reflective sorting distillation method for entity knowledge indexing disclosed in an embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of a CVDI disclosed in an embodiment of the present invention.

[0050] Figure 3 This is a bar chart comparing the performance of different parameter configurations under conditions of complete modal missing and no missing modes, as disclosed in an embodiment of the present invention.

[0051] Figure 4 This is a two-dimensional thermogram showing the effect of a parameter combination on H@1 and MRR of a model under complete modal loss, as disclosed in an embodiment of the present invention.

[0052] Figure 5 This is a comparison chart of five different missing data rate evaluation metrics across datasets disclosed in an embodiment of the present invention. Detailed Implementation

[0053] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0054] According to embodiments of the present invention, it should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the following manufacturing method, in some cases the steps shown or described may be performed in a different order than that shown here.

[0055] like Figure 1 As shown, this invention provides a cross-view reflective ranking distillation method for entity knowledge indexing, comprising the following steps:

[0056] Step S100: Train a teacher model based on complete multimodal data, wherein the complete multimodal data includes mention-entity pairs of textual and visual modalities;

[0057] Textual modalities refer to the textual descriptions or contextual information associated with entities (e.g., names of people, attribute descriptions, event narratives), while visual modalities refer to the images or visual features corresponding to entities (e.g., portraits of people, images of objects, photographs of scenes). Each mention-entity pair is strictly aligned in the knowledge base, ensuring semantic consistency between the textual description and visual representation of the same entity. For example, for the entity "Eiffel Tower," the textual modality might include descriptions of its historical background and architectural features, while the visual modality corresponds to a panoramic photograph or a detailed image of its structure. This multimodal aligned data structure enables the teacher model to learn the complementary associations between textual and visual information.

[0058] During training, the teacher model first extracts textual and visual features using a multimodal encoder. The textual modality is typically encoded using a pre-trained language model, transforming the textual description into a high-dimensional semantic vector; the visual modality utilizes a convolutional neural network to extract image features, capturing visual cues such as spatial structure, color, and texture. Subsequently, multimodal fusion strategies (such as gated fusion, attention mechanisms, or cross-modal Transformers) map both types of features to a unified semantic space. For example, the gated fusion module dynamically adjusts the weights of textual and visual features, allowing visual features to dominate semantic discrimination when the text is ambiguous (e.g., "apple" could refer to a fruit or a company); while when image quality is low (e.g., blurry or occluded), textual features enhance the reliability of inference. This process enables the teacher model to fully leverage the synergistic effects of multimodal information, improving the accuracy and robustness of entity linking.

[0059] The training of the teacher model relies on a contrastive learning loss function, the core objective of which is to maximize the similarity score of correct mention-entity pairs and impose a negative sample penalty term on the similarity score of incorrect mention-entity pairs to make it approach zero. The parameters of the teacher model are iteratively optimized using a stochastic gradient descent algorithm until the contrastive learning loss function converges.

[0060] Step S200: By masking the visual modalities in the complete multimodal data, incomplete mention data containing only text modalities is generated;

[0061] By replacing visual modal features (such as image embedding vectors) in complete multimodal data with zero vectors of the same dimension, the contribution of visual information is completely eliminated. This method directly reflects situations in real-world scenarios where images cannot be acquired due to user privacy restrictions, device acquisition failures, or operating environment limitations.

[0062] Zero vector substitution is an extreme but controllable masking strategy that accurately simulates input conditions where the visual modality is completely absent. By setting visual features to zero, the model is forced to rely solely on the textual modality for entity linking inference during training, thereby enhancing the semantic expressive power of textual features. For example, when mentioning "apple," the model cannot obtain any image cues (such as a company logo or the appearance of the fruit) from the zero-vectorized visual features and must rely entirely on the textual description (such as "technology company" or "fruit") for ambiguity resolution. This process effectively simulates the modal asymmetry between user input and knowledge base entities in an open environment.

[0063] Through zero-vector substitution, the original complete multimodal data is reconstructed into mention-entity pairs containing only the text modality. Notably, the entity-side multimodal data (i.e., entity descriptions and images in the knowledge base) remains intact to ensure the student model can fully utilize the semantic information of multimodal entities during retrieval. For example, the student model needs to match entities in the knowledge base that contain both text and images based on the text mention "Washington." This input asymmetry forces the model to learn to extract semantic features aligned with multimodal entities from the text, even in the absence of visual information, thereby improving robustness in open environments.

[0064] Step S300: Construct a student model, which adopts the same network architecture as the teacher model and is initialized based on the parameters of the teacher model;

[0065] The student model copies all network weights (such as the encoder, fusion module, and similarity calculation layer parameters) from the teacher model trained in the first stage, ensuring that its initial state inherits the semantic representation capabilities learned by the teacher model under multimodal complete data. During subsequent training, the teacher model's parameters are frozen (i.e., they do not participate in gradient updates), while the student model's parameters are dynamically adjusted through backpropagation. This design, based on a self-distillation framework, aims to leverage the teacher model's strong discriminative ability on complete modal data to guide the student model to achieve robust entity linking performance on incomplete modal data (text input only), while avoiding transfer efficiency loss due to differences in network structure.

[0066] Step S400: Input the complete multimodal data into the teacher model, and simultaneously input the incomplete mention data into the student model;

[0067] Specifically, the teacher model receives mention-entity pairs containing both textual and visual modalities (such as "Eiffel Tower" and its corresponding image), and generates multimodal semantic representations and entity similarity scores through forward computation. The student model receives mention data with the visual modal replaced by zero vectors (such as retaining only the text "Eiffel Tower"), and performs inference under the same network architecture. This stage involves parallel input of both modalities to construct a response comparison between the teacher model (complete modality) and the student model (missing modality), ensuring that the student model inherits the multimodal semantic discriminative ability from the teacher model without modifying the architecture.

[0068] Step S500: Based on the mention-entity similarity output by the teacher model, generate the first ranking distribution of entities in the full modality;

[0069] When processing complete multimodal data (text + visual), the teacher model calculates similarity scores between the mention and candidate entities in the knowledge base through a contrastive learning mechanism, and ranks the candidate entities based on the scores. For example, given the mention "apple", the teacher model combines the text description ("technology company") and the image (company logo) to output the matching scores of each candidate entity (such as "Apple Inc." "fruit apple"), generating a list of entities sorted in descending order of similarity.

[0070] Step S600: Based on the mention-entity similarity output by the student model, generate a second ranking distribution of entities under the incomplete modality;

[0071] The student model receives text-only mention data (e.g., the text "apple," with the visual modality replaced by a zero vector), calculates matching scores with candidate entities using the same network architecture, and generates ranking results. For example, if the student model relies solely on the textual description "apple" to infer semantics, it might incorrectly overestimate the score for "fruit apple," leading to a deviation in the ranking distribution from the teacher model. This step quantifies the difference in semantic discriminative ability between the student model and the teacher model by comparing the first ranking distribution (complete modality) with the second ranking distribution (missing modality).

[0072] Step S700: Define a sorting alignment loss function and minimize the difference between the first sorting distribution and the second sorting distribution using a differentiable strategy;

[0073] By defining a ranking alignment loss function (MARD), the difference between the complete modality ranking distribution of the teacher model (first ranking distribution) and the incomplete modality ranking distribution of the student model (second ranking distribution) is minimized using a differentiable strategy. Specifically, a hyperbolic tangent function is used to continuously approximate the pairwise similarity difference between the teacher and student models, generating a differentiable comparison matrix, thereby transforming the discrete ranking relationship into an optimizable numerical form. By calculating the normalized difference between the two ranking distributions and backpropagating gradients to update the student model parameters, the student is forced to inherit the teacher model's ability to discriminate semantic differences between entities even when the visual modality is missing, ultimately improving the ranking consistency of the retrieval results.

[0074] Step S800: Based on the entity-mention similarity of the teacher model, generate a cross-view ranking distribution and define a cross-view distillation loss function to align the cross-view ranking distribution of the student model;

[0075] Cross-view distillation (CVDI) aligns the entity-mention ranking distributions of the teacher and student models to mitigate asymmetric retrieval bias caused by modality loss. Specifically, the teacher model calculates entity-to-mention (E2M) similarity based on complete multimodal data (e.g., the matching score between the entity "Apple Inc." in the knowledge base and the user-inputted mention "Apple"), generating a cross-view ranking distribution under the complete modality. The student model, on the other hand, generates a corresponding cross-view ranking distribution based on incomplete mention data (text only). By calculating the cross-entropy loss of the two distributions, backpropagation adjusts the student model parameters, forcing it to inherit the teacher's multimodal discriminative ability in the reverse entity-to-mention retrieval. For example, the teacher model, combining the company logo image with the text "Apple Inc.", achieves a higher matching score for the mention "Apple" in the E2M direction, while the student model needs to rely solely on text for similarity ranking. This step, through bidirectional retrieval supervision (M2E and E2M), enhances the model's adaptability to modality loss scenarios, overcoming the limitations of unidirectional links.

[0076] Step S900: Define a self-consistency loss function to force the mention-entity ranking distribution of the student model to be consistent with its reverse-generated entity-mention ranking distribution;

[0077] Specifically, similarity ranking distributions for the M2E (entity retrieval based on text mentions) and E2M (entity retrieval based on text mentions) directions are extracted from the student model. The absolute difference between each entity in the two distributions is calculated, and the mean is used to generate a self-consistency loss value. This loss function forces the student model to maintain logical consistency in bidirectional retrieval under unimodal input. For example, if the mention "apple" highly matches the entity "Apple Inc." in the M2E direction, then this entity should also highly match the original mention in the E2M direction, avoiding inference contradictions caused by modality loss. Through introspective ranking distillation (IRD), the model can self-correct inconsistencies in cross-view retrieval, thereby improving the robustness of entity links in open environments.

[0078] Step S1000: Jointly optimize the sorting alignment loss function, cross-view distillation loss function, and self-consistency loss function, update the parameters of the student model, and obtain the trained student model;

[0079] Specifically, the total loss function is defined as a weighted sum of the three parameters. An adaptive optimizer (such as AdamW) is used to calculate the gradient of the total loss with respect to the parameters, and the student model is iteratively updated. The optimization objective is to minimize the comprehensive loss under multi-level supervision. This process enables the student model to inherit the teacher's multimodal discrimination ability when visual modalities are missing, while maintaining the logical consistency of cross-view retrieval, ultimately achieving improved robustness.

[0080] Step S1100: Deploy the trained student model to handle the mention-entity linking task with missing visual modalities.

[0081] During the inference phase, the visual modalities mentioned in the input are automatically replaced with zero vectors (as described in step S200), retaining only the text description (such as "apple"). The student model calculates the similarity between the input text and multimodal entities in the knowledge base and outputs a Top-K candidate entity list. For example, when a user inputs the mention of "Washington" with a missing image, the model relies on text features to retrieve entities such as "George Washington" (person) or "Washington State" (place name), and combines multimodal information (text + image) from the entity side to optimize the ranking results.

[0082] As described in steps S100-S1100 above, it is understood that the main workflow of the proposed Cross-View Reflective Ranking Distillation (CVIRD) method for entity knowledge indexing is based on the knowledge distillation (KD) paradigm to enhance missing modalities. Specifically, the teacher and student networks employ the same model architecture but have different learning parameters in the self-distillation process, which comprises two training phases. In the first phase, the teacher follows a traditional training strategy, training the model using complete modality samples to fully acquire multimodal representations. Given a training batch, a contrastive loss function is used as the task loss function, prompting the model to give higher matching scores for correct mention-entity pairs and lower scores for incorrect mention-entity pairs. For each training batch of mention-entity pairs... The loss function for the MEL task can be expressed as follows:

[0083] (1)

[0084] in, For the multimodal entity linking (MEL) task, it is a contrastive loss function used to measure the model's ability to correctly match mentions with entities during training; This is the set of mentions in the current batch, where each mention consists of text and possible visual modalities (such as images). It is a collection of entities in a knowledge base, where each entity contains a text description and visual information; These are the learnable parameters of the model, i.e., the weights that need to be adjusted through optimization; The size of the training batch, i.e., the number of mention-entity pairs contained in each batch; For indexing each sample in the batch Perform a traversal and summation; References to model calculations Its corresponding correct entity The similarity score is determined by the parameter. Decide; To exclude currently correct entity pairs during summation (i.e. j=i In the case of [the negative sample], only the similarity of other negative samples is calculated. Essentially, it is a filtering condition used to exclude correct entity pairs from the denominator and retain only negative samples for calculation.

[0085] Regarding the second stage, given a training mention Visual features Replace with a zero vector to form an incomplete mention sample. This simulates the modality loss situation in open environment testing. Specifically, before the second stage of training, the student model is copied from the teacher model frozen in the first stage, with the same parameters and network structure. Then, using the complete... and incomplete The mentioned samples and their corresponding correct entities are used as input to train the student network using a task loss function. Simultaneously, the teacher model processes data with complete modalities and provides supervision to the student model, which processes data with incomplete modalities. The loss function of the student network in this stage is... for:

[0086] (2)

[0087] in, and These represent the parameters of the trainable student model and the frozen teacher model, respectively. This indicates a missing visual modality (e.g., replacing an image portion with a zero vector). This represents the candidate entity similarity score output by the student model when processing visual missing mentions; Indicates an incomplete mention; This represents the candidate entity similarity score output by the teacher model when processing a complete mention; This represents the knowledge distillation loss. It can be explicitly stated as:

[0088] (3)

[0089] in, Indicates the distillation temperature. Indicates the Kullback-Leibler divergence. This represents the softmax function; Represents the student model for the first The similarity score vector of all candidate entities in a sample; The teacher model represents the first The similarity score vector of all candidate entities for each sample. Through the knowledge distillation paradigm, the student network can achieve robust retrieval performance by mimicking the linking behavior of the teacher model in handling multimodal data, even with only unimodal input. During inference, samples are only input into the student network for downstream tasks.

[0090] While knowledge distillation (KD) enables student networks to perform multimodal reasoning using only textual information, its application in the MEL-MM task remains significantly limited. Semantic distinguishability is a significant contribution of visual modalities to the MEL task. Specifically, accompanied by images, multimodal mentions exhibit stronger semantic diversity in the database compared to unrelated entities, which helps to make the data distribution around each mention more dispersed. However, KL divergence in the KD process minimizes the difference between teachers and students by normalizing the matching score based on the response to each mention. This supervision fails to effectively convey the semantic diversity between mentions and different candidate entities, which is possessed by well-performing teachers. Furthermore, the inherent information imbalance between teacher and student models inevitably increases the difficulty of KD due to the knowledge transfer from multimodal to text-only data on the mention side. Therefore, the overly strict absolute similarity constraints in traditional KD may not be suitable for the MEL-MM task.

[0091] In this embodiment, the proposed CVIRD method introduces Missing-Aware Ranking Distillation (MARD), which aims to deliver a mention-centric similarity ranking to enhance distinguishability. Specifically, given a mention... Similarity centered on mentions is defined as The matching scores of entities within batches in both the complete and incomplete modal teacher and student models can be expressed as:

[0092] and (4)

[0093] in, This indicates that the teacher model is for complete mentions. and the j individual entities The matching score; This indicates that the student model addresses incomplete mentions. and the j individual entities Match score, j =1,2,..., ;

[0094] Then, unlike traditional KD which directly uses similarity for supervision, MARD aims to calculate their similarity in... and The ranking order within. This encourages distillation processes to be better based on given content (e.g., This model models the semantic variations of candidate entities within a batch, thereby improving the discriminability of students trained on incomplete data. MARD first constructs priority comparison matrices for both teachers and students. Taking the teacher's response as an example, we define... for:

[0095] (5)

[0096] in, For priority comparison matrix, It is a pairwise contrast function used to compare the significance of two similarities between mentions from two different entities. For example, This achieves the implementation of the mention of a given number in the training batch up to the first number. The entity and the first The similarity comparison result of several entities can be expressed by the function as:

[0097] (6)

[0098] in, and This indicates that the teacher model (complete modality) represents the first term within the batch. The entity and the first Matching score of each entity.

[0099] The first batch Reference-centric similarity ranking of individual entities It can be represented as:

[0100] (7)

[0101] Similarly, the ranking of students under modality-deficient mentions can be calculated using formula (5-7). Then, a basic ranking distillation loss can be formulated. The consistency between mention-centric similarity rankings of normalized teacher and student networks can be expressed as:

[0102] (8)

[0103] in, The teacher model (complete modality) represents the first... i One mention and the j individual entities The matching ranking.

[0104] However, in MEL-MM, it is pointed out that... There are still two significant limitations. The first and main challenge lies in the pairwise contrast function. The discontinuity of this property conflicts with optimization methods in deep learning. To facilitate gradient backpropagation in the MEL network, the hyperbolic tangent function is utilized. To approximate the pairwise comparison function :

[0105] (9)

[0106] in, Represents a smooth approximation function; and The teacher model represents the first j The and the first k The matching score of each entity comes from the similarity vector of the teacher model; This represents the hyperbolic tangent function.

[0107] Replacing formula (6) with formula (9), the final approximate mention-centric similarity ranking is expressed as follows: :

[0108] (10)

[0109] Regarding the second defect, let's assume the basic... The proposed algorithm fails to adequately consider ranking distillation in modality-deficient scenarios. Specifically, when the mention-side input changes from multimodal (teacher) to text-only (student) data, mention-centric similarity between entities within different batches typically varies. Entities with significant similarity gaps between teacher and student responses during retrieval are unstable candidate entities and sensitive to image deficiencies. Therefore, a deficiency-aware scaling strategy is introduced to dynamically adjust the ranking distillation for modality-deficient scenarios. The ranking of different candidate entities is regularized and training weights are assigned. Specifically, the ranking of the first candidate entity is regularized and training weights are assigned. Missing entity perception weight It can be defined as:

[0110] (11)

[0111] After overcoming the two limitations mentioned above, the improvements in formulas (10) and (11) can be integrated into formula (8), and the final loss function of the proposed MARD is expressed as:

[0112] (12)

[0113] in, i and j Used as a loop variable to iterate through all entity pairs within the batch; These are weighting coefficients used to adjust the contribution of differences between different entity pairs to the loss function; For the teacher model (complete modality) to the first i The first mention and the first j The target matching value for each entity; For the student model (missing modality) for the 1st i The first mention and the first j The predicted matching value for each entity.

[0114] When the image input on the mention side is missing, the linking process becomes asymmetrical. The MEL algorithm always retrieves multimodal entities based on plain text mentions, which may compromise its robustness and semantic distinguishability in information retrieval. To address this, Cross-View Distillation with Introspection (CVDI) is proposed to mitigate the asymmetry in two-layer linking. Specifically, in addition to passing mention-centric similarity between entities in a batch of training, CVDI... CVDI also provides a cross-view approach to distill the similarity of entity centers into batch mentions. .like Figure 2 As shown, it does not link from unimodal to multimodal data, but rather serves as supplementary knowledge, empowering students to retrieve unimodal candidate entities from multimodal inputs, with the expected output being... Modally complete Supervision. Based on the proposed MARD, the loss function for cross-view distillation (CVDI) from entity to mention in CVDI is:

[0115] (13)

[0116] in, , and They are based on and The missing perception weights and similarity rankings are calculated. Furthermore, based on the cross-view distillation process, CVDI proposes to enhance the modal missing robustness of asymmetric students through self-introspection. Specifically, the mention-centric similarity rankings generated by students should be consistent with their own entity-centric similarity rankings to achieve self-consistency. Introspective ranking distillation (IRD) in CVDI can be defined as:

[0117] (14)

[0118] Based on the key concepts MARD and CVDI mentioned above, the second-stage loss function of the student network (Equation (3)) can be restated in the knowledge distillation paradigm as follows:

[0119] (15)

[0120] (16)

[0121] in, It is the final distillation function of the proposed CVIRD method. and It is a compromise factor for weighing the contributions of distillation from different perspectives; This represents the ranking difference loss from modality to entity, measuring the consistency between the teacher model (complete modality) and the student model (missing modality) in entity matching ranking; The ranking difference loss from noise to modality is represented. Inference is performed using only the student model trained in the second stage for link evaluation, where the visual embeddings of the missing modalities in the test mentions are filled with zero vectors of the same dimension. Notably, as a KD-based method, CVIRD requires no additional model parameters or special structural design, offering flexibility and model independence, and can be applied to any MEL algorithm to improve robustness in the case of missing visual modalities.

[0122] To verify the effectiveness of the proposed CVIRD method, the following systematic experiments were conducted on two public multimodal entity linking (MEL) datasets: WikiMel and RichpediaMel.

[0123] For the MEL data, two commonly used MEL datasets, WikiMEL and RichpediaMEL, were used to evaluate the effectiveness of the proposed method. These datasets primarily contain entities of the person / entity type. WikiMEL contains over 22,000 multimodal sentences from Wikipedia. RichpediaMEL, derived from MMKG Richpedia, involves entities extracted from Richpedia and is supplemented with multimodal data from Wikidata. A subset of the Wikidata knowledge graph was used for each dataset while maintaining the original data partitioning. Specifically, the training-validation-test set ratio for both the WikiMEL and RichpediaMEL datasets was 70% - 10% - 20%. During evaluation, the similarity between mentions and all entities in the knowledge graph was measured to assess their alignment probability. Two key metrics were used in the experiments: 1) H@k refers to the hit rate of ground truth (GT) entities among the top k entities; 2) MRR represents the average reciprocal rank of ground truth entities.

[0124] For the MEL network architecture, the state-of-the-art MIMIC was used as the backbone network for most experimental results, while GHMFC and CLIP models were also involved to evaluate model independence. For comparative methods targeting the missing modality problem, the baseline was the MEL model trained using only the traditional task loss (Equation (2)). Other competitive methods included: 1) Combinatorial-based methods: DDMI and RMM attempt to borrow or blend images from complete samples of other modalities to the missing visual modality, requiring additional memory banks; 2) Generative-based methods: CASAE trains a generative model using text data to generate images by recovering the missing images at the feature level; 3) Distillation-based methods: PMKL and CorrKD utilize feature and logits-level knowledge transfer, where the student model in the second stage is trained from scratch. For further comparison, the KD paradigm was compared with traditional knowledge distillation (CKD) and a multi-level distillation method called DIST.

[0125] Following MIMIC, all visual and text feature extractors in the referenced MEL backbone network are initialized using a pre-trained CLIP-Vit-Base-Patch32. Optimization is performed using the AdamW optimizer and a batch size of 128. In CVIRD, the MEL model is trained for 20 epochs in the first stage (E1) with a learning rate of 1e-5, and for 10 epochs in the second stage (E2) with the same learning rate. (Trade-off factors are also mentioned.) α and β Simply defined as α=β = 1. All experiments were performed using an Intel(R) Core i9-14900KF CPU and an NVIDIA RTX4090 GPU with 24 GB of memory. As with previous methods, each model trained in the first stage was evaluated using a validation dataset, and the checkpoint that produced the maximum MRR was selected for the second stage of training. The checkpoint of the last epoch of the second stage served as the final missing-resistant MEL model. The missing-aware robustness of the MEL algorithm was evaluated in the MEL-MM setting at various missing percentages of 0%, 25%, 50%, 75%, and 100%.

[0126] Table 1: A comparison of state-of-the-art image missing rates across five different reference sources on the WikiMel and RichpediaMel datasets. Best and second-best data are indicated by bold underlines.

[0127]

[0128] For RQ1, CVIRD is proposed to be compared with other competitive methods for modality missing, with five different scenarios for the proportion of missing images on two benchmark datasets. As shown in Table 1, replacing missing modality data with duplicated or generated samples is an intuitive solution to mitigate the performance degradation caused by uncertain missing data, such as MMDI, RMM, and CASAE. However, these methods involve additional memory or model architecture and are less flexible. From the results of distillation-based methods, it can be noted that self-distillation procedures (e.g., CKD, DIST, and the method of this invention) achieve comparable link performance to alternative methods (e.g., PMKL and CorrKD) with fewer training cycles in the second stage, where the student model in the second stage is trained from scratch. This indicates that initializing the teacher model trained from complete data helps the student accumulate robust multimodal knowledge in the MEL-MM task. Among competitive methods, the CVIRD proposed in this invention achieves significant retrieval performance in the missing data scenario, showing superior performance compared to baseline methods on WikiMEL and RichpediaMEL in cases of complete modality missing data. In the case of 1), Hit@1 was improved by 20.5% and 11.6%, respectively. These findings strongly support the effectiveness of CVIRD in improving the robustness of mention-side deletions in the MEL-MM task.

[0129] For RQ2, the contributions of different concepts in the CVIRD constructed based on the KD paradigm were first verified, and the results are shown in Table 2. The observations can be divided into the following three aspects:

[0130] 1. As can be seen from lines 1-4, compared with the traditional KD method (which uses smoothing temperatures T=1, 2, 3 to regularize the absolute matching score), the MARD method of this invention significantly improves retrieval performance in both H@1 and MRR metrics. This is because MARD's ranking mechanism effectively conveys semantic distinguishability through pairwise similarity comparison when establishing the ranking order.

[0131] 2. In MARD and MARD+CVD, missing loss-aware weights W impair link performance because W promotes ordination distillation, enabling it to dynamically focus on entities that are sensitive to missing data and exhibit instability.

[0132] 3. By integrating MARD and CVDI (CVD and IRD), the final CVIRD is able to fully distill mention-centric similarity rankings from cross-view and introspective teacher knowledge. Notably, when the mention data contains only text (ratio=1), CVIRD's H@1 performance surpasses CKD's performance at Ratio=0.5.

[0133] Regarding the hyperparameters, the change in the number of epochs E2 during the second stage of training was first verified, where the number of epochs in the first stage was E1 = 20. For example... Figure 3 As shown, when there is no visual information on the mentioned side (ratio = 1), increasing E2 from 0 to 10 improves the accuracy of MEL, then plateaus. If the ratio = 0, MEL reaches its best performance at E2 = 10, but performance declines after more than a few epochs. The results indicate that the method of this invention is less time-consuming in the second-stage training, unlike other KD-based methods such as PMKL and CorrKD which require training from scratch. The tradeoff factors α and β are dynamically adjusted within a reasonable range [0, 0.5, 1, 2]. Figure 4 It can be seen that they are not sensitive to change, therefore they can be simply defined α=β =1, to achieve the appropriate result in the proposed CVIRD.

[0134] Table 2: Reduction study of different concepts in CVIRD based on H@1 and MRR indices on the WikiMel test set. The traditional KD and the method of this invention respectively set temperature and missing perception weights.

[0135]

[0136] For RQ3, the baseline methods listed in Table 1 and four efficient methods—CASAE, RMM, CKD, and the CVIRD method of this invention—were selected for cross-dataset validation. Specifically, all five methods were trained on the MIMIC backbone network on the WikiMel dataset and evaluated on RichpediaMEL. Figure 5 The results shown can be summarized as follows:

[0137] 1. When there is a discrepancy between the training and test data, even if both databases are related to person information, the accuracy of the MEL model using traditional training methods in retrieving the correct entity will decrease significantly.

[0138] 2. Compared to in-distribution inference, the standard MEL model (baseline model) exhibits greater vulnerability in the case of modal missing values, with H@1 dropping sharply from 65.30 to 22.46. This highlights the urgent need for MEL systems to possess missing value resistance in open-world applications.

[0139] 3. Compared with the other three competing methods, the method of the present invention ranks first in both H@1 and MRR metrics, and the performance degradation is smaller as the missing ratio increases.

[0140] For RQ4, CVIRD enhances resistance to visual modality uncertainty by distilling robust multimodal knowledge into the student model with incomplete data. Therefore, CVIRD should be orthogonal to the model architecture design and can be incorporated into various MEL models. To verify its independence from the backbone network, CVIRD was integrated into three different MEL algorithms listed in Table 3. Although entity retrieval accuracy varied among different MEL networks, the performance of all three methods under standard training (BASE) significantly decreased with increasing missing modality rates. While the two feature recovery-based methods, CASAE and RMM, each have their advantages in different MEL architectures, the knowledge distillation paradigm remains a more efficient approach to mitigating performance degradation caused by missing modalities. The CVIRD method of this invention ranked first in almost all metrics and performed excellently under three different missing modality ratios. This strongly demonstrates the effectiveness of the method in MEL-MM and its great potential for application in various multimodal network architectures.

[0141] Table 3: Evaluation of model differences based on three different MEL models (BASE) and four anti-omission strategies: MIMIC, GHMFC and CLIP.

[0142]

[0143] When the MEL system is deployed in open environments, images on the mention side are prone to being missing. To address this issue, this invention proposes a novel cross-view introspective ranking distillation (CVIRD), which enhances semantic discriminability by introducing two key concepts: Missing-Aware Ranking Distillation (MARD) and Cross-View Distillation with Introspection (CVDI). Considering the ranking factor, which is particularly important in the variation of distance from mentions to candidate entities, MARD is designed to distill ranking knowledge in a missing-aware and differentiable manner. The introduction of CVDI aims to fully extract the discriminability of ranking distillation from both cross-view and introspective perspectives. Extensive experiments validate the effectiveness of this framework in addressing the missing image problem, achieving significantly better performance than competing methods on two widely used datasets.

[0144] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor and a memory, wherein the processor is configured to implement the steps of the method when executing a computer program stored in the memory.

[0145] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0147] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0148] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0149] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A cross-view reflective sorting distillation method for entity knowledge indexing, characterized in that, Includes the following steps: The teacher model is trained based on complete multimodal data, which includes mention-entity pairs in both textual and visual modalities; By masking the visual modalities in the complete multimodal data, incomplete mention data containing only the text modalities is generated; Construct a student model, which uses the same network architecture as the teacher model and is initialized based on the parameters of the teacher model; The complete multimodal data is input into the teacher model, while the incomplete mention data is input into the student model; Based on the mention-entity similarity output by the teacher model, the first ranking distribution of entities in the full modality is generated; Based on the mention-entity similarity output by the student model, a second ranking distribution of entities under the incomplete modality is generated; Define a sort alignment loss function, and minimize the difference between the first sort distribution and the second sort distribution using a differentiable strategy; Based on the entity-mention similarity of the teacher model, a cross-view ranking distribution is generated, and a cross-view distillation loss function is defined to align the cross-view ranking distribution of the student model. Define a self-consistency loss function to force the mention-entity ranking distribution of the student model to be consistent with its reverse-generated entity-mention ranking distribution; The sorting alignment loss function, cross-view distillation loss function, and self-consistency loss function are jointly optimized to update the parameters of the student model, resulting in a trained student model. Deploy the trained student model to handle mention-entity linking tasks with missing visual modalities.

2. The cross-view reflective sorting distillation method for entity knowledge indexing as described in claim 1, characterized in that, The method for minimizing the difference between the first sorting distribution and the second sorting distribution using the differentiable strategy includes: The pairwise similarity difference between the teacher model and the student model is approximated by the hyperbolic tangent function to generate a differentiable comparison matrix. Based on the comparison matrix, the normalized difference between the first sorting distribution and the second sorting distribution is calculated; The parameters of the student model are updated by minimizing the normalized difference value.

3. The cross-view reflective sorting distillation method for entity knowledge indexing as described in claim 1, characterized in that, The alignment method for the cross-view sorting distribution includes: Based on the entity-mention similarity output by the teacher model, the cross-view ranking distribution of the entity-mention in the full modality is generated; Based on the entity-mention similarity output by the student model, a cross-view ranking distribution of students from entities to mentions is generated under incomplete modality; Calculate the cross-entropy loss between the teacher cross-view ranking distribution and the student cross-view ranking distribution, and generate the cross-view distillation loss function; The parameters of the student model are adjusted to align with the cross-view ranking distribution of the student model by minimizing the cross-view distillation loss function using the backpropagation algorithm.

4. The cross-view reflective sorting distillation method for entity knowledge indexing as described in claim 1, characterized in that, The calculation of the self-consistency loss function includes the following operations: Extract the mention-entity similarity ranking distribution and the entity-mention similarity ranking distribution of the student model; Calculate the absolute difference between each entity in the two sorting distributions; The average of the absolute differences among all entities is used to generate the self-consistency loss value.

5. The cross-view reflective sorting distillation method for entity knowledge indexing as described in claim 1, characterized in that, The specific operations of the joint optimization include: The sorting alignment loss function, cross-view distillation loss function, and self-consistency loss function are weighted and summed according to a preset weight ratio to generate the total loss function; The gradient of the total loss function with respect to the student model parameters is calculated using the backpropagation algorithm. The parameters of the student model are updated using an adaptive optimizer, with the optimization objective being to minimize the total loss function.

6. The cross-view reflective sorting distillation method for entity knowledge indexing as described in claim 1, characterized in that, A method for generating incomplete mention data containing only text modalities by masking the visual modalities in the complete multimodal data includes replacing the visual modal features in the complete multimodal data with zero vectors.

7. The cross-view reflective sorting distillation method for entity knowledge indexing as described in claim 1, characterized in that, The training method for the teacher model includes: Define a contrastive learning loss function, which includes: Maximize the similarity score of correct mentions-entity pairs; Apply a negative sample penalty term to the similarity score of the incorrect mention-entity pair, making it approach zero; The parameters of the teacher model are iteratively optimized using the stochastic gradient descent algorithm until the contrastive learning loss function converges.

8. The cross-view reflective sorting distillation method for entity knowledge indexing as described in claim 1, characterized in that, The parameter initialization of the student model includes: copying the network weights of the teacher model and freezing the parameters of the teacher model during training.

9. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the cross-view reflective sorting distillation method for entity knowledge indexing according to any one of claims 1-8, the processor being configured to execute the program stored in the memory.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the cross-view reflective sorting distillation method for entity knowledge indexing according to any one of claims 1-8.

Images