Physical knowledge index cross-view reflection sorting distillation method, equipment and medium

Abstract

The invention relates to a cross-view reflection sorting distillation method and device for entity knowledge indexes and a medium. According to the method, entity link robustness under visual deficiency is improved through sorting knowledge migration. Training a teacher model based on complete multi-modal data, generating plain text data, and inputting the plain text data into a student model (inheriting teacher architecture and parameters); the method specifically comprises the following steps: sorting alignment loss: aligning entity sorting distribution of teachers (complete modals) and students (missing modals) through differentiable sorting, and retaining semantic discrimination; cross-view distillation: enhancing multi-modal representation understanding by using teacher entity-mention reverse sorting; the self-consistency constraint is that student mention-entity sorting and reverse entity-mention sorting are forced to be consistent, and the limitation of one-way retrieval is broken through. And after three-loss joint optimization, deploying a student model which only needs text input. Compared with traditional probabilistic distillation, the missing scene stability is remarkably improved through sorting transmission and a cross-view introspection mechanism, and no extra parameter exists.

Description

Technical Field

[0001] The present invention relates to the field of multimodal machine learning and knowledge graphs, and specifically to a cross-view reflective ranking distillation method, device and medium for entity knowledge indexing. Background Art

[0002] Multimodal Entity Linking (MEL) is a key technology in information retrieval. It aims to associate ambiguous mentions containing both textual and visual information with corresponding entities in a multimodal knowledge base. With the rapid development of social media and knowledge-intensive applications (such as search engines and question-answering systems), MEL effectively alleviates the ambiguity of traditional text entity linking by integrating multimodal information, becoming a key technology for improving content understanding and reasoning capabilities.

[0003] However, existing MEL methods are usually based on the modality completeness assumption, that is, both the training and inference stages assume that the input data contains complete textual and visual information. This assumption has significant limitations in practical applications in the open world. Due to factors such as user privacy protection, device acquisition limitations, or the complexity of the operating environment, input mentions in real-world scenarios may frequently lack the visual modality, causing the MEL system to face the input asymmetry problem (i.e., text-only mentions need to match multimodal entities). Experiments show that when the visual modality is completely missing, the retrieval performance (such as Top1 hit rate) of existing state-of-the-art MEL models may drop by more than 20%, seriously threatening their practicality in open environments.

[0004] To address the problem of missing modalities, various solutions have been proposed in the field of multimodal learning, including data completion based on generative models, designing missing-proof network architectures, or using knowledge distillation to transfer multimodal knowledge. However, these methods are primarily targeted at classification tasks (such as sentiment analysis and medical diagnosis) and suffer from the following shortcomings in MEL tasks based on the retrieval paradigm: 1) Generative methods rely on additional model parameters and have low reliability of completed data; 2) Dedicated architectures are difficult to adapt to different base models and have poor generalization; 3) Traditional knowledge distillation only constrains the output distribution through KL divergence, failing to effectively transfer the key contribution of visual modalities to semantic discriminability, resulting in insufficient robustness of the student model in missing data scenarios.

[0005] Therefore, there is an urgent need for an innovative method for MEL tasks that can improve the semantic discrimination ability and retrieval stability of the model in the absence of visual modalities through an efficient knowledge transfer mechanism without having to fill in missing data or modify the model architecture. Summary of the Invention

[0006] The present invention provides a cross-view reflective ranking distillation method, device and medium for entity knowledge indexing, which aims to solve the problem of degraded retrieval performance of multimodal entity links due to insufficient semantic distinguishability in the absence of visual modality.

[0007] To achieve the above objectives, the first aspect of the present invention provides a cross-view reflective ranking distillation method for entity knowledge indexing, comprising the following steps:

[0008] training a teacher model based on full multimodal data, wherein the full multimodal data includes mention-entity pairs in textual and visual modalities;

[0009] generating incomplete mention data containing only textual modality by masking the visual modality in the complete multimodal data;

[0010] Constructing a student model, wherein the student model and the teacher model use the same network architecture and are initialized based on the parameters of the teacher model;

[0011] Inputting the complete multimodal data into the teacher model and the incomplete mention data into the student model;

[0012] Based on the mention-entity similarity output by the teacher model, generating a first ranked distribution of entities in the full modality;

[0013] generating a second ranked distribution of entities in the incomplete modality based on the mention-entity similarity output by the student model;

[0014] defining a ranking alignment loss function to minimize the difference between the first ranking distribution and the second ranking distribution through a differentiable strategy;

[0015] Based on the entity-mention similarity of the teacher model, generating a cross-view ranking distribution, and defining a cross-view distillation loss function to align the cross-view ranking distribution of the student model;

[0016] Define a self-consistency loss function to enforce that the student model’s mention-entity ranking distribution is consistent with its reverse-generated entity-mention ranking distribution;

[0017] Jointly optimizing the ranking alignment loss function, the cross-view distillation loss function, and the self-consistency loss function, updating the parameters of the student model, and obtaining a trained student model;

[0018] Deploy the trained student model to handle the mention-entity linking task in the absence of visual modality.

[0019] Furthermore, the method of minimizing the difference between the first ranking distribution and the second ranking distribution using the differentiable strategy includes:

[0020] The pairwise similarity differences between the teacher model and the student model are continuously approximated by a hyperbolic tangent function to generate a differentiable contrast relationship matrix;

[0021] Calculating a normalized difference value between the first ranking distribution and the second ranking distribution based on the comparison relationship matrix;

[0022] The parameters of the student model are updated by minimizing the normalized difference value.

[0023] Furthermore, the cross-view sorting distribution alignment method includes:

[0024] Based on the entity-mention similarity output by the teacher model, generate a teacher cross-view ranking distribution of entities to mentions in the full modality;

[0025] Based on the entity-mention similarity output by the student model, generating a student cross-view ranking distribution of entities to mentions in the incomplete modality;

[0026] Calculating the cross-entropy loss between the teacher's cross-view ranking distribution and the student's cross-view ranking distribution to generate a cross-view distillation loss function;

[0027] The cross-view distillation loss function is minimized by a back-propagation algorithm, and the parameters of the student model are adjusted to align the cross-view ranking distribution of the student model.

[0028] Furthermore, the calculation of the self-consistency loss function includes the following operations:

[0029] Extracting the mention-entity similarity ranking distribution and the entity-mention similarity ranking distribution of the student model;

[0030] Calculate the absolute difference between the two sorted distributions for each entity;

[0031] The absolute differences of all entities are averaged to generate the self-consistency loss.

[0032] Furthermore, the specific operations of the joint optimization include:

[0033] The ranking alignment loss function, the cross-view distillation loss function and the self-consistency loss function are weighted and summed according to a preset weight ratio to generate a total loss function;

[0034] Calculate the gradient of the total loss function with respect to the student model parameters by a back-propagation algorithm;

[0035] An adaptive optimizer is used to update the parameters of the student model, and the optimization goal is to minimize the total loss function.

[0036] Furthermore, the method of generating incomplete mention data containing only text modality by masking the visual modality in the complete multimodal data includes: replacing the visual modality features in the complete multimodal data with a zero vector.

[0037] Furthermore, the training method of the teacher model includes:

[0038] Define a contrastive learning loss function, which includes:

[0039] Maximize the similarity score of correct mention-entity pairs;

[0040] Apply a negative sample penalty to the similarity score of the wrong mention-entity pair to make it approach zero;

[0041] The parameters of the teacher model are iteratively optimized by a stochastic gradient descent algorithm until the contrastive learning loss function converges.

[0042] Furthermore, the parameter initialization of the student model includes: copying the network weights of the teacher model and freezing the parameters of the teacher model during the training process.

[0043] To achieve the above-mentioned purpose, the second aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the cross-view reflective ranking distillation method of the entity knowledge index, and the processor is configured to execute the program stored in the memory.

[0044] To achieve the above-mentioned objectives, the third aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, the steps of the cross-view reflective ranking distillation method of the entity knowledge index are executed.

[0045] Beneficial effects of the present invention:

[0046] Compared with the prior art, the entity knowledge index cross-view reflection ranking distillation method, device and medium provided by the present application can effectively transfer the key contribution of the visual modality to semantic distinguishability by constructing a teacher-student distillation framework and migrating the semantic discriminability of the teacher model trained by complete multi-modal data to the student model processing only the text modality. Unlike the traditional knowledge distillation that only aligns the output probability distribution, the entity knowledge index cross-view reflection ranking distillation method introduces missing perception ranking distillation to dynamically align the similarity ranking distribution of the candidate entities of the teacher model (complete modality) and the student model (missing modality) by a differentiable strategy. This process no longer strictly constrains the absolute similarity value, but focuses on the relative ranking relationship between entities in the batch, thereby effectively transferring the key contribution of the visual modality to semantic distinguishability and avoiding the semantic distribution collapse problem caused by the missing of the modality. Secondly, to address the challenge of the asymmetry between the mention side and the entity side modality in an open environment, the entity knowledge index cross-view reflection ranking distillation method proposes cross-view distillation, which not only distills the ranking knowledge from the mention to the entity, but also constructs cross-view supervision through the reverse retrieval from the entity to the mention, forcing the student model to still be able to bidirectionally understand the multi-modal semantic association when the visual information is missing, thereby significantly alleviating the structural bias of the single-modal input and multi-modal entity matching. In addition, by using a self-consistency loss function, the student model is forced to keep the internal consistency between the mention-to-entity ranking distribution and the entity-to-mention ranking distribution, further constraining the logical self-consistency of the model reasoning under the missing modality and enhancing the anti-interference ability. Finally, the ranking alignment, cross-view distillation and self-consistency loss are jointly optimized to form multi-level supervision, so that the student model can fully inherit the multi-modal semantic discriminability of the teacher model without relying on visual completion and without modifying the architecture. Compared with the existing methods, the entity knowledge index cross-view reflection ranking distillation method solves the problem that the traditional knowledge distillation cannot transfer the semantic distinguishability by using a ranking-driven rather than probability-matching knowledge transfer mechanism; by using the cross-view and self-reflection mechanisms, the limitations of one-way retrieval are broken through, the robustness and generalization of the model in an open environment are significantly improved, and the model independence and computational efficiency are maintained. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows.

[0048] Figure 1 is a flowchart of an entity knowledge index cross-view reflection ranking distillation method disclosed by the embodiments of the present application.

[0049] Figure 2 is a schematic diagram of the comparison between the traditional method and the new method in knowledge distillation disclosed by the embodiments of the present application.

[0050] Figure 3 is a schematic diagram of the comparison between the traditional method and the new method in knowledge distillation disclosed by the embodiments of the present application.

[0051] Figure 4 This is a schematic diagram comparing the performance indicators of the multimodal data processing disclosed in an embodiment of the present invention under closed environment training, open environment reasoning and different missing rates.

[0052] Figure 5 It is a schematic diagram of a CVDI disclosed in an embodiment of the present invention.

[0053] Figure 6 This is a bar chart comparing the performance of different parameter configurations under complete mode loss and no mode loss, as disclosed in an embodiment of the present invention.

[0054] Figure 7 It is a two-dimensional heat map of the impact of a parameter combination disclosed in an embodiment of the present invention on the H@1 and MRR of the model under complete mode loss.

[0055] Figure 8 This is a comparison chart of five cross-dataset evaluation indicators with different missing rates disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0057] According to an embodiment of the present invention, it should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the following production method, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0058] like Figure 1 As shown, the present invention provides a cross-view reflective ranking distillation method for entity knowledge indexing, comprising the following steps:

[0059] Step S100: training a teacher model based on complete multimodal data, wherein the complete multimodal data includes mention-entity pairs in textual and visual modalities;

[0060] The textual modality refers to the textual description or contextual information associated with an entity (such as a person's name, attribute description, event narrative, etc.), while the visual modality refers to the image or visual features corresponding to the entity (such as a person's portrait, an object image, a scene photo, etc.). Each mention-entity pair is strictly aligned in the knowledge base to ensure that the textual description and visual representation of the same entity are semantically consistent. For example, for the entity "Eiffel Tower", the textual modality may include descriptions of its historical background, architectural features, etc., while the visual modality corresponds to its panoramic photos or images of structural details. This multimodal aligned data structure enables the teacher model to learn the complementary associations between text and visual information.

[0061] During the training process, the teacher model first extracts text and visual features through a multimodal encoder. The text modality is usually encoded using a pre-trained language model to convert the text description into a high-dimensional semantic vector; the visual modality uses a convolutional neural network to extract image features and capture visual clues such as spatial structure, color texture, etc. Subsequently, the two types of features are mapped to a unified semantic space through a multimodal fusion strategy (such as gated fusion, attention mechanism or cross-modal Transformer). For example, the gated fusion module dynamically adjusts the weights of text and visual features so that when the text is ambiguous (such as "apple" may refer to fruit or company), visual features can dominate semantic discrimination; while when the image quality is low (such as blur or occlusion), text features enhance the reliability of reasoning. This process enables the teacher model to fully tap the synergy of multimodal information and improve the accuracy and robustness of entity linking.

[0062] The training of the teacher model relies on the contrastive learning loss function, whose core goal is to maximize the similarity score of correct mention-entity pairs and impose a negative sample penalty on the similarity score of incorrect mention-entity pairs to make it approach zero; the parameters of the teacher model are iteratively optimized through the stochastic gradient descent algorithm until the contrastive learning loss function converges.

[0063] Step S200: generating incomplete mention data containing only textual modality by masking the visual modality in the complete multimodal data;

[0064] The visual modality features (e.g., image embedding vectors) in the complete multimodal data are replaced with zero vectors of the same dimension, completely eliminating the contribution of visual information. This approach directly reflects the real-world scenarios where images cannot be acquired due to user privacy restrictions, device acquisition failures, or operating environment limitations.

[0065] Zero-vector replacement is an extreme but controllable masking strategy that can accurately simulate input conditions where the visual modality is completely missing. By setting the visual features to zero, the model is forced to rely solely on the text modality for entity linking reasoning during the training phase, thereby strengthening the semantic expression ability of the text features. For example, when mentioning "apple", the model cannot obtain any image clues (such as company logos or fruit appearance) through zero-vectorized visual features and must rely entirely on text descriptions (such as "technology companies" or "fruits") for ambiguity resolution. This process effectively simulates the modal asymmetry between user input and knowledge base entities in an open environment.

[0066] Through zero-vector replacement, the original complete multimodal data is reconstructed into mention-entity pairs containing only text modality. It is worth noting that the multimodal data on the entity side (i.e., entity descriptions and images in the knowledge base) remains intact to ensure that the student model can fully utilize the semantic information of multimodal entities during retrieval. For example, the student model needs to match entities in the knowledge base that contain both text ("the first president of the United States" or "name of a US state") and images (portraits of people or photos of landmarks) through the text mention "Washington". This input asymmetry forces the model to learn to extract semantic features aligned with multimodal entities from text in the absence of visual information, thereby improving robustness in open environments.

[0067] Step S300: constructing a student model, wherein the student model and the teacher model use the same network architecture and are initialized based on the parameters of the teacher model;

[0068] The student model copies all network weights (such as the parameters of the encoder, fusion module, and similarity calculation layer) from the teacher model trained in the first phase, ensuring that its initial state inherits the semantic representation capabilities learned by the teacher model under multimodal complete data. During subsequent training, the parameters of the teacher model are frozen (i.e., they do not participate in gradient updates), while the parameters of the student model can be dynamically adjusted through backpropagation. This design is based on the self-distillation framework and aims to guide the student model to achieve robust entity linking performance under incomplete modal data (text input only) through the strong discrimination ability of the teacher model on complete modal data, while avoiding the loss of transfer efficiency due to differences in network structure.

[0069] Step S400: inputting the complete multimodal data into the teacher model, and inputting the incomplete mention data into the student model;

[0070] Specifically, the teacher model receives mention-entity pairs containing both text and visual modalities (e.g., "Eiffel Tower" and its corresponding image), generates multi-modal semantic representations and entity similarity scores through forward computation; the student model receives mention data with visual modalities replaced by zero vectors (e.g., only retaining the "Eiffel Tower" text), and performs inference under the same network architecture. This stage constructs a response comparison between the teacher model (complete modalities) and the student model (missing modalities) by inputting two modalities in parallel, ensuring that the student model inherits the multi-modal semantic discrimination ability from the teacher model without modifying the architecture.

[0071] Step S500, based on the mention-entity similarity output by the teacher model, generating a first ranking distribution of entities under complete modalities;

[0072] When processing complete multi-modal data (text + visual), the teacher model calculates the similarity scores of mentions and candidate entities in the knowledge base through a contrastive learning mechanism, and ranks the candidate entities based on the scores. For example, given the mention "apple", the teacher model combines the text description ("technology company") and the image (company logo) to output the matching scores of each candidate entity (such as "Apple Inc." and "fruit apple"), generating a list of entities ranked in descending order of similarity.

[0073] Step S600, based on the mention-entity similarity output by the student model, generating a second ranking distribution of entities under incomplete modalities;

[0074] The student model receives mention data containing only text (e.g., "apple" text, with visual modalities replaced by zero vectors), calculates the matching scores with candidate entities through the same network architecture, and generates the ranking results. For example, the student model only relies on the text description "apple" to infer semantics, which may mistakenly overestimate the score of "fruit apple", resulting in a deviation in the ranking distribution from the teacher model. This step quantifies the gap in semantic discrimination ability of the student model by comparing the first ranking distribution (complete modalities) with the second ranking distribution (missing modalities).

[0075] Step S700, defining a ranking alignment loss function to minimize the difference between the first ranking distribution and the second ranking distribution through a differentiable strategy;

[0076] By defining a ranking alignment loss function (MARD), a differentiable strategy is used to minimize the difference between the teacher model's complete modality ranking distribution (first ranking distribution) and the student model's incomplete modality ranking distribution (second ranking distribution). Specifically, a hyperbolic tangent function is used to continuously approximate the pairwise similarity differences between the teacher and student models, generating a differentiable contrast relationship matrix, thereby converting the discrete ranking relationship into an optimizable numerical form. By calculating the normalized difference between the two ranking distributions and backpropagating the gradient to update the student model parameters, the student is forced to inherit the teacher model's ability to discriminate semantic differences between entities even when the visual modality is missing, ultimately improving the ranking consistency of the retrieval results.

[0077] Step S800: generating a cross-view ranking distribution based on the entity-mention similarity of the teacher model, and defining a cross-view distillation loss function to align the cross-view ranking distribution of the student model;

[0078] Cross-view distillation (CVDI) is used to align the entity-mention ranking distributions of the teacher and student models to mitigate asymmetric retrieval bias caused by modality loss. Specifically, the teacher model calculates entity-to-mention (E2M) similarity based on complete multimodal data (e.g., the matching score between the entity "Apple" in the knowledge base and the user-input mention "apple"), generating a cross-view ranking distribution under the full modality. The student model generates a corresponding cross-view ranking distribution based on incomplete mention data (text only). By calculating the cross-entropy loss between the two distributions, backpropagation is used to adjust the student model parameters, forcing it to inherit the teacher's multimodal discriminative ability in reverse entity-to-mention retrieval. For example, the teacher model achieves a higher matching score for the entity "Apple" in the E2M direction for mentions of "apple" when combining the company logo image and text, while the student model relies solely on text for similarity ranking. This step enhances the model's adaptability to modality loss scenarios through bidirectional retrieval supervision (M2E and E2M), overcoming the limitations of a unidirectional link.

[0079] Step S900: define a self-consistency loss function to force the mention-entity ranking distribution of the student model to be consistent with the entity-mention ranking distribution generated by its reverse generation;

[0080] Specifically, the similarity ranking distributions of the M2E direction (entity retrieval based on text mentions) and the E2M direction (text mention retrieval based on entities) are extracted from the student model, the absolute difference of each entity in the two distributions is calculated, and the self-consistency loss value is generated by averaging. This loss function forces the student model to maintain the logical consistency of bidirectional retrieval under single-modal input. For example, if the mention of "apple" highly matches the entity "Apple" in the M2E direction, then the entity should also highly match the original mention in the E2M direction to avoid reasoning contradictions due to modality loss. Through introspective ranking distillation (IRD), the model can self-correct inconsistencies in cross-view retrieval, thereby improving the robustness of entity linking in open environments.

[0081] Step S1000: jointly optimize the ranking alignment loss function, the cross-view distillation loss function, and the self-consistency loss function, update the parameters of the student model, and obtain a trained student model;

[0082] Specifically, the total loss function is defined as the weighted sum of the three. An adaptive optimizer (such as AdamW) is used to calculate the gradient of the total loss with respect to the parameters, and the student model is iteratively updated with the optimization goal of minimizing the combined loss under multi-level supervision. This process enables the student model to inherit the multimodal discrimination capabilities of the teacher when visual modalities are missing, while maintaining logical consistency in cross-view retrieval, ultimately achieving improved robustness.

[0083] Step S1100: deploy the trained student model to handle the mention-entity linking task with missing visual modality.

[0084] During the inference phase, the visual modality of the input mention is automatically replaced with a zero vector (as described in step S200), retaining only the text description (such as "apple"). The student model calculates the similarity between the plain text input and the multimodal entities in the knowledge base and outputs a list of top-K candidate entities. For example, when the user inputs a mention of "Washington" with a missing image, the model relies on text features to retrieve entities such as "George Washington" (a person) or "Washington State" (a place name), and combines the multimodal information (text + image) on the entity side to optimize the ranking results.

[0085] As described in steps S100 to S1100 above, it can be understood that the main workflow of the cross-view reflective ranking distillation method (CVIRD) for entity knowledge indexing proposed in this method is to enhance the missing modality based on the knowledge distillation (KD) paradigm. Specifically, the teacher and student networks use the same model architecture but have different learning parameters in the self-distillation process, which includes two training stages, such as Figure 2As shown in Figure 2. In the first stage, the teacher follows the traditional training strategy and uses full modality samples to train the model to fully obtain multimodal representations. Given a training batch, the contrastive loss function is used as the task loss function to encourage the model to give higher matching scores for correct mention-entity pairs and lower scores for incorrect mention-entity pairs. For the training batch of mention-entity pairs The loss function of the MEL task can be expressed as follows:

[0086]

[0087] in, is the contrastive loss function for the Multimodal Entity Linking (MEL) task, which is used to measure the model's ability to correctly match mentions and entities during training; M is the set of mentions in the current batch, each of which consists of text and a possible visual modality (such as an image); E is the set of entities in the knowledge base, each of which contains a text description and visual information; θ is the learnable parameter of the model, that is, the weight that needs to be adjusted through optimization; is the size of the training batch, i.e., the number of mention-entity pairs included in each batch; S is to traverse and sum each sample index i in the batch; θ (M i ,E i ) is the reference M calculated by the model i Corresponding to the correct entity E i The similarity score of is determined by the parameter θ; b1 [j≠i] In order to exclude the current correct entity pair (i.e., j=i) when summing, only the similarity of other negative samples is calculated. This is essentially a filtering condition used to exclude the correct entity pair in the denominator and only retain negative samples for calculation.

[0088] Regarding the second stage, given a training mention M i , the visual features Replace with zero vector to form an incomplete mention sample This simulates the mode loss situation in open environment testing. Figure 2 The teacher-student architecture is presented, in which a self-KD method is used to establish an end-to-end distillation process. Specifically, before the second stage of training, the student model is copied from the frozen teacher model in the first stage, and its parameters and network structure are the same as the teacher model. Then, using the complete (M i ,E i ) and incomplete The mention samples and their corresponding correct entities are taken as input and used Figure 3The task loss function shown in the middle trains the student network. Meanwhile, the teacher model processes the modality-complete data and provides supervision for the student model that processes the modality-incomplete data. The loss function of the student network in this stage is

[0089]

[0090] where θ s and θ t represent the parameters of the trainable student model and the frozen teacher model, respectively, represents the mention with missing visual modality (e.g., replaced by a zero vector for the image part), represents the candidate entity similarity scores output by the student model when processing the visual missing mention; represents the incomplete mention; represents the candidate entity similarity scores output by the teacher model when processing the complete mention; represents the knowledge distillation loss. The knowledge distillation loss can be explicitly expressed as:

[0091]

[0092] where τ represents the distillation temperature, KL(,.) represents the Kullback-Leibler divergence, and σ represents the softmax function; represents the similarity score vector of all candidate entities for the i-th sample by the student model; represents the similarity score vector of all candidate entities for the i-th sample by the teacher model. Through the knowledge distillation paradigm, even with single-modality input, the student network can mimic the linking behavior of the teacher model when processing multi-modal data, thus achieving robust retrieval performance. In the inference process, the sample is only input to the student network for downstream tasks.

[0093] Although knowledge distillation (KD) enables the student network to have multi-modal inference capability using only textual information, its application in the MEL-MM task is still significantly limited. Semantic discriminativeness is an important contribution of the visual modality to the MEL task. Specifically, with the image, the multi-modal mention exhibits stronger semantic diversity compared to irrelevant entities in the database, which helps to make the data distribution around each mention more scattered (see Figure 3 ​). However, the KL divergence in the KD process minimizes the difference between the teacher and the student by normalizing the matching scores according to the response of each mention. This supervision fails to effectively convey the semantic diversity between mentions and different candidate entities, which is possessed by a well-performing teacher. In addition, the inherent information imbalance between the teacher and student models inevitably increases the difficulty of KD due to the knowledge transfer from multimodal to text-only data on the mention side. Therefore, the overly strict absolute similarity constraint in traditional KD may not be suitable for the MEL-MM task.

[0094] In this embodiment, the proposed CVIRD method introduces Missing-Aware Ranking Distillation (MARD), which aims to transfer mention-centric similarity ranking to enhance distinguishability. Specifically, given a mention M i , define the mention-centered similarity as M i The matching scores of entities within the batch for the teacher and student models in both the complete and incomplete modalities can be expressed as:

[0095]

[0096] in, It represents the teacher model’s complete mention M i and the jth entity E j Matching score; Indicates that the student model is incomplete and the jth entity E j Matching score of , j = 1, 2, ..., |B|;

[0097] Then, unlike traditional KD which directly uses similarity for supervision, MARD aims to calculate their and This enables the distillation process to be better based on the given content (such as M i ) models the semantic variation of candidate entities within a batch, thereby improving the distinguishability of students trained on incomplete data. MARD first constructs priority comparison matrices for teachers and students respectively. Taking the teacher’s response as an example, define C t for:

[0098]

[0099] Among them, C t is the priority comparison matrix, and I(·) is the pairwise contrast function used to compare the significance between two similarities of mentions from two different entities. For example, The function compares the similarity between the j-th entity and the k-th entity mentioned in the training batch. The function can be expressed as:

[0100]

[0101] in, and represents the matching score of the teacher model (full modality) for the j-th entity and the k-th entity in the batch.

[0102] Mention-centric similarity ranking of the j-th entity within a batch It can be expressed as:

[0103]

[0104] Similarly, the ranking r obtained by the student under the modality missing mention can be calculated according to formula (5-7) s Then, the basic ranking distillation loss can be formulated The consistency between the mention-centric similarity rankings of the normalized teacher and student networks can be expressed as:

[0105]

[0106] in, Represents the teacher model (full modality) for the i-th mention M i and the jth entity E j Match ranking.

[0107] However, in MEL-MM, it is stated that There are still two significant limitations. The first and main challenge lies in the discontinuity of the pairwise contrast function I(·), which conflicts with the optimization methods in deep learning. In order to facilitate the gradient backpropagation in the MEL network, the hyperbolic tangent function g(·) is used to approximate the pairwise contrast function I(·):

[0108]

[0109] Among them, g(,.,) represents the smooth approximate function; and represents the matching score of the teacher model for the j-th and k-th entities, which comes from the similarity vector of the teacher model; tanh(,.,) represents the hyperbolic tangent function.

[0110] Replacing Formula (6) with Formula (9), the final approximate mention-centered similarity ranking is expressed as

[0111]

[0112] For the second flaw, assuming the basic Failed to fully consider the ranking distillation exploited in the modality missing scenario. Specifically, when the input on the mention side changes from multimodal (teacher) to text-only (student) data, the mention-centric similarities between entities in different batches are usually different. During the retrieval process, entities with a large similarity gap between teacher and student responses are unstable candidate entities and are sensitive to image missing. Therefore, a missingness-aware scaling strategy is introduced to dynamically The ranking regularization of different candidate entities in is used to assign training weights. Specifically, the missing-aware weight W of the j-th entity is j It can be defined as:

[0113]

[0114] After overcoming the above two limitations, the improvements in formulas (10) and (11) can be integrated into formula (8), and the final loss function of the proposed MARD is expressed as:

[0115]

[0116] Among them, i and j are loop variables, traversing all entity pairs in the batch; W ij is the weight coefficient, which is used to adjust the contribution of the differences between different entity pairs to the loss function; is the target matching value of the teacher model (full modality) for the i-th mention and the j-th entity; is the predicted match value of the student model (missing modality) for the i-th mention and the j-th entity.

[0117] Figure 4 This is a conceptual illustration of the MEL-MM mission, taking into full consideration the MEL system deployed in actual applications. Figure 4 (a) For closed environment training, the MEL algorithm learns from modally complete data; Figure 4 (b) is reasoning in an open environment, where the visual information of the mentioned party is uncertain and missing due to user reasons. Figure 4 (c) As the missing rate increases, both MEL datasets show a significant performance degradation. Figure 4 As shown in (b), when the image input on the mention side is missing, the linking process becomes asymmetric and the MEL algorithm always retrieves multimodal entities based on plain text mentions, which may damage its robustness and semantic distinguishability in information retrieval. To this end, Cross-View Distillation with Introspection (CVDI) is proposed to alleviate the asymmetry in the two-layer linking. Specifically, in addition to transferring the mention-centric similarity between entities in a batch of training CVDI also provides a cross-view method to distill the similarity of entity centers to within-batch mentions like Figure 5As shown in , it is not a link from unimodal to multimodal data, but as a supplementary knowledge, it gives students the ability to reversely retrieve unimodal candidate entities through multimodal input. The expected output is Complete by modality Supervision. Based on the proposed MARD, the loss function of cross-view distillation (CVDI) from entities to mentions in CVDI is:

[0118]

[0119] in, and According to and In addition, based on the cross-view distillation process, CVDI further proposes to enhance the modality-missing robustness of asymmetric students through self-introspection. Specifically, the mention-centric similarity ranking generated by the student should be consistent with its own entity-centric similarity ranking to achieve self-consistency. Introspective Ranking Distillation (IRD) in CVDI can be defined as:

[0120]

[0121] Based on the above key concepts MARD and CVDI, the second-stage loss function of the student network (Formula (3)) can be reformulated in the knowledge distillation paradigm as:

[0122]

[0123] in, is the final distillation function of the proposed CVIRD method, α and β are trade-off factors that weigh the contributions of distillation from different perspective rankings; Represents the modality-to-entity ranking difference loss, which measures the consistency of the entity matching ranking between the teacher model (complete modality) and the student model (missing modality); represents the noise-to-modality ranking difference loss. Inference uses only the student model trained in the second stage for link evaluation, where the visual embeddings of the missing modalities in the test mentions are padded with zero vectors of the same dimension. Notably, as a KD-based method, CVIRD requires no additional model parameters or special structural design, is flexible and model-independent, and can be applied to any MEL algorithm to improve robustness in the absence of visual modalities.

[0124] To verify the effectiveness of the proposed CVIRD method, the following systematic experiments were conducted on two public multimodal entity linking (MEL) datasets: the Wikipedia Multimodal Entity Linking Dataset (WikiMel) and the RichpediaMel Multimodal Entity Linking Dataset.

[0125] For MEL data, we used two commonly used MEL datasets: WikiMEL and RichpediaMEL, to evaluate the effectiveness of our approach. Both datasets primarily contain person-type entities. WikiMEL contains over 22,000 multimodal sentences from Wikipedia. RichpediaMEL, derived from MMKG Richpedia, involves entities extracted from Richpedia and supplemented with multimodal data from Wikidata. A subset of the Wikidata knowledge graph was used for each dataset, while maintaining the original data partition. Specifically, the training-validation-test set ratio for both the WikiMEL and RichpediaMEL datasets was 70%-10%-20%. During evaluation, the similarity between mentions and all entities in the knowledge graph was measured to assess their alignment probability. Two key metrics were used in the experiment: 1) H@k, which refers to the hit rate of the ground-truth (GT) among the top k ranked entities; and 2) MRR, which represents the mean reciprocal ranking of the ground-truth entities.

[0126] For the MEL network architecture, the state-of-the-art MIMIC is used as the backbone network for most experimental results, while GHMFC and CLIP models are also involved to evaluate model independence. For the comparison methods for the missing modality problem, the baseline is the MEL model trained only with the traditional task loss (Formula (2)). Other competitive methods include: 1) Combination-based methods: DDMI and RMM attempt to borrow or mix images from complete samples of other modalities to the missing visual modality, requiring an additional memory bank; 2) Generation-based methods: CASAE uses text data to train a generative model to generate images for the missing images through feature-level recovery. 3) Distillation-based methods: PMKL and CorrKD utilize knowledge transfer at the feature and logits levels, where the student model in the second stage is trained from scratch. For further comparison, the KD paradigm is compared with traditional knowledge distillation (CKD) and a multi-stage distillation method called DIST.

[0127] Following MIMIC, the visual and textual feature extractors in all referenced MEL backbone networks were initialized using the pretrained CLIP-Vit-Base-Patch32. Optimization was performed using the AdamW optimizer with a batch size of 128. In CVIRD, the MEL model was trained for 20 epochs with a learning rate of 1e-5 in the first phase (E1) and for 10 epochs with the same learning rate in the second phase (E2). The trade-off factors α and β were simply defined as α = β = 1. All experiments were performed using an Intel(R) Core i9-14900KF CPU and an NVIDIA RTX4090 GPU with 24GB of memory. As in previous methods, each model trained in the first phase was evaluated on the validation dataset, and the checkpoint that yielded the largest MRR was selected for the second phase of training. The checkpoint at the last epoch of the second phase was used as the final dropout-resistant MEL model. In the MEL-MM setting, the missingness-aware robustness of the MEL algorithm is evaluated at various missingness ratios of 0%, 25%, 50%, 75% and 100%.

[0128] Table 1: Comparison of the state-of-the-art on image missing rates for five different mention methods on the WikiMel and RichpediaMel datasets. The best and second-best results are in bold and underlined.

[0129]

[0130]

[0131] For RQ1, CVIRD is proposed to be compared with other competitive methods for modality missingness, with the proportion of missing images divided into five different cases on two benchmark datasets. As shown in Table 1, replacing missing modality data with copied or generated samples is an intuitive solution to alleviate the performance degradation caused by uncertain missingness, such as MMDI, RMM, and CASAE. However, these methods involve additional memory or model architecture and have poor flexibility. From the results of distillation-based methods, it can be noted that self-distillation processes (such as CKD, DIST, and the proposed method) achieve comparable linking performance with alternative methods (such as PMKL and CorrKD) with fewer second-stage training cycles, while in the alternative methods, the second-stage student model is trained from scratch. This suggests that the initialization of the teacher model trained from complete data helps the student accumulate robust multimodal knowledge in the MEL-MM task. Among competitive methods, the proposed CVIRD achieves significant retrieval performance in the absence of missing references. On WikiMEL and RichpediaMEL, the Hit@1 is improved by 20.5% and 11.6% respectively compared with the baseline method in the case of complete modality missing (Ratio=1). These findings strongly support the effectiveness of CVIRD in improving the robustness of missing mentions in the MEL-MM task.

[0132] For RQ2, we first verify the contributions of different concepts in CVIRD built based on the KD paradigm, and the results are shown in Table 2. The observations can be divided into the following three aspects:

[0133] 1. As can be seen from rows 1-4, compared with the traditional KD method (which uses smoothing temperature T = 1, 2, and 3 to regularize the absolute match scores), the proposed MARD significantly improves retrieval performance in both H@1 and MRR metrics. This is because MARD's ranking mechanism effectively conveys semantic distinguishability through pairwise similarity comparisons when establishing the ranking order.

[0134] 2. The absence of missingness-aware weight W in MARD and MARD+CVD hurts the linking performance because W promotes ranking distillation, enabling it to dynamically focus on entities that are sensitive to missing data and exhibit instability.

[0135] 3. By integrating MARD and CVDI (CVD and IRD), the final CVIRD is able to fully distill mention-centric similarity ranking from cross-view and self-introspective teacher knowledge. It is worth noting that when the mention data only contains text (ratio = 1), the H@1 performance of CVIRD exceeds that of CKD at Ratio = 0.5.

[0136] For hyperparameters, we first verified the variation of the number of cycles E2 in the second stage of training, where the number of cycles E1 in the first stage is 20. Figure 6 As shown in Figure 2, when there is no visual information on the mentioned side (ratio = 1), increasing E2 from 0 to 10 can improve the accuracy of MEL and then stabilize. If ratio = 0, MEL achieves the best performance when E2 = 10, but the performance decreases after more cycles. The results show that the method of the present invention takes less time in the second stage training, unlike other KD-based methods such as PMKL and CorrKD, which need to be trained from scratch. For the trade-off factors α and β, they are dynamically adjusted within a reasonable range of [0, 0.5, 1, 2]. From Figure 7 It can be seen that they are insensitive to variations, so we simply define α = β = 1 to achieve suitable results in the proposed CVIRD.

[0137] Table 2: Study on the reduction of different concepts in CVIRD based on H@1 and MRR metrics on the WikiMel test set. The traditional KD and the proposed method set temperature and missingness awareness weights respectively.

[0138]

[0139] For RQ3, the baseline method (Baseline) in Table 1 and four efficient methods: CASAE, RMM, CKD and CVIRD of the present invention were selected for cross-dataset validation. Specifically, these five methods are trained based on the MIMIC backbone network on the WikiMel dataset and evaluated on RichpediaMEL. Figure 8 The results shown, the findings can be summarized as follows:

[0140] 1. When there is a discrepancy between the training and test data, the accuracy of the MEL model trained using traditional methods in retrieving the correct entity decreases significantly, even if both databases are related to person information.

[0141] 2. Compared to in-distribution reasoning, the standard MEL model (baseline model) shows greater vulnerability in the case of modal missingness, with H@1 dropping sharply from 65.30 to 22.46. This highlights the urgent need for MEL systems to be robust to missingness in open-world applications.

[0142] 3. Compared with the other three competing methods, the method of the present invention ranks first in both H@1 and MRR indicators, and the performance degradation is smaller as the missing ratio increases.

[0143] Regarding RQ4, CVIRD enhances resilience to the uncertainty of missing visual modalities by distilling robust multimodal knowledge into the student model with incomplete data. Therefore, CVIRD should be orthogonal to model architecture design and can be incorporated into various MEL models. To verify its independence from the backbone network, CVIRD was integrated into the three different MEL algorithms listed in Table 3. Despite differences in entity retrieval accuracy between different MEL networks, the performance of all three methods under standard training (BASE) significantly decreases with increasing missing modality ratios. Although the two feature recovery-based methods, CASAE and RMM, have their own advantages in different MEL architectures, the knowledge distillation paradigm remains a more effective technical approach to mitigate the performance degradation caused by missing modalities. Our CVIRD ranked first in almost all metrics and performed well under three different missing modality ratios. This strongly demonstrates the effectiveness of our method in MEL-MM and its great potential for application in various multimodal network architectures.

[0144] Table 3: Model difference evaluation of the baseline (BASE) and four anti-omission strategies based on three different MEL models: MIMIC, GHMFC, and CLIP.

[0145]

[0146]

[0147] When the MEL system is deployed in open environment applications, the image on the mention side is easily missing. To address this problem, the present invention proposes a novel cross-view introspective ranking distillation (CVIRD) to enhance semantic distinguishability by introducing two key concepts: Missing-Aware Ranking Distillation (MARD) and Cross-View Distillation with Introspection (CVDI). Considering the particularly important ranking factor in the distance change from the mention to the candidate entity, MARD is designed to distill ranking knowledge in a missing-aware and differentiable manner. The introduction of CVDI is to fully extract the distinguishability of ranking distillation from the perspective of cross-view and self-introspection. A large number of experiments verify the effectiveness of the framework of the present invention in dealing with the missing problem, and achieves significantly better performance on two widely used datasets compared with competing methods.

[0148] According to another aspect of an embodiment of the present application, an electronic device is provided, including a processor and a memory, wherein the processor is configured to implement the steps of the method when executing a computer program stored in the memory.

[0149] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0151] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0153] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A cross-view reflective ranking distillation method for entity knowledge indexing, characterized by: The steps include: training a teacher model based on full multimodal data, wherein the full multimodal data includes mention-entity pairs in textual and visual modalities; generating incomplete mention data containing only textual modality by masking the visual modality in the complete multimodal data; Constructing a student model, wherein the student model and the teacher model use the same network architecture and are initialized based on the parameters of the teacher model; Inputting the complete multimodal data into the teacher model and the incomplete mention data into the student model; Based on the mention-entity similarity output by the teacher model, generating a first ranked distribution of entities in the full modality; generating a second ranked distribution of entities in the incomplete modality based on the mention-entity similarity output by the student model; defining a ranking alignment loss function to minimize the difference between the first ranking distribution and the second ranking distribution through a differentiable strategy; Based on the entity-mention similarity of the teacher model, generating a cross-view ranking distribution, and defining a cross-view distillation loss function to align the cross-view ranking distribution of the student model; Define a self-consistency loss function to enforce that the student model’s mention-entity ranking distribution is consistent with its reverse-generated entity-mention ranking distribution; Jointly optimizing the ranking alignment loss function, the cross-view distillation loss function, and the self-consistency loss function, updating the parameters of the student model, and obtaining a trained student model; Deploy the trained student model to handle the mention-entity linking task in the absence of visual modality.

2. The cross-view reflective ranking distillation method for entity knowledge indexing according to claim 1 is characterized in that: The method of minimizing the difference between the first ranking distribution and the second ranking distribution by the differentiable strategy includes: The pairwise similarity differences between the teacher model and the student model are continuously approximated by a hyperbolic tangent function to generate a differentiable contrast relationship matrix; Calculating a normalized difference value between the first ranking distribution and the second ranking distribution based on the comparison relationship matrix; The parameters of the student model are updated by minimizing the normalized difference value.

3. The cross-view reflective ranking distillation method for entity knowledge indexing according to claim 1 is characterized in that: The cross-view sorting distribution alignment method includes: Based on the entity-mention similarity output by the teacher model, generate a teacher cross-view ranking distribution of entities to mentions in the full modality; Based on the entity-mention similarity output by the student model, generating a student cross-view ranking distribution of entities to mentions in the incomplete modality; Calculating the cross-entropy loss between the teacher's cross-view ranking distribution and the student's cross-view ranking distribution to generate a cross-view distillation loss function; The cross-view distillation loss function is minimized by a back-propagation algorithm, and the parameters of the student model are adjusted to align the cross-view ranking distribution of the student model.

4. The cross-view reflective ranking distillation method for entity knowledge indexing according to claim 1 is characterized in that: The calculation of the self-consistency loss function includes the following operations: Extracting the mention-entity similarity ranking distribution and the entity-mention similarity ranking distribution of the student model; Calculate the absolute difference between the two sorted distributions for each entity; The absolute differences of all entities are averaged to generate the self-consistency loss.

5. The cross-view reflective ranking distillation method for entity knowledge indexing according to claim 1 is characterized in that: The specific operations of the joint optimization include: The ranking alignment loss function, the cross-view distillation loss function and the self-consistency loss function are weighted and summed according to a preset weight ratio to generate a total loss function; Calculate the gradient of the total loss function with respect to the student model parameters by a back-propagation algorithm; An adaptive optimizer is used to update the parameters of the student model, and the optimization goal is to minimize the total loss function.

6. The cross-view reflective ranking distillation method for entity knowledge indexing according to claim 1 is characterized in that: The method for generating incomplete mention data containing only text modality by masking the visual modality in the complete multimodal data includes: replacing the visual modality features in the complete multimodal data with a zero vector.

7. The cross-view reflective ranking distillation method for entity knowledge indexing according to claim 1 is characterized in that: The training method of the teacher model includes: Define a contrastive learning loss function, which includes: Maximize the similarity score of correct mention-entity pairs; Apply a negative sample penalty to the similarity score of the wrong mention-entity pair to make it approach zero; The parameters of the teacher model are iteratively optimized by a stochastic gradient descent algorithm until the contrastive learning loss function converges.

8. The cross-view reflective ranking distillation method for entity knowledge indexing according to claim 1 is characterized in that: Initializing the parameters of the student model includes copying the network weights of the teacher model and freezing the parameters of the teacher model during training.

9. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the cross-view reflective ranking distillation method for entity knowledge indexing as described in any one of claims 1-8, and the processor is configured to execute the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the cross-view reflective ranking distillation method for entity knowledge indexing according to any one of claims 1 to 8 are executed.

Images