An adaptive weight graph learning driven semantic enhanced cross-modal hash retrieval method

Abstract

The application discloses a semantic enhancement cross-modal hash retrieval method driven by adaptive weight graph learning, relates to the technical field of information retrieval, and comprises the following steps: acquiring multi-modal data and semantic labels; jointly modeling the multi-modal data and the semantic labels, and constructing semantic enhancement label representation; adopting an asymmetric inner product strategy to depict the pair-wise semantic similarity relationship between samples; using an adaptive weight graph learning mechanism to learn class weights and suppress noise; designing an optimization algorithm under discrete constraint conditions to generate binary hash codes; adopting a linear regression framework to learn a modal-specific hash function, and realizing fast extrapolation coding of new samples; and through the construction of the semantic enhancement label representation, high-order correlations between multi-labels are mined, the asymmetric inner product strategy is introduced to accurately depict the pair-wise semantic similarity of cross-modal samples, and the adaptive weight graph learning mechanism is combined to dynamically adjust the importance of classes and effectively suppress noise label interference.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology, and in particular to an adaptive weighted graph learning-driven semantically enhanced cross-modal hash retrieval method. Background Technology

[0002] With the continuous growth of multimedia data volume, cross-modal retrieval has received widespread attention in the field of information retrieval. Cross-modal hashing (CMH) methods map data from different modalities to a unified binary hash code, achieving fast approximate retrieval of cross-modal samples while maintaining retrieval efficiency and low storage overhead. Based on whether semantic tag information is utilized, existing cross-modal hashing methods are mainly divided into unsupervised and supervised CMH methods. Unsupervised CMH methods primarily rely on data distribution or feature similarity for modeling; due to the lack of semantic constraints, they struggle to accurately characterize high-level semantic relationships between cross-modal samples, thus limiting retrieval performance. Supervised CMH methods, by introducing semantic tag information to construct similarity relationships between samples or directly learning hash functions, have improved retrieval performance to some extent and have therefore attracted widespread attention.

[0003] However, existing supervised CMH methods typically assume that the semantic labels of training samples are complete and accurate. In real-world applications, semantic labels are often missing or incorrect, and the direct use of noisy labels can easily interfere with the hash code learning process, reducing retrieval performance. Furthermore, multimedia data often has multi-label attributes, and existing methods often ignore the potential semantic relationships between labels during the modeling process, treating them as independent of each other. This leads to insufficient semantic information mining and limits the semantic expressive power of hash codes. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides an adaptive weight graph learning-driven semantic enhancement cross-modal hash retrieval method to address the problem that existing supervised CMH methods typically assume complete and accurate semantic labels, which can easily lead to decreased hash code discrimination ability and degraded retrieval performance when labels are missing or incorrect or when the semantic association of multiple labels is not fully modeled.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides an adaptive weight graph learning-driven semantically enhanced cross-modal hash retrieval method, comprising: S1 acquires multimodal data and semantic labels; S2 performs joint modeling of multimodal data and its semantic labels to construct semantically enhanced label representations; S3 employs an asymmetric inner product strategy to characterize pairwise semantic similarity relationships between samples; S4 uses an adaptive weight graph learning mechanism to learn class weights and suppress noise; S5 Designs an optimization algorithm to generate binary hash codes under discrete constraints; S6 uses a linear regression framework to learn a modality-specific hash function, enabling fast extrapolation encoding of new samples; As a preferred embodiment of the adaptive weight graph learning-driven semantically enhanced cross-modal hash retrieval method of the present invention, the specific steps for jointly modeling multimodal data and its semantic labels to construct a semantically enhanced label representation are as follows: Discriminative hash codes are learned using a linear regression model, as described below: ; Introducing a label relevance matrix This describes the association information between label categories, and uses self-representation learning to map discrete semantic label information into an enhanced label matrix carrying higher-order semantic associations, expressed as: ; in, For a hash matrix, For the mapping matrix, This is a label matrix.

[0007] As a preferred embodiment of the adaptive weight graph learning-driven semantic enhancement cross-modal hash retrieval method of the present invention, wherein: the asymmetric inner product strategy is used to characterize the pairwise semantic similarity relationship between samples, the specific steps are as follows: semantic tag matrix Normalization is performed, and the similarity matrix is ​​calculated using its inner product. ; We use a symmetric nonnegative matrix factorization model to learn pairwise semantic similarity relationships between samples, expressed as follows: ; Considering that the binary discrete inner product in the above equation makes the optimization problem difficult to solve, an asymmetric inner product strategy is introduced, expressed as: ; Introducing orthogonality and balance to constrain hash bits to be uncorrelated and balanced respectively, the expression is: ; in and It is a balance parameter. It is a hash continuous matrix. It is the hash code length. It is the number of samples. It is the identity matrix. It is a unit vector.

[0008] As the semantically enhanced cross-modal hash retrieval method driven by adaptive weight graph learning described in this invention, the specific steps of learning category weights and suppressing noise using the adaptive weight graph learning mechanism are as follows: In the enhanced label space carrying higher-order semantic associations In this process, a graph learning strategy is used to dynamically optimize the similarity matrix, expressed as: ; Diagonal matrix Introduced into the above model, the weights for adaptively learning the categories to suppress category noise are expressed as follows: ;

[0009] in This is the weight matrix. For the weight vector, For the first Weighted scores for each category, It is a flexible similarity matrix.

[0010] As a preferred embodiment of the adaptive weight graph learning-driven semantic enhancement cross-modal hash retrieval method of the present invention, the specific steps of designing an optimization algorithm to generate binary hash codes under discrete constraints are as follows: The above three parts are integrated into a unified objective function, expressed as follows: ;

[0011] Under discrete constraints, an alternating iterative update method is used to update the variables respectively. , , , , and Update; When the objective function converges, stop iterating and output each variable and its binary hash code.

[0012] As a preferred embodiment of the adaptive weight graph learning-driven semantic enhancement cross-modal hash retrieval method of the present invention, the method employs a linear regression framework to learn a modality-specific hash function to achieve fast extrapolation encoding of new samples. The specific steps are as follows: The nonlinear RBF kernel function is used to generate kernelized data features, so that the data dimensions of different modalities are the same; A mode-specific hash function is learned using a linear regression framework, expressed as:

[0013] pass Calculate the optimal hash function ; use To obtain the hash codes of different modalities of the new sample ; By querying hash codes, new samples can be quickly extrapolated and encoded. in For balancing parameters, For the first Hash functions of various modes For the first The training samples after modal kernelization, It is the identity matrix. For the first Hash encoding of modal test samples.

[0014] The beneficial effects of this invention are as follows: by constructing semantically enhanced label representations to mine higher-order associations between multiple labels, introducing an asymmetric inner product strategy to accurately characterize the pairwise semantic similarity of cross-modal samples, combining an adaptive weighted graph learning mechanism to dynamically adjust class importance and effectively suppress noise label interference, co-optimizing to generate highly discriminative binary hash codes under strict discrete constraints, and learning modality-specific hash functions based on a linear regression framework to achieve efficient extrapolation encoding of new samples, thereby improving the robustness, accuracy, and practicality of cross-modal hash retrieval in scenarios with incomplete, noisy, or semantically complex labels. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the semantically enhanced cross-modal hash retrieval method driven by adaptive weight graph learning in Example 1.

[0017] Figure 2 This is a schematic diagram of the semantically enhanced cross-modal hash retrieval framework driven by adaptive weight graph learning in Example 1. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0019] The overall structure of the model is as follows Figure 1 As shown, the semantically enhanced cross-modal hash retrieval method driven by adaptive weight graph learning mainly consists of five parts: semantically enhanced label modeling, pairwise semantic similarity modeling, adaptive weight graph learning, discrete hash code optimization, and modality-specific hash function learning and retrieval. Semantically enhanced label modeling maps discrete semantic label information into an enhanced label matrix carrying high-order semantic associations through self-representation learning, explicitly exposing label noise. Pairwise semantic similarity modeling preserves semantic information from the perspective of sample pairs, maintaining pairwise semantic similarity between samples. The adaptive weight graph learning strategy learns category weights for the enhanced label matrix without prior knowledge, suppressing noise information. For discrete hash code optimization, under discrete constraints, the proposed model is non-convex for all variables, making optimization difficult. Therefore, we use an alternating iterative update method to optimize each variable in the model separately. Based on the obtained hash code, modality-specific hash function learning and retrieval uses a linear regression framework to learn the modality-specific hash function, achieving fast extrapolation encoding of new samples. The following detailed implementation method further illustrates this: This implementation method, driven by adaptive weight graph learning, includes the following steps: Step S1: Obtain multimodal data and semantic labels.

[0020] This invention was tested on two publicly available datasets: MIRFlickr-25 K and IAPR TC-12. The MIRFlickr-25 K dataset contains 25,000 images and corresponding text descriptions, with 24 semantic label categories, and each instance is associated with at least one label from these 24 categories. The IAPR TC-12 dataset contains 20,000 images and corresponding text descriptions, with 225 semantic label categories, and each instance is associated with one or more labels.

[0021] It should be noted that semantic labels in real-world scenarios are often missing or even incorrect. Directly using noisy labels for training can easily interfere with the learning of hash codes, thereby dragging down retrieval performance. The proposed method significantly improves the accuracy and efficiency of cross-modal retrieval under such weakly supervised conditions.

[0022] In particular, this method fully considers the incompleteness of label information in real-world application scenarios during the data input stage. Real-world multimodal data often originates from user-generated content or automatic labeling systems, and its semantic labels inevitably contain missing, redundant, or even erroneous elements. Directly using such weakly supervised signals for model training can easily introduce bias, causing the learned hash representation to deviate from the true semantic structure. Therefore, this invention starts from the source, explicitly designing the overall architecture based on a weakly labeled environment, ensuring that subsequent modules possess inherent robustness against noise and missing information, thereby guaranteeing the method's practical value and deployment feasibility in real-world retrieval systems.

[0023] Step S2: Jointly model the multimodal data and its semantic labels to construct a semantically enhanced label representation.

[0024] Discriminative hash codes are learned using a linear regression model, as described below: ;

[0025] Introducing a label relevance matrix This describes the association information between label categories, and uses self-representation learning to map discrete semantic label information into an enhanced label matrix carrying higher-order semantic associations, expressed as: ;

[0026] in, For a hash matrix, For the mapping matrix, This is a label matrix.

[0027] It should be noted that the elements in the previous semantic label matrix consisted of 0 and 1, which did not have a statistical structure and could not be directly analyzed. Therefore, by using self-representation learning, the discrete semantic label information was mapped into an enhanced label matrix carrying higher-order semantic associations, thus making the label noise explicitly exposed.

[0028] In particular, original semantic labels typically exist in discrete binary form, merely reflecting whether a sample belongs to a certain class, failing to reveal the inherent relationships or semantic hierarchy between classes. This representation lacks structural information, making it difficult to support fine-grained semantic modeling. Therefore, this method employs joint learning to collaboratively optimize the label matrix and data features, automatically generating an enhanced label representation that contains higher-order semantic associations. This representation not only preserves the basic semantics of the original labels but also explicitly reveals the co-occurrence patterns and potential dependencies between classes, exposing previously hidden label noise and providing a more reliable foundation for subsequent noise suppression and similarity modeling.

[0029] Step S3: Use an asymmetric inner product strategy to characterize the pairwise semantic similarity relationships between samples.

[0030] semantic tag matrix Normalization is performed, and the similarity matrix is ​​calculated using its inner product. ; We use a symmetric nonnegative matrix factorization model to learn pairwise semantic similarity relationships between samples, expressed as follows: ;

[0031] Considering that the binary discrete inner product in the above equation makes the optimization problem difficult to solve, an asymmetric inner product strategy is introduced, expressed as: ;

[0032] Introducing orthogonality and balance to constrain hash bits to be uncorrelated and balanced respectively, the expression is: ;

[0033] in and It is a balance parameter. It is a hash continuous matrix. It is the hash code length. It is the number of samples. It is the identity matrix. It is a unit vector.

[0034] It should be noted that this strategy preserves semantic information from the perspective of sample pairs, maintains the pairwise semantic similarity between samples, breaks the assumption that previous methods often assumed that samples follow independent and identically distributed distributions, and mines the interconnected information between samples.

[0035] In particular, traditional methods often assume that samples are independent of each other or rely solely on global statistical properties to construct similarity relationships, ignoring the fine-grained associations between sample pairs in the local semantic space. This method, however, takes a paired sample approach, flexibly characterizing semantic similarity through an asymmetric strategy, avoiding information loss caused by forced symmetry constraints. This design allows the model to more accurately capture the semantic similarity between different sample pairs while maintaining computational efficiency. Furthermore, by introducing orthogonality and balance constraints, it ensures that each bit of the generated hash code carries independent and balanced information, thereby improving the discriminative power and retrieval accuracy of the entire hash space.

[0036] Step S4: Use an adaptive weight graph learning mechanism to learn class weights and suppress noise.

[0037] In the enhanced label space carrying higher-order semantic associations In this process, a graph learning strategy is used to dynamically optimize the similarity matrix, expressed as: ;

[0038] Diagonal matrix Introduced into the above model, the weights for adaptively learning the categories to suppress category noise are expressed as follows: ;

[0039] in This is the weight matrix. For the weight vector, For the first Weighted scores for each category, It is a flexible similarity matrix.

[0040] It should be noted that S1 explicitly exposes label noise, enabling statistical analysis of category information. Therefore, an adaptive weight graph learning strategy is introduced to map the space... Learn more accurate structural information and class weights for this space, and suppress noise information.

[0041] In particular, the reliability of different semantic categories varies significantly in practical applications, and some category labels may introduce a large amount of noise due to strong subjectivity or vague labeling standards. If all categories are treated equally, the noise information will interfere with the learning of the overall similarity structure. This method dynamically evaluates the confidence of each category through a graph learning mechanism and assigns adaptive weights to it. High-confidence categories dominate in constructing sample similarity relationships, while low-quality categories are automatically weakened. This data-driven weight adjustment mechanism requires no manual intervention, can automatically identify and suppress the influence of noise during training, and significantly enhances the model's adaptability to complex labeling environments.

[0042] Step S5: Design an optimization algorithm to generate binary hash codes under discrete constraints.

[0043] The above three parts are integrated into a unified objective function as follows: ;

[0044] Under discrete constraints, an alternating iterative update method is used to update the variables respectively. , , , , and Update; When the objective function converges, stop iterating and output each variable and its binary hash code.

[0045] It should be noted that under discrete constraints, the proposed model is non-convex with respect to all variables, making it difficult to optimize simultaneously. Therefore, we use an alternating iterative update method to optimize each variable in the model separately.

[0046] In particular, since hash codes must be in discrete binary form, directly optimizing the overall objective function is mathematically extremely difficult. To address this challenge, this method employs an alternating iterative strategy, decomposing the complex joint optimization problem into multiple subproblems. Each subproblem updates only a subset of variables, while the remaining variables remain temporarily fixed. This step-by-step solution not only significantly reduces computational complexity but also fully utilizes the structural characteristics of each subproblem to design efficient update rules. Although the overall objective is non-convex, through proper initialization and constraint control, the algorithm can achieve stable convergence in practice, ultimately obtaining high-quality binary hash codes, balancing theoretical rationality and engineering feasibility.

[0047] Step S6: Use a linear regression framework to learn a modality-specific hash function to achieve fast extrapolation encoding of new samples.

[0048] The nonlinear RBF kernel function is used to generate kernelized data features, so that the data dimensions of different modalities are the same; A mode-specific hash function is learned using a linear regression framework, expressed as: ;

[0049] pass Calculate the optimal hash function ; use To obtain the hash codes of different modalities of the new sample ; By querying hash codes, new samples can be quickly extrapolated and encoded. in For balancing parameters, For the first Hash functions of various modes For the first The training samples after modal kernelization, It is the identity matrix. For the first Hash encoding of modal test samples.

[0050] It should be noted that, based on the obtained hash code, the modality-specific hash function learning and retrieval adopts a linear regression framework to learn the modality-specific hash function, thereby achieving fast extrapolation encoding of new samples. Moreover, this scheme is not limited to image-text; it can also be directly applied to other modalities such as video.

[0051] Specifically, to support rapid encoding of unseen samples, this method employs a modality-specific hash function learning mechanism. This mechanism first maps the original features of different modalities to a unified high-dimensional space using a kernel method, eliminating modality heterogeneity. Then, a concise regression framework is used to establish the mapping relationship from features to hash codes. Since the regression model has a closed-form solution, the training process is efficient and stable, and the inference phase only requires a single linear transformation plus a sign function to complete the encoding. This design not only ensures real-time retrieval capability for new samples but also possesses good scalability, naturally adapting to any newly added modality without requiring a complete redesign of the system architecture.

[0052] The above description, in conjunction with specific preferred embodiments, provides a more detailed explanation of the present invention. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such deductions or substitutions should be considered to fall within the scope of protection of the present invention.

Claims

1. An adaptive weight graph learning-driven semantically enhanced cross-modal hash retrieval method, characterized in that: include: Acquire multimodal data and semantic labels; Joint modeling of multimodal data and its semantic labels is performed to construct semantically enhanced label representations; An asymmetric inner product strategy is used to characterize the pairwise semantic similarity relationships between samples; An adaptive weighted graph learning mechanism is used to learn class weights and suppress noise. Design an optimization algorithm to generate binary hash codes under discrete constraints; A linear regression framework is used to learn a modality-specific hash function to achieve fast extrapolation encoding of new samples.

2. The adaptive weight graph learning-driven semantic enhancement cross-modal hash retrieval method as described in claim 1, characterized in that: The specific steps for jointly modeling multimodal data and its semantic labels to construct semantically enhanced label representations are as follows: Discriminative hash codes are learned using a linear regression model, as described below: ； Introducing a label relevance matrix This describes the association information between label categories, and uses self-representation learning to map discrete semantic label information into an enhanced label matrix carrying higher-order semantic associations, expressed as: ； in, For a hash matrix, For the mapping matrix, This is a label matrix.

3. The adaptive weight graph learning-driven semantic enhancement cross-modal hash retrieval method as described in claim 1, characterized in that: The specific steps for using the asymmetric inner product strategy to characterize the pairwise semantic similarity relationship between samples are as follows: semantic tag matrix Normalization is performed, and the similarity matrix is ​​calculated using its inner product. ; We use a symmetric nonnegative matrix factorization model to learn pairwise semantic similarity relationships between samples, expressed as follows: ； Considering that the binary discrete inner product in the above equation makes the optimization problem difficult to solve, an asymmetric inner product strategy is introduced, expressed as: ； Introducing orthogonality and balance to constrain hash bits to be uncorrelated and balanced respectively, the expression is: ； in and It is a balance parameter. It is a hash continuous matrix. It is the hash code length. It is the number of samples. It is the identity matrix. It is a unit vector.

4. The adaptive weight graph learning-driven semantic enhancement cross-modal hash retrieval method as described in claim 1, characterized in that: The specific steps for learning class weights and suppressing noise using the adaptive weight graph learning mechanism are as follows: In the enhanced label space carrying higher-order semantic associations In this process, a graph learning strategy is used to dynamically optimize the similarity matrix, expressed as: ； Diagonal matrix Introduced into the above model, the weights for adaptively learning the categories to suppress category noise are expressed as follows: ； in This is the weight matrix. For the weight vector, For the first Weighted scores for each category, It is a flexible similarity matrix.

5. The adaptive weight graph learning-driven semantic enhancement cross-modal hash retrieval method as described in claim 1, characterized in that: The specific steps for designing an optimization algorithm to generate binary hash codes under discrete constraints are as follows: The above three parts are integrated into a unified objective function, expressed as follows: ； Under discrete constraints, an alternating iterative update method is used to update the variables respectively. , , , , and Update; When the objective function converges, stop iterating and output each variable and its binary hash code.

6. The adaptive weight graph learning-driven semantic enhancement cross-modal hash retrieval method as described in claim 1, characterized in that: The method employs a linear regression framework to learn a modality-specific hash function, enabling fast extrapolation encoding of new samples. The specific steps are as follows: The nonlinear RBF kernel function is used to generate kernelized data features, so that the data dimensions of different modalities are the same; A mode-specific hash function is learned using a linear regression framework, expressed as: ； pass Calculate the optimal hash function ; use To obtain the hash codes of different modalities of the new sample ; By querying hash codes, new samples can be quickly extrapolated and encoded. in For balancing parameters, For the first Hash functions of various modes For the first The training samples after modal kernelization, It is the identity matrix. For the first Hash encoding of modal test samples.

Images