Semi-supervised cross-modal hash method based on hypergraph attention convolution

Abstract

The invention provides a semi-supervised cross-modal hashing method based on hypergraph attention convolution, and belongs to the technical field of computers. Firstly, a robust similarity matrix construction method is provided; secondly, a hypergraph attention convolution module is provided, an embedded vector is generated through hypergraph convolution and an attention mechanism of input data, important high-order semantics are captured, and a loss function is independently designed for a labeled data sample and a non-labeled data sample to optimize a hash code; and finally, designing a final objective function based on the loss function to learn a final hash code. According to the method, the high-order semantic information in the input sample can be fully mined and utilized. And a loss function is independently designed for a labeled data sample and a non-labeled data sample to optimize a hash code, so that unsupervised model training can be realized, semi-supervised model learning can be carried out under a small amount of labeled data, and finally, a final objective function is designed based on the loss function to learn a final hash code.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and relates to a semi-supervised cross-modal hash retrieval method for grape pest and disease images and texts, and in particular to a semi-supervised cross-modal hash method based on hypergraph attention convolution. Background Art

[0002] As a characteristic economic industry, grapes are of great significance in promoting local economic growth and rural revitalization. However, in the process of grape cultivation, due to the incomplete popularization of pest and disease control knowledge and improper management, grapes are easily attacked by pests and diseases, which affects the yield and quality, resulting in serious economic losses. At present, the field of grape pest and disease control faces the following problems: First, the knowledge of pest and disease control is scattered and uncentralized, and there is a lack of effective data integration and management methods; second, the existing prevention and control technology relies too much on a large amount of labeled data, which has become a significant bottleneck in the context of scarce data in the field of grape pests and diseases; third, traditional pest and disease identification methods lack efficient cross-modal data integration and intelligent retrieval capabilities, and it is difficult to meet the application requirements of large-scale, real-time and high precision.

[0003] In the prior art, Zhao et al. proposed a grape disease recognition method based on transfer learning, using a convolutional neural network model pre-trained on a large-scale dataset and fine-tuning it to adapt to the grape disease dataset (Zhao H, Zhang Y, Li W. Transfer learning-based grape disease detection using convolutional neural networks [J]. Computers and Electronics in Agriculture, 2020, 175: 105576.). This method significantly reduces the need for labeled data and achieves high recognition accuracy on small-scale datasets. However, this technology is still limited to the analysis of unimodal data, fails to fully utilize the potential of cross-modal information, and its recognition accuracy is insufficient in complex scenarios.

[0004] In addition, Wang et al. proposed a pest and disease monitoring system based on image segmentation and multimodal feature fusion (Wang T, Liu Q, Chen Y, et al. Multimodal grape pest monitoring system using image segmentation and feature fusion [J]. Precision Agriculture, 2021, 22 (5): 1416-1432.). This method improves the comprehensiveness and early prediction capabilities of pest and disease detection by combining images, meteorological data and environmental features. However, this technology mainly uses a simple feature splicing strategy, and does not adequately mine the deep semantic relationships between different modalities, making it impossible to accurately model complex dynamic environments. In addition, the efficiency of high-dimensional data processing is low, making it difficult to meet real-time requirements.

[0005] In another study, Chen et al. proposed a grape pest and disease image restoration and recognition method based on deep generative adversarial networks (GAN), aiming to improve the robustness of the recognition model by generating unlabeled image data (Chen L, Sun Y, Zhang R. Grape pest and disease recognition via deep generative adversarial networks [J]. Biosystems Engineering, 2021, 204: 105-115.). Although GAN shows high flexibility in data generation and can alleviate the problem of insufficient labeled data, the technology lacks control over the quality of generated data and cannot fully utilize multimodal information to assist generation and recognition, resulting in limited practicality in actual complex scenarios.

[0006] Therefore, under the condition of limited labeled data, how to fully explore and utilize the deep semantic associations of cross-modal data to achieve efficient, accurate and real-time monitoring of grape pests and diseases is a key issue that needs to be urgently solved in the current field. Summary of the invention

[0007] In order to solve these problems, the present invention proposes a semi-supervised cross-modal hashing method based on hypergraph attention convolution (SSHGNN for short). This method aims to improve the retrieval accuracy of cross-modal data by effectively integrating data of different modalities (such as images, texts, etc.) and breaking through the traditional bottleneck of relying on a large amount of labeled data. Specifically, firstly, a robust similarity matrix construction method is proposed for the redundant information that is prevalent in existing methods. Secondly, the present invention proposes a new hypergraph attention convolution module, which generates embedded questions through hypergraph convolution and the attention mechanism of input data, captures important high-order semantics, and designs loss functions separately for labeled data samples and unlabeled data samples to optimize hash codes, which can realize unsupervised model training and semi-supervised model learning with a small amount of labeled data. Finally, the final objective function is designed based on the loss function to learn the final hash code. This method can not only effectively use unlabeled data for training, but also improve the accurate matching and rapid positioning capabilities of cross-modal data, solving the problem of dependence on data volume in the prior art. Through this method, growers can more efficiently and accurately identify and locate grape diseases and pests, so as to scientifically carry out prevention and control management and reduce the economic losses caused by diseases and pests.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A semi-supervised cross-modal hashing method based on hypergraph attention convolution. The semi-supervised cross-modal hashing method first proposes a robust similarity matrix construction method for the redundant information commonly present in existing methods. Secondly, the present invention proposes a new hypergraph attention convolution module, which generates an embedding vector through hypergraph convolution and the attention mechanism of input data, captures important high-order semantics, and separately designs loss functions for labeled data samples and unlabeled data samples to optimize hash codes. It can realize unsupervised model training and semi-supervised model learning with a small amount of labeled data. Finally, the final objective function is designed based on the loss function to learn the final hash code. The module for constructing a robust similarity matrix, the hypergraph-based enhancement component, and the semi-supervised cross-modal hashing method refer to both labeled data and unlabeled data. Specifically, the following steps are included:

[0010] Step 1, construct a robust similarity matrix;

[0011] Step 2: Hypergraph-based attention convolution module;

[0012] Step 3: Loss function design;

[0013] Step 4: Objective function;

[0014] Furthermore, the step 1, constructing a robust similarity matrix, is specifically:

[0015] Step 1.1 Image feature extraction: The fc6 layer of the pre-trained neural network model VGG19 is used to extract the image features of grape pests and diseases, denoted as F I ; Text feature representation: Use bag-of-words (BoW) features to extract features from grape pest text data, denoted as F T .

[0016] Step 1.2 constructs a similarity matrix: The grape pest image features F obtained in step 1.1 I and text features F T , construct the similarity matrix S, each element S in the similarity matrix S i,j It represents the similarity between the i-th grape pest and disease sample and the j-th sample. The details are as follows:

[0017] First, calculate the image features F of grape pests and diseases respectively I and text features F T The intra-modal similarity matrix is ​​as follows:

[0018]

[0019] Among them, cos(,) represents cosine similarity; Represents the similarity between the i-th grape pest sample and the j-th sample in image features or text modality; represents the i-th grape pest sample feature in image mode or text mode; F j * represents the j-th grape pest and disease sample feature in image mode or text mode; i represents the image mode of grape pest and disease; T represents the text mode of grape pest and disease.

[0020] Then, the inter-modality similarity matrix between the grape pest image modality and the text modality is calculated as follows, denoted as S fusion :

[0021]

[0022] Among them, S I ,S T is the intra-modal similarity matrix calculated in formula (1). I S represents the similarity matrix of grape pest image modalities; T The similarity matrix of the text modality of grape pests and diseases is shown in Figure 2. fusion Combined, we get the final similarity matrix S:

[0023] S=λ v S I +λt S T +λ f S fusion (3)

[0024] Among them, λ v represents the weight parameter of the modal similarity matrix of grape pest and disease images; t Represents the weight parameter of the modal similarity matrix of grape pests and diseases text; f Represents the weight parameters of the inter-modality similarity matrix between the grape pest and disease image modality and the text modality.

[0025] In view of the differences in statistical characteristics and perspectives between the image modality and text modality of grape pests and diseases, the weight parameter λ v ,λ t , and λ f It is used to adaptively fuse the information of these two modalities.

[0026] Next, the similarity matrix S obtained by formula (3) is fitted with two half-Gaussian distributions to the histogram of similarity between the image features and text features of grape pests and diseases, so as to reduce the redundant information in the similarity matrix. First, the similarity threshold and the difference threshold are calculated, and the similarity threshold is set to The difference threshold is set to in and are the mean and variance of the right half Gaussian distribution and the left half Gaussian distribution, β1 is the similarity threshold parameter, and β2 is the difference threshold parameter. Then, the similarity matrix S is scaled according to the similarity threshold and difference threshold to obtain the robust similarity matrix The formula is shown below.

[0027]

[0028] in, Represents the robust similarity matrix Each element in Specifically, it is the similarity between the i-th grape pest and disease sample and the j-th sample; S i,j Represents each element S in the similarity matrix S i,j , specifically, the similarity between the i-th grape pest and disease sample and the j-th sample.

[0029] When the value of the similarity matrix S is greater than or equal to the similarity threshold t + , it is corrected to +1; when the value of the similarity matrix S is less than or equal to the difference threshold t - , is corrected to -1, indicating dissimilarity, and the other values ​​of the similarity matrix S remain unchanged.

[0030] Furthermore, the step 2 is based on the hypergraph attention convolution module, specifically:

[0031] Step 2.1, hypergraph construction: First, the i-th eigenvector f of the grape pest feature matrix F is i As a node n i , through the robust similarity matrix Identify the top k nodes that are most similar to each node and connect these nodes into hyperedges e i , construct an initial hypergraph association matrix H, defined as follows:

[0032]

[0033] Among them, N k (n i ) represents the set of the first k neighboring nodes similar to the current node; n j represents the jth node; H ij Represents the association relationship between the i-th node and the j-th node in the initial hypergraph association matrix H.

[0034] Next, similar to the process of constructing a robust similarity matrix, a Gaussian fitting threshold t is used. h =μ+β3*σ, to modify the hypergraph association matrix H, so as to effectively screen out nodes with significant associations and reduce noise, where β3 is a hyperparameter, μ represents the distribution mean, and σ represents the variance. Then, according to the Gaussian fitting threshold t h , the robust similarity matrix The initial hypergraph association matrix H is modified again to obtain a new hypergraph association matrix for:

[0035]

[0036] in, Represents the new hypergraph incidence matrix The association relationship between the i-th node and the j-th node in t h represents the Gaussian fitting threshold.

[0037] Step 2.2, Hypergraph Learning: Once the hypergraph association matrix After the construction is completed, the task of hypergraph learning is to aggregate the feature information from different neighborhood structures and generate the feature representation of each node. To achieve this goal, the hypergraph association matrix is ​​constructed. Introduce a general Laplace matrix L:

[0038]

[0039] In formula (7), D represents a diagonal matrix, each element of which represents the node degree, and M represents another diagonal matrix, each element of which is the size of each hyperedge.

[0040] The Laplacian matrix captures the similarity relationship between nodes and effectively extracts information from different modalities, thereby enhancing the robustness of node feature representation. The hypergraph convolutional layer of is defined as follows:

[0041] X (l+1) =LeakyReLU(LX (l) W (l+1) ) (8)

[0042] Among them, X (l) The hypergraph incidence matrix The representation in the lth layer of the hypergraph convolutional layer of X (l+1) Hypergraph Incidence Matrix The representation in the l+1th layer of the hypergraph convolutional layer; represents the weight matrix of the (l+1)th layer. Correspondingly, d (l) and d (l+1) They represent the dimensions of the nodes in the lth and (l+1)th layers respectively, and LeakyReLU represents the activation function.

[0043] The hypergraph incidence matrix The hypergraph convolution layer X (l+1) High-order information can be propagated through hyperedges, but the different importance between nodes and hyperedges is not considered. To solve this problem, a hypergraph attention mechanism module is introduced into the original hypergraph convolutional layer to make up for the different importance between nodes and hyperedges when constructing the hypergraph in step 2.1, so as to capture the complex relationship between nodes and their associated hyperedges. The hypergraph attention mechanism module measures the importance of nodes and their hyperedges by learning attention weights. Specifically:

[0044] First, an aggregation function is used to aggregate the associated nodes of the hyperedge to obtain the representation of each hyperedge e; the aggregation function is AGG(·):x e =AGG(x i ∣i∈N e ), where AGG(·) is the aggregation function, N e is the associated node of the hyperedge, x i represents the ith hyperedge. Then based on node n i Hyperedge e associated with the node j The final hypergraph association matrix is ​​calculated by

[0045]

[0046] in, Represents the final hypergraph incidence matrix is the association between the i-th node and the j-th node in ; σ(·) is a nonlinear activation function; ε i is the node n i is a set of hyperedges, P is the parameter matrix for calculating the hyperedge attention of nodes. sim(·) is a similarity function, which can be implemented as:

[0047]

[0048] Where [,||,] represents a connection, and a is used to calculate node n i Hyperedge e associated with the node j The hypergraph attention mechanism module enriches the hypergraph association matrix It replaces the original hypergraph association matrix in formulas (7) and (8). The final output of the hypergraph convolutional layer is represented as H I and H T , which effectively mines the high-order information of each node.

[0049] Furthermore, the step 3, loss function design, is specifically as follows: firstly, the data samples N of grape pest and disease images and texts are divided into labeled data samples, denoted as N l ; The unlabeled data sample is represented by N u Then, through steps 1 and 2, we can get H I and H T , in order to obtain the final grape pest image and text hash code B I and B T , we can learn the hash codes of grape pest images and texts by using the sign(·) function: B I =sign(H I ), B T =sign(B T ), where B I Represents the hash code of the grape pest image; B T Represents a text hash code for grape pests and diseases.

[0050] First, for the labeled data sample N l We design the intra-modality similarity preservation loss and the inter-modality similarity preservation loss to better maintain the semantic similarity between instances and generate a unified hash code B for both modalities. I and B T .

[0051] Step 3.1, intra-modal similarity preservation loss: Intra-modal similarity preservation aims to ensure that the projections of semantically similar data in the same modality in the hash space are as close as possible, while the projections of semantically dissimilar data should be as far apart as possible. To enhance the preservation effect of intra-modal similarity, two asymmetric negative log-likelihood pairwise loss functions are defined. First, the label similarity is calculated based on the labels of the labeled data samples of grape pest images and texts. Then, according to the unbinarized hash code B in each modality I and B T Half of the inner product between Last pair Logistic transformation is performed to ensure numerical stability, as follows:

[0052]

[0053] in, Represents the unbinarized hash code B within each modality I and B T The inner product between; · represents the inner product operation; is the unbinarized hash code B I and B T ; Represents the label similarity matrix. If samples i and j share at least one common semantic label, then we define Otherwise define N l represents a labeled data sample; β4 is a weight parameter for controlling the intra-modality similarity preservation loss, and the values ​​set for image and text modalities are different; Represents the similarity preservation loss within each modality; * represents image modality or text modality.

[0054] Step 3.2, inter-modality similarity preservation loss: In contrast, inter-modality similarity preservation aims to maintain the semantic similarity between different modalities. To this end, the present invention introduces an asymmetric negative log-likelihood pairwise loss. Similarly, the label similarity is first calculated based on the labels of the labeled data samples of grape pest images and texts. Then, according to the unbinarized hash code B of all image modalities I and the unbinarized hash code B of the text modal T Θ is calculated by taking half of the inner product between ij , and according to the unbinarized hash code B of all text modalities T and the unbinarized hash code B of the image modality I The inner product between them is used to calculate Φ ij , and finally ij and Φ ij Logistic transformation is performed to ensure numerical stability, as follows:

[0055]

[0056] in, Represents the unbinarized hash code B of all image modalities I and the unbinarized hash code B of the text modal T The inner product between; · represents the inner product operation; The hash code B representing all text modalities without binarization T and the unbinarized hash code B of the image modality I The inner product between l Indicates labeled data samples; represents the label similarity matrix; L inter Represents the inter-modality similarity preserving loss between image modality and text modality.

[0057] The final supervised loss is expressed as:

[0058] L1 = λ1 × L inter +λ2×L intra (13)

[0059] Among them, λ1 is the loss weight for preserving similarity between modalities; λ2 is the loss weight for preserving similarity within modality.

[0060] Next, for the unlabeled data sample N u Design a mean square error loss based on inter-modal and intra-modal cosine similarity, hash code B I and B T The learning is done by minimizing the robust similarity matrix It is achieved by reconstructing the inter-modality and intra-modality cosine similarity with the target hash code, as shown below:

[0061]

[0062] Among them, ρ1 is the trade-off parameter for balancing the reconstruction of the image modality; ρ2 is the trade-off parameter for balancing the reconstruction of the text modality; Used as a scale parameter to make reconstruction more flexible; N u represents an unlabeled data sample; represents the robust similarity matrix of unlabeled data samples; h represents the length of the hash code; L2 represents the mean square error loss based on inter-modal and intra-modal cosine similarity.

[0063] In order to avoid the interference of irrelevant elements on training, the similarity between hash codes with the same label is enhanced. The goal of the present invention is to minimize the difference between the diagonal elements in the cross-modal similarity matrix and 1. The error term can be defined as follows:

[0064]

[0065] Among them, L3 represents the difference loss between the diagonal elements in the cross-modal similarity matrix and 1; B I,i represents the i-th hash code of the image modality hash code; B T,i Represents the i-th hash code of the text modal hash code.

[0066] Furthermore, the objective function of step 4 is specifically: optimizing the output representation of the hypergraph convolutional layer according to the design of the loss function of step 3, and designing the objective function to learn the final hash code. The objective function is defined as:

[0067] L=L1+L2+L3 (16)

[0068] The final hash code is obtained by minimizing the objective function and the scaled hyperbolic tangent (tanh) function. The scaled hyperbolic tangent (tanh) function is based on a smoothing objective, making it suitable for the ambiguous gradient problem that occurs when the training process becomes more non-smooth.

[0069]

[0070] Where N is the data sample of grape pests and diseases images and texts; represents the set of non-negative real numbers; H * represents the final output H of the image modality hypergraph convolutional layer obtained in step 2.2 I and the final output H of the text modality hypergraph convolutional layer T ; I represents image mode; T represents text mode; α>0 is a scaling parameter, which is set to 1 at the beginning and increases as training progresses. When α is large enough, the network converges to the original hash coding problem with a symbolic activation function and generates accurate binary hash codes with minimal quantization error.

[0071] Innovation analysis of the present invention: (1) The present invention searches for cross-modal data and can receive cross-modal data with or without labels, not limited to image, text, and voice data. (2) The present invention constructs a robust similarity matrix to guide the learning of hash functions. (3) The present invention constructs a hypergraph-based attention convolution module, which aggregates input data through hypergraph convolution and attention mechanism, and can fully mine and utilize high-order semantic information in input samples.

[0072] The beneficial effects of the present invention are:

[0073] The present invention constructs a robust similarity matrix to guide the learning of hash functions, and at the same time constructs a hypergraph-based attention convolution module. By aggregating input data through hypergraph convolution and attention mechanism, the high-order semantic information in the input samples can be fully mined and utilized. Loss functions are designed separately for labeled data samples and unlabeled data samples to optimize hash codes, which can realize unsupervised model training and semi-supervised model learning with a small amount of labeled data. Finally, the final objective function is designed based on the loss function to learn the final hash code. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is the algorithm structure diagram;

[0075] Figure 2 This is a diagram of the algorithm steps. DETAILED DESCRIPTION

[0076] The implementation of the present invention is further described below in conjunction with cross-modal retrieval between images and texts on the Internet.

[0077] A semi-supervised cross-modal hashing method based on hypergraph attention convolution is implemented in the following steps:

[0078] 1. Construct a robust similarity matrix

[0079] The fc6 layer of the pre-trained neural network model VGG19 is used to extract the features of grape pests and diseases images, denoted as F I ; Text feature representation: Use bag-of-words (BoW) features to extract features from grape pest text data, denoted as F T The obtained grape pest image feature F I and text features F T , construct the similarity matrix N represents the data samples of grape pests and diseases images and texts, similarity matrix Each element S in i,j represents the similarity between the i-th grape pest sample and the j-th sample. First, the image features F of grape pests and diseases are calculated respectively. I and text features F T The intra-modal similarity matrix is ​​as follows:

[0080]

[0081] Among them, cos(,) represents cosine similarity; Represents the similarity between the i-th grape pest sample and the j-th sample in image features or text modality; represents the i-th grape pest sample feature in image mode or text mode; F j *represents the j-th grape pest and disease sample feature in image mode or text mode; i represents the image mode of grape pest and disease; T represents the text mode of grape pest and disease.

[0082] Then, the inter-modality similarity matrix between the grape pest image modality and the text modality is calculated as follows, denoted as S fusion :

[0083]

[0084] Among them, S I ,S T is the intra-modality similarity matrix calculated in formula (1). I S represents the similarity matrix of grape pest image modalities; T The similarity matrix of the text modality of grape pests and diseases is shown in Figure 2. fusion Combined, the final similarity matrix is ​​obtained

[0085] S=λ v S I +λ t S T +λ f S fusion (3)

[0086] Among them, λ v represents the weight parameter of the modal similarity matrix of grape pest and disease images; t Represents the weight parameter of the modal similarity matrix of grape pests and diseases text; f Represents the weight parameters of the inter-modality similarity matrix between the grape pest and disease image modality and the text modality.

[0087] In view of the differences in statistical characteristics and perspectives between the image modality and text modality of grape pests and diseases, the weight parameter λ v ,λ t , and λ f It is used to adaptively fuse the information of these two modalities.

[0088] Next, the similarity matrix obtained by formula (3) Two half-Gaussian distributions are used to fit the histogram of similarity between grape pest image features and text features to reduce redundant information in the similarity matrix. First, the similarity threshold and the difference threshold are calculated. The similarity threshold is set to The difference threshold is set to in and are the mean and variance of the right half Gaussian distribution and the left half Gaussian distribution, β1 is the similarity threshold parameter, and β2 is the difference threshold parameter. Then, the similarity matrix is Scaling to get the robust similarity matrix The formula is shown below.

[0089]

[0090] in, Represents the robust similarity matrix Each element in Specifically, it is the similarity between the i-th grape pest and disease sample and the j-th sample; S i,j Represents the similarity matrix Each element S in i,j , specifically the similarity between the i-th grape pest sample and the j-th sample

[0091] When the value of the similarity matrix S is greater than or equal to the similarity threshold t + , is corrected to +1; when the similarity matrix The value is less than or equal to the difference threshold t - When , it is corrected to -1, indicating dissimilarity, the similarity matrix The other values ​​remain unchanged.

[0092] 2. Hypergraph-based Attention Convolution Module

[0093] First, the i-th eigenvector f of the grape pest and disease feature matrix F is i As a node n i , through the robust similarity matrix Identify the top k nodes that are most similar to each node and connect these nodes into hyperedges e i , construct an initial hypergraph association matrix The definition is as follows:

[0094]

[0095] Among them, N k (n i ) represents the set of the first k neighboring nodes similar to the current node; n j represents the jth node; H ij Represents the initial hypergraph incidence matrix The association relationship between the i-th node and the j-th node in .

[0096] Next, similar to the process of constructing a robust similarity matrix, a Gaussian fitting threshold t is used. h =μ+β3*σ, to calculate the hypergraph association matrix Modification is performed to effectively screen out nodes with significant associations and reduce noise, where β3 is a hyperparameter, μ represents the distribution mean, and σ represents the variance. Then, according to the Gaussian fitting threshold th , the robust similarity matrix Hypergraph Incidence Matrix Make another correction to get the new hypergraph association matrix for:

[0097]

[0098] in, Represents the new hypergraph incidence matrix The association relationship between the i-th node and the j-th node in t h represents the Gaussian fitting threshold.

[0099] Once the hypergraph incidence matrix After the construction is completed, the task of hypergraph learning is to aggregate the feature information from different neighborhood structures and generate the feature representation of each node. To achieve this goal, the hypergraph association matrix is ​​constructed. Introduce a general Laplace matrix L:

[0100]

[0101] In formula (7), D represents a diagonal matrix, each element of which represents the node degree, and M represents another diagonal matrix, each element of which is the size of each hyperedge.

[0102] The Laplacian matrix captures the similarity relationship between nodes and effectively extracts information from different modalities, thereby enhancing the robustness of node feature representation. The hypergraph convolutional layer of is defined as follows:

[0103] X (l+1) =LeakyReLU(LX (l) W (l+1) ) (8)

[0104] Among them, X (l) The hypergraph incidence matrix The representation in the lth layer of the hypergraph convolutional layer of X (l+1) Hypergraph Incidence Matrix The representation in the l+1th layer of the hypergraph convolutional layer; represents the weight matrix of the (l+1)th layer. Correspondingly, d (l) and d (l+1) They represent the dimensions of the nodes in the lth and (l+1)th layers respectively, and LeakyReLU represents the activation function.

[0105] The hypergraph incidence matrix The hypergraph convolution layer X (l+1)High-order information can be propagated through hyperedges, but the different importance between nodes and hyperedges is not considered. To solve this problem, a hypergraph attention mechanism module is introduced into the original hypergraph convolutional layer to make up for the different importance between nodes and hyperedges when constructing the hypergraph in step 2, so as to capture the complex relationship between nodes and their associated hyperedges. The hypergraph attention mechanism module measures the importance of nodes and their hyperedges by learning attention weights. Specifically:

[0106] First, an aggregation function is used to aggregate the associated nodes of the hyperedge to obtain the representation of each hyperedge e; the aggregation function is AGG(·):x e =AGG(x i ∣i∈N e ), where AGG(·) is the aggregation function, N e is the associated node of the hyperedge, x i represents the ith hyperedge. Then based on node n i Hyperedge e associated with the node j The final hypergraph association matrix is ​​calculated by

[0107]

[0108] in, Represents the final hypergraph incidence matrix is the association between the i-th node and the j-th node in ; σ(·) is a nonlinear activation function; ε i is the node n i is a set of hyperedges, P is the parameter matrix for calculating the hyperedge attention of nodes. sim(·) is a similarity function, which can be implemented as:

[0109]

[0110] Where [, ∥,] represents a connection, and a is used to calculate node n i Hyperedge e associated with the node j The hypergraph attention mechanism module enriches the hypergraph association matrix It replaces the original hypergraph association matrix in formulas (7) and (8). The final output of the hypergraph convolutional layer is represented as H I and H T , which effectively mines the high-order information of each node.

[0111] 3. Loss function design

[0112] First, the data samples N of grape pest and disease images and texts are divided into labeled data samples, denoted as N l ; The unlabeled data sample is represented by N uThen, through steps 1 and 2, we can get H I and H T , in order to obtain the final grape pest image and text hash code B I and B T , we can learn the hash codes of grape pest images and texts by using the sign(·) function: B I =sign(H I ), B T =sign(B T ), where B I Represents the hash code of the grape pest image; B T Represents a text hash code for grape pests and diseases.

[0113] Then for the labeled data sample N l We design the intra-modality similarity preservation loss and the inter-modality similarity preservation loss to better maintain the semantic similarity between instances and generate a unified hash code B for both modalities. I and B T .

[0114] Intra-modality similarity preservation loss: Intra-modality similarity preservation aims to ensure that the projections of semantically similar data in the same modality are as close as possible in the hash space, while the projections of semantically dissimilar data should be as far apart as possible. To enhance the preservation effect of intra-modality similarity, two asymmetric negative log-likelihood pairwise loss functions are defined. First, the label similarity is calculated based on the labels of the labeled data samples of grape pest images and texts. Then, according to the unbinarized hash code B in each modality I and B T Half of the inner product between Last pair Logistic transformation is performed to ensure numerical stability, as follows:

[0115]

[0116] in, Represents the unbinarized hash code B within each modality I and B T The inner product between; · represents the inner product operation; is the unbinarized hash code B I and B T ; Represents the label similarity matrix. If samples i and j share at least one common semantic label, then we define Otherwise define N lrepresents a labeled data sample; β4 is a weight parameter for controlling the intra-modality similarity preservation loss, and the values ​​set for image and text modalities are different; Represents the similarity preservation loss within each modality; * represents image modality or text modality.

[0117] Inter-modality similarity preservation loss: In contrast, inter-modality similarity preservation aims to maintain the semantic similarity between different modalities. To this end, the present invention introduces an asymmetric negative log-likelihood pairwise loss. Similarly, the label similarity is first calculated based on the labels of the labeled data samples of grape pest images and texts. Then, according to the unbinarized hash code B of all image modalities I and the unbinarized hash code B of the text modal T Θ is calculated by taking half of the inner product between ij , and according to the unbinarized hash code B of all text modalities T and the unbinarized hash code B of the image modality I The inner product between them is used to calculate Φ ij , and finally ij and Φ ij Logistic transformation is performed to ensure numerical stability, as follows:

[0118]

[0119] in, Represents the unbinarized hash code B of all image modalities I and the unbinarized hash code B of the text modal T The inner product between; · represents the inner product operation; The hash code B representing all text modalities without binarization T and the unbinarized hash code B of the image modality I The inner product between l Indicates labeled data samples; represents the label similarity matrix; L inter Represents the inter-modality similarity preserving loss between image modality and text modality.

[0120] The final supervised loss is expressed as:

[0121] L1 = λ1 × L inter +λ2×L intra (13)

[0122] Among them, λ1 is the loss weight for preserving similarity between modalities; λ2 is the loss weight for preserving similarity within modality.

[0123] Next, for the unlabeled data sample N uDesign a mean square error loss based on inter-modal and intra-modal cosine similarity, hash code B I and B T The learning is done by minimizing the robust similarity matrix It is achieved by reconstructing the inter-modality and intra-modality cosine similarity with the target hash code, as shown below:

[0124]

[0125] Among them, ρ1 is the trade-off parameter for balancing the reconstruction of the image modality; ρ2 is the trade-off parameter for balancing the reconstruction of the text modality; Used as a scale parameter to make reconstruction more flexible; N u represents an unlabeled data sample; represents the robust similarity matrix of unlabeled data samples; h represents the length of the hash code; L2 represents the mean square error loss based on inter-modal and intra-modal cosine similarity.

[0126] In order to avoid the interference of irrelevant elements on training, the similarity between hash codes with the same label is enhanced. The goal of the present invention is to minimize the difference between the diagonal elements in the cross-modal similarity matrix and 1. The error term can be defined as follows:

[0127]

[0128] Among them, L3 represents the difference loss between the diagonal elements in the cross-modal similarity matrix and 1; B I,i represents the i-th hash code of the image modality hash code; B T,i Represents the i-th hash code of the text modal hash code.

[0129] 4. Objective Function

[0130] According to the design of the loss function in step 3, the output representation of the hypergraph convolution layer is optimized, and the objective function is designed to learn the final hash code. The objective function is defined as:

[0131] L=L1+L2+L3 (16)

[0132] The final hash code is obtained by minimizing the objective function and the scaled hyperbolic tangent (tanh) function. The scaled hyperbolic tangent (tanh) function is based on a smoothing objective, making it suitable for the ambiguous gradient problem that occurs when the training process becomes more non-smooth.

[0133]

[0134] Where N is the data sample of grape pests and diseases images and texts; represents the set of non-negative real numbers; H *represents the final output H of the image modality hypergraph convolutional layer obtained in step 2 I and the final output H of the text modality hypergraph convolutional layer T ; I represents image mode; T represents text mode; α>0 is a scaling parameter, which is set to 1 at the beginning and increased as training progresses. When α is large enough, the network converges to the original hash coding problem with a symbolic activation function and generates an accurate binary hash code with minimal quantization error.

[0135] Specific application examples are as follows:

[0136] The effect of the present invention is illustrated by applying it to two image-text pair data sets.

[0137] (1) Dataset

[0138] MIRFLICKR-25K consists of 25,000 images collected from the Flickr website. Each image is associated with several text tags. Therefore, each point is an image-text pair manually annotated with 24 unique labels. The text of each point is represented as a 1,386-dimensional bag-of-words (BoW) vector. 20,015 points with at least 1 label and 1 text label are selected for experiments. 2,000 samples are randomly selected as the query set and 18,015 samples are used as the database. 10,000 samples are randomly selected from the database for training, of which 30% of the training data points are randomly selected as labeled data.

[0139] NUS-WIDE contains 260,648 web images, some of which are associated with text labels. Each image is annotated with one or more of 81 concept labels. We selected 195,834 image-text pairs that belong to the 21 most frequent concepts. The text of each point is represented as a 1,000-dimensional bag-of-words vector. 2,100 samples were randomly selected as the query set, and the remaining 193,734 samples were used as the database. 10,500 samples were randomly selected from the database for training, of which 30% of the training data points were randomly selected as labeled data.

[0140] (2) Experimental setup

[0141] The invention is implemented using PyTorch on a server equipped with a 13th generation Intel i9-13900KF CPU (5.80GHz) and an NViDiA4090 GPU. For all experiments, the mini-batch size is equal to 64, the momentum of the mini-batch SGD optimizer is 0.9%, and the weight decay is 0.0005. For the NU_WIDE dataset, we cross-validated the hyperparameters and set k=4, λ v =0.4,λ t =0.3,λf =0.3, φ=1.5, ρ1=0.3, ρ2=0.3, β1=4, β2=1, β3=-0.45, that is, t + =0.55, t - =0.17, t h = -0.30, β4 is 4 for image modality and 3 for text modality. For MiRFLiCKR-25K dataset, we set k = 5, λ v =0.5,λ t =0.1,λ f =0.4, φ=1.5, ρ1=0.15, ρ2=0.15, β1=4, β2=1, β3=1.34, that is, t + =0.64, t - =0.13, t h =0.27, β4 is 4 for image modality and 3 for text modality. At the same time, for the supervised loss weight, λ1=λ2=2 is set for both MIRFLICKR-25K and NUS-WIDE.

[0142] (3) Evaluation indicators

[0143] The mean average precision (MAP) evaluation index was selected to evaluate the retrieval quality. Considering the actual application scenario, the present invention sets MAP@R to R=all, where the hash code length is set to 16 bits, 32 bits, and 64 bits respectively to check the application effects of different lengths.

[0144] (4) Effect comparison

[0145] The present invention sets the percentage of label samples in the training set to 30%, and the results of the compared semi-supervised hashing methods MGCH, SSCH, TS3H, and different bits of GCSCH are shown in Tables 1 and 2.

[0146] Table 1 Performance comparison of MAP scores on the MIRFLICKR-25K dataset with 30% labeled samples (MAP@all)

[0147]

[0148] Table 2. Semi-supervised cross-modal retrieval results on the NUS-WIDE dataset with 30% labeled samples (MAP@all)

[0149]

[0150] The bold data in the table show the best results. Therefore, from the overall application effect, the method proposed in the present invention performs well in making full use of unlabeled image and text data for semi-supervised cross-modal retrieval, and surpasses previous methods in the accuracy of cross-modal retrieval.

[0151] The present invention is a semi-supervised cross-modal hashing method based on hypergraph attention convolution. Through a robust similarity matrix and a hypergraph attention convolution module, the important role of unlabeled data in maintaining semantic structure is fully utilized, thereby effectively improving the learning performance and retrieval ability of the semi-supervised cross-modal retrieval model. This method successfully solves the problem of difficulty and high cost in labeling labeled data.

[0152] The above examples are only embodiments of the present invention and do not limit the protection scope of the present invention. It should be noted that for those skilled in the art, various modifications and improvements can be made without departing from the concept of the present invention, and these modifications and improvements are within the protection scope of the present invention.

Claims

1. A semi-supervised cross-modal hashing method based on hypergraph attention convolution, characterized in that: The semi-supervised cross-modal hashing method first constructs a robust similarity matrix; secondly, a hypergraph attention convolution module is constructed, which generates an embedding vector through hypergraph convolution and input data attention mechanism, captures important high-order semantics, and designs loss functions separately for labeled data samples and unlabeled data samples to optimize hash codes, which can realize unsupervised model training and semi-supervised model learning with a small amount of labeled data; finally, the final objective function is designed based on the loss function to learn the final hash code; specifically, the following steps are included: Step 1, construct a robust similarity matrix; Step 2: Hypergraph-based attention convolution module; Step 3: Design loss function; Step 4: Design the objective function; According to the loss function in step 3, the output representation of the hypergraph convolution layer is optimized, and the objective function is designed to learn the final hash code. The final hash code is obtained by minimizing the objective function and scaling the hyperbolic tangent function.

2. According to claim 1, a semi-supervised cross-modal hashing method based on hypergraph attention convolution is characterized in that: The step 1, constructing a robust similarity matrix, is specifically as follows: Step 1.1 Image feature extraction: The fc6 layer of the pre-trained neural network model VGG19 is used to extract the image features of grape pests and diseases, denoted as F I ; Text feature representation: Use bag-of-words BoW features to extract features from grape pest text data, denoted as F T ; Step 1.2 constructs a similarity matrix: The grape pest image features F obtained in step 1.1 I and text features F T , construct the similarity matrix S, each element S in the similarity matrix S i,j Represents the similarity between the i-th grape pest and disease sample and the j-th sample.

3. According to claim 2, a semi-supervised cross-modal hashing method based on hypergraph attention convolution is characterized in that: The step 1.2 is as follows: First, calculate the image features F of grape pests and diseases respectively I and text features F T The intra-modal similarity matrix is ​​as follows: Among them, cos(,) represents cosine similarity; Represents the similarity between the i-th grape pest sample and the j-th sample in image features or text modality; represents the i-th grape pest sample feature in image mode or text mode; F j * represents the jth grape pest sample feature in image mode or text mode; i represents the image mode of grape pest; T represents the text mode of grape pest; Then, the inter-modality similarity matrix between the grape pest image modality and the text modality is calculated as follows, denoted as S fusion : Among them, S I ,S T is the intra-modality similarity matrix calculated in formula (1); S I S represents the similarity matrix of grape pest image modalities; T The similarity matrix of the text mode of grape pests and diseases is shown; the similarity matrix within the mode is compared with S fusion Combined, we get the final similarity matrix S: S=λ v S I +λ t S T +λ f S fusion (3) Among them, λ v represents the weight parameter of the modal similarity matrix of grape pest and disease images; t represents the weight parameter of the modal similarity matrix of grape pests and diseases text; f represents the weight parameter of the inter-modality similarity matrix between the grape pest and disease image modality and the text modality; Next, the similarity matrix S obtained by formula (3) is used to fit the histogram of the similarity between the image features and text features of grape pests and diseases. First, the similarity threshold and the difference threshold are calculated, and the similarity threshold is set to The difference threshold is set to in and are the mean and variance of the right half Gaussian distribution and the left half Gaussian distribution, β1 is the similarity threshold parameter, and β2 is the difference threshold parameter; according to the similarity threshold and difference threshold, the similarity matrix S is scaled to obtain the robust similarity matrix The formula is as follows: in, Represents the robust similarity matrix Each element in Specifically, it is the similarity between the i-th grape pest and disease sample and the j-th sample; S i,j Represents each element S in the similarity matrix S i,j , specifically the similarity between the i-th grape pest and disease sample and the j-th sample; When the value of the similarity matrix S is greater than or equal to the similarity threshold t + , it is corrected to +1; when the value of the similarity matrix S is less than or equal to the difference threshold t - , is corrected to -1, indicating dissimilarity, and the other values ​​of the similarity matrix S remain unchanged.

4. According to claim 3, a semi-supervised cross-modal hashing method based on hypergraph attention convolution is characterized in that: The step 2 is based on the hypergraph attention convolution module, specifically: Step 2.1, hypergraph construction: First, the i-th eigenvector f of the grape pest and disease feature matrix F is i As a node n i , through the robust similarity matrix Identify the top k nodes that are most similar to each node and connect these nodes into hyperedges e i , construct an initial hypergraph association matrix H, defined as follows: Among them, N k (n i ) represents the set of the first k neighboring nodes similar to the current node; n j represents the jth node; H ij Represents the association relationship between the i-th node and the j-th node in the initial hypergraph association matrix H; Next, similar to the process of constructing a robust similarity matrix, a Gaussian fitting threshold t is used. h =μ+β3*σ, the hypergraph association matrix H is modified, where β3 is a hyperparameter, μ represents the distribution mean, and σ represents the variance; according to the Gaussian fitting threshold t h , the robust similarity matrix The initial hypergraph association matrix H is modified again to obtain a new hypergraph association matrix for: in, Represents the new hypergraph incidence matrix The association relationship between the i-th node and the j-th node in t h represents the Gaussian fitting threshold; Step 2.2, Hypergraph Learning: Hypergraph Incidence Matrix After the construction is completed, the feature information from different neighborhood structures is aggregated through hypergraph learning, and the feature representation of each node is generated; the hypergraph association matrix constructed is Introduce a general Laplace matrix L: In formula (7), D represents a diagonal matrix, each element represents the node degree, and M represents another diagonal matrix, each element of which is the size of each hyperedge; Hypergraph Incidence Matrix The hypergraph convolutional layer of is defined as follows: X (l+1) =LeakyReLU(LX (l) IN (l+1) ) (8) Among them, X (l) The hypergraph incidence matrix The representation in the lth layer of the hypergraph convolutional layer of X (l+1) Hypergraph Incidence Matrix The representation in the l+1th layer of the hypergraph convolutional layer; represents the weight matrix of the (l+1)th layer; accordingly, d (l) and d (l+1) They represent the dimensions of the nodes in the lth and (l+1)th layers respectively, and LeakyReLU represents the activation function; The hypergraph incidence matrix The hypergraph convolution layer X (l+1) High-order information is propagated through hyperedges, and a hypergraph attention mechanism module is introduced into the original hypergraph convolutional layer. The hypergraph attention mechanism module measures the importance of nodes and their hyperedges by learning attention weights.

5. According to claim 4, a semi-supervised cross-modal hashing method based on hypergraph attention convolution is characterized in that: In step 2.2, the specific process of the hypergraph attention mechanism module measuring the importance of nodes and their hyperedges by learning the attention weights is as follows: First, an aggregation function is used to aggregate the associated nodes of the hyperedge to obtain the representation of each hyperedge e; the aggregation function is AGG(·):x e =AGG(x i ∣i∈N e ), where AGG(·) is the aggregation function, N e is the associated node of the hyperedge, x i Represents the i-th hyperedge; then based on node n i Hyperedge e associated with the node j The final hypergraph association matrix is ​​calculated by in, Represents the final hypergraph incidence matrix is the association between the i-th node and the j-th node in ; σ(·) is a nonlinear activation function; ε i is the node n i is a set of hyperedges, P is the parameter matrix for calculating the hyperedge attention of nodes; sim(·) is a similarity function, which can be implemented as: Where [, ∥,] represents a connection, and a is used to calculate node n i Hyperedge e associated with the node j The weight vector of the similarity between them; the hypergraph attention mechanism module enriches the hypergraph association matrix It replaces the original hypergraph association matrix in formulas (7) and (8); the final output of the hypergraph convolutional layer is represented as H I and H T , which effectively mines the high-order information of each node.

6. The method of semi-supervised cross-modal hashing based on hypergraph attention convolution according to claim 4, characterized in that: The loss function of step 3 is designed as follows: First, the data samples N of grape pest and disease images and texts are divided into labeled data samples, represented as N l ; The unlabeled data sample is represented by N u ; Then, H is obtained through steps 1 and 2 I and H T , in order to obtain the final grape pest image and text hash code B I and B T , by using the sign(·) function to learn the hash codes of grape pest images and texts: B I =sign(H I ), B T =sign(B T ), where B I Represents the hash code of the grape pest image; B T Represents the text hash code of grape pests and diseases; For labeled data samples N l Design intra-modality similarity preservation loss and inter-modality similarity preservation loss, and generate unified hash codes for both modalities. I and B T .

7. The method of semi-supervised cross-modal hashing based on hypergraph attention convolution according to claim 6, characterized in that: In step 3, the intra-modality similarity preservation loss and the inter-modality similarity preservation loss are designed as follows: Step 3.1, intra-modal similarity preservation loss: define two asymmetric negative log-likelihood pairwise loss functions, first calculate the label similarity based on the labels of the labeled data samples of grape pest images and texts Then, according to the unbinarized hash code B in each modality I and B T Half of the inner product between Last pair Logistic transformation is performed to ensure numerical stability, as follows: in, Represents the unbinarized hash code B within each modality I and B T The inner product between; · represents the inner product operation; is the unbinarized hash code B I and B T ; Represents the label similarity matrix. If samples i and j share at least one common semantic label, then we define Otherwise define N l represents a labeled data sample; β4 is a weight parameter for controlling the intra-modality similarity preservation loss, and the values ​​set for image and text modalities are different; Indicates the similarity preservation loss within each modality; * indicates image modality or text modality; Step 3.2, inter-modal similarity preservation loss: an asymmetric negative log-likelihood pairwise loss is introduced. Similarly, the label similarity is first calculated based on the labels of the labeled data samples of grape pest images and texts. Then, according to the unbinarized hash code B of all image modalities I and the unbinarized hash code B of the text modal T Θ is calculated by taking half of the inner product between ij , and according to the unbinarized hash code B of all text modalities T and the unbinarized hash code B of the image modality I The inner product between them is used to calculate Φ ij , and finally ij and Φ ij Logistic transformation is performed to ensure numerical stability, as follows: in, Represents the unbinarized hash code B of all image modalities I and the unbinarized hash code B of the text modal T The inner product between; · represents the inner product operation; The hash code B representing all text modalities without binarization T and the unbinarized hash code B of the image modality I The inner product between l Indicates labeled data samples; represents the label similarity matrix; L inter represents the inter-modality similarity preserving loss between image modality and text modality; The final supervised loss is expressed as: L1=λ1×L inter +λ2×L intra (13) Among them, λ1 is the loss weight for preserving the similarity between modalities; λ2 is the loss weight for preserving the similarity within modality; For the unlabeled data sample N u Design a mean square error loss based on inter-modal and intra-modal cosine similarity, hash code B I and B T The learning is done by minimizing the robust similarity matrix It is achieved by reconstructing the inter-modality and intra-modality cosine similarity with the target hash code, as shown below: Among them, ρ1 is the trade-off parameter for balancing the reconstruction of the image modality; ρ2 is the trade-off parameter for balancing the reconstruction of the text modality; Used as a scale parameter to make reconstruction more flexible; N u represents an unlabeled data sample; represents the robust similarity matrix of unlabeled data samples; h represents the length of the hash code; L2 represents the mean square error loss based on inter-modal and intra-modal cosine similarity; The goal is to minimize the difference between the diagonal elements in the cross-modal similarity matrix and 1, resulting in an error term.

8. The semi-supervised cross-modal hashing method based on hypergraph attention convolution according to claim 7, characterized in that: The error term is defined as follows: Among them, L3 represents the difference loss between the diagonal elements in the cross-modal similarity matrix and 1; B I,i represents the i-th hash code of the image modality hash code; B T,i Represents the i-th hash code of the text modal hash code.

9. The method of semi-supervised cross-modal hashing based on hypergraph attention convolution according to claim 1, characterized in that: In step 4, the objective function is defined as: L=L1+L2+L3 (16).

10. The method of semi-supervised cross-modal hashing based on hypergraph attention convolution according to claim 1, characterized in that: The scaled hyperbolic tangent function described is based on a smoothing objective, making it suitable for the ambiguous gradient problem that arises when the training process becomes more non-smooth; Where N is the data sample of grape pests and diseases images and texts; represents the set of non-negative real numbers; H * represents the final output H of the image modality hypergraph convolutional layer obtained in step 2.2 I and the final output H of the text modality hypergraph convolutional layer T ; I represents image mode; T represents text mode; α>0 is a scaling parameter, α=1 is set at the beginning and α is increased as the training progresses.

Images