Dynamic cross-modal hashing retrieval method for concept drift

Abstract

The application discloses a kind of dynamic cross-modal hash retrieval methods for concept drift, comprising: feature extraction multi-modal data, obtain text feature and image feature;Carry out label regression learning, label dragging learning and semantic embedding learning, reduce the information imbalance between image data and text data, strengthen multi-modal information interaction, and map multi-modal data features to hash space, train to obtain hash function and hash code;The weight of the graph-text similarity relationship of hash function and hash code, the variance weight of hash code, the pairing weight of hash code is calculated to the weight of the hash function, and the optimal hash function and hash code are selected;Similarity is calculated according to hash code, and sorting is carried out.The application solves the problem of old and new information imbalance of single table and the catastrophic forgetting problem occurring with continuous accumulation.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal retrieval, and in particular to a dynamic cross-modal hash retrieval method oriented to concept drift. Background Art

[0002] With the widespread and booming development of the internet, multimodal data such as images, text, and videos has experienced explosive growth. Efficient and accurate cross-modal retrieval within this massive amount of diverse data is a current research hotspot in the field of information retrieval. Hash retrieval, with its sublinear retrieval time complexity and extremely low space overhead, has become the primary approach for solving the problem of nearest neighbor retrieval in large-scale data. In real-world environments, data often emerges dynamically over time. To better adapt cross-modal hash retrieval to these dynamic data environments, dynamic cross-modal retrieval methods have been proposed.

[0003] While significant progress has been made in dynamic cross-modal hash retrieval, existing dynamic cross-modal methods fail to consider concept drift and perform poorly in data scenarios with concept drift. To address this, a multi-hash table structure is designed to address the imbalance between new and old information and the catastrophic forgetting issues inherent in single-table structures. This multi-hash table structure uses weights for image-text similarity, hash code variance, and hash code pairing to assign different weights to the multiple hash tables, maximizing their effectiveness. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings and deficiencies of the existing technology and propose a dynamic cross-modal hash retrieval method for concept drift, which can solve the imbalance problem of new and old information in a single table and the catastrophic forgetting problem that occurs with continuous accumulation.

[0005] To achieve the above objectives, the present invention provides a technical solution: a dynamic cross-modal hash retrieval method for concept drift, comprising the following steps:

[0006] S1: Acquire multimodal data, including image data and text data, perform feature extraction on the multimodal data, extract corresponding image features and text features, and perform centralization operations on the image features and text features;

[0007] S2: Image and text features after the centering operation are processed, and the radial basis kernel function is used to capture the potential nonlinear features in their feature space to obtain image and text features with richer semantic information. The image and text features are divided into M batches, and the data appears in batches, thus simulating a dynamic data environment. As the image and text features appear in batches, the distribution of the data changes, and concept drift occurs.

[0008] S3: For each batch of image and text features, label regression learning, label drag learning, and semantic embedding learning are performed to reduce the information imbalance between image and text data, strengthen multimodal information interaction, and map image and text features to a hash space to train hash functions and hash codes.

[0009] S4: Based on the obtained hash function, the weight of the hash function is calculated by the image-text similarity relationship weight, the hash code variance weight and the hash code pairing weight; the image-text similarity relationship weight is calculated by the similarity matrix, using the Hamming distance as an indicator, so that the Hamming distance between similar image features and text features is as small as possible, and the Hamming distance between dissimilar image features and text features is as large as possible; the hash code variance weight is calculated by respectively calculating the variance of the image data hash code and the variance of the text data hash code, so that the hash code variance is maximized, the entropy is highest, and the most information is contained, thereby obtaining the optimal hash function The hash code pairing weight calculates the similarity between the hash code of the image data and the hash code of the text data. The higher the hash code similarity, the better the hash function effect. Then, the weights of the NUM saved hash functions are calculated by the image-text similarity relationship weight, the hash code variance weight, and the hash code pairing weight. Among the NUM+1 hash functions, the first NUM hash functions with the highest weights and their hash codes are retained. These NUM hash functions and their hash codes constitute a multi-hash table mechanism, which can solve the imbalance problem of new and old information in a single hash table and the catastrophic forgetting problem that occurs with continuous accumulation.

[0010] S5: Calculate similarity based on the obtained NUM hash codes, sort them, and select the top n candidate set instances as the final query results.

[0011] Furthermore, in step S1, the VGG19 neural network is used to extract image features for image data, and the BOW model is used to extract text features for text data.

[0012] Furthermore, in step S2, the radial basis kernel function Φ is used on the image features and text features to capture the potential nonlinear features in the feature space and obtain image features and text features with richer semantic information. In the simulated dynamic data environment, there is a concept drift phenomenon.

[0013] Furthermore, the specific operation steps of step S3 are as follows:

[0014] S31: Using label regression learning, hash codes can learn label information, so that image data and text data with more similar labels have more similar hash codes, thereby reducing the information imbalance between image data and text data, strengthening multimodal information interaction, and calculating the loss function loss1 of label regression learning:

[0015]

[0016] W T W=I

[0017] Where H represents the hash code, L represents the label, W represents the mapping function from the public real-valued space to the hash space and is orthogonal, I is the identity matrix, || || F represents the F-norm;

[0018] S32: Use label drag learning to increase the difference between data with different labels, thereby learning more discriminative hash codes and calculating the loss function loss2 of label drag learning:

[0019]

[0020] Where D represents the direction of label dragging, and A represents the amplitude of label dragging;

[0021] S33: Using semantic embedding learning, the image features processed by the radial basis kernel function are mapped to a common real-valued space. The image data distribution loss function in the common real-valued space is loss. image :

[0022]

[0023]

[0024] Where t represents the time t, image represents the image-related parameters, S is the data distribution in the public real-valued space, and R t,image is the image feature-public real-valued space mapping function, Φ is the radial basis kernel function, Represents the image features that appear at time t; Represents R t,image orthogonal;

[0025] S34: The image data distribution in the public real-valued space is connected to the label through S=W(L+D⊙A), and the image data distribution loss function in the public real-valued space is converted into the image data label loss function loss image2 :

[0026]

[0027] Since W is an orthogonal matrix, loss image2 Equivalent to the loss function loss image3 :

[0028]

[0029] S35: Using semantic embedding learning, the text features processed by the radial basis kernel function are mapped to a common real-valued space. The text data distribution loss function in the common real-valued space is loss. text :

[0030]

[0031]

[0032] In the formula, text represents text-related parameters, R t,text is the text feature-public real-valued space mapping function, Represents the text features that appear at time t; Represents R t,text orthogonal;

[0033] S36: The text data distribution in the public real-valued space is connected to the label through S=W(L+D⊙A), and the text data distribution loss function in the public real-valued space is converted into a text label loss function loss text2 :

[0034]

[0035] Since W is an orthogonal matrix, loss text2 Equivalent to the loss function loss text3 :

[0036]

[0037] S37: The final loss function loss is:

[0038] loss=loss2+loss image3 +loss text3 .

[0039] Furthermore, the specific operation steps of step S4 are as follows:

[0040] S41: Based on the hash function obtained from the loss function, the image-text similarity relationship weight γ is calculated so that the Hamming distance between similar image-text pairs is as close as possible, and the Hamming distance between dissimilar image-text pairs is as far as possible. The image-text similarity relationship weight γ is expressed as follows:

[0041]

[0042] Where x j is the jth image feature, x k is the kth text feature, f(x j ,x k) represents the Hamming distance between image features and text features in the hash space, and N represents the image feature x j With text feature x k With different label information, P represents the image feature x j With text feature x k Have the same label information;

[0043] S42: Based on the hash function obtained from the loss function, calculate the hash code variance weight δ to maximize the variance so that the hash code contains the most information. The hash code variance weight δ is expressed as follows:

[0044]

[0045]

[0046] Where, δ image is the image variance, δ text is the text variance, B represents the number of hash code bits, σ b is the variance of the b-th hash code;

[0047] S43: Based on the hash function obtained from the loss function, the hash code pairing weight λ is calculated so that the hash codes of image-text pairs with the same label are more similar. The hash code pairing weight λ is expressed as follows:

[0048]

[0049] Where H image Represents the hash code of the image data calculated by the hash function, H text Represents the hash code of the text data calculated by the hash function, xor is the exclusive OR operation, and Hsize is the hash code size;

[0050] S44: Finally, the weight formula is:

[0051] weight=γ+δ image +δ text +λ

[0052] S45: At time t, the newly arrived data X is trained by the loss function loss t,image and X t,text Get the hash function and hash code, then perform the weight formula weight calculation to get the weight weight1, then calculate the weights of the NUM hash functions saved at time [1, t-1] using the weight formula weight, compare all the NUM+1 weights at time t, eliminate the hash function with the smallest weight, and save the remaining NUM hash functions and their hash codes.

[0053] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0054] 1. The present invention designs a label regression learning mechanism to reduce the information imbalance between image data and text data and enhance multimodal information interaction.

[0055] 2. The present invention designs a label dragging learning mechanism to increase the difference between data with different labels, thereby learning a more discriminative hash code.

[0056] 3. The present invention designs a multi-hash table mechanism in a dynamic cross-modal data environment to solve the imbalance problem of new and old information in a single hash table and the catastrophic forgetting problem that occurs with continuous accumulation.

[0057] In summary, this invention can use the radial basis kernel function to obtain semantically richer image and text features, enhancing intermodal information complementarity. Through label regression learning, label dragging learning, and semantic embedding learning, it reduces the information imbalance between image and text data and strengthens multimodal information interaction. By determining the weights of multiple hash tables using image-text similarity relationship weights, hash code variance weights, and hash code pairing weights, it addresses the imbalance between new and old information in a single table and the catastrophic forgetting that occurs with continuous accumulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Learn the logic flow diagram for feature extraction and hashing functions.

[0059] Figure 2 This is a diagram of the hash table weight calculation and hash table screening logic flow. DETAILED DESCRIPTION

[0060] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0061] like Figure 1 As shown, this embodiment provides a dynamic cross-modal hash retrieval method for concept drift, using VGG19 and BOW as the base model for feature extraction, which includes the following steps:

[0062] 1) Cross-modal feature learning:

[0063] For image feature extraction, VGG19 is used to gradually abstract and extract low-level to high-level features of the image by stacking multiple layers of small-size convolutional layers and pooling layers.

[0064] For text feature extraction, the BOW model is used to learn text features by counting the frequency of occurrence of each word in the text.

[0065] 2) Nonlinear feature learning and feature batching:

[0066] For image and text features, the radial basis kernel function is used to capture potential nonlinear characteristics in the feature space, resulting in image and text features with richer semantic information. The image and text features are then divided into M batches, and the data is presented in batches, thus simulating a dynamic data environment.

[0067] 3) Hash function and hash code learning:

[0068] like Figure 1 As shown, the present invention designs label regression learning, label drag learning and semantic embedding learning to reduce the information imbalance between image features and text, strengthen multimodal information interaction, and map multimodal data features to hash space, and train to obtain hash functions and hash codes, which includes the following steps:

[0069] 3.1) Using label regression learning, hash codes can learn label information, so that image data and text data with more similar labels have more similar hash codes, thereby reducing the information imbalance between image data and text data and strengthening multimodal information interaction. Calculate the loss function loss1 of label regression learning:

[0070]

[0071] W T W=I

[0072] Where H represents the hash code, L represents the label, W represents the mapping function from the public real-valued space to the hash space and is orthogonal, I is the identity matrix, || || F represents the F-norm.

[0073] 3.2) Using label drag learning, we increase the difference between data with different labels, thereby learning more discriminative hash codes and calculating the loss function loss2 of label drag learning:

[0074]

[0075] Where D represents the direction of label dragging, and A represents the amplitude of label dragging.

[0076] 3.3) Using semantic embedding learning, the image features processed by the radial basis kernel function are mapped to a common real-valued space. The image data distribution loss function in the common real-valued space is image :

[0077]

[0078]

[0079] Where t represents the time t, image represents the image-related parameters, S is the data distribution in the public real-valued space, and R t,image is the image feature-public real-valued space mapping function, Φ is the radial basis kernel function, Represents the image features that appear at time t. Represents R t,image Orthogonal.

[0080] 3.4) The image data distribution in the public real-valued space is linked to the label through S = W(L + D⊙A). The image data distribution loss function in the public real-valued space is converted into the image data label loss function loss image2 :

[0081]

[0082] Since W is an orthogonal matrix, loss image2 Equivalent to the loss function loss image3 :

[0083]

[0084] 3.5) Using semantic embedding learning, the text features processed by the radial basis kernel function are mapped to a common real-valued space. The text data distribution loss function in the common real-valued space is loss text :

[0085]

[0086]

[0087] In the formula, text represents text-related parameters, R t,text is the text feature-public real-valued space mapping function, Represents the text features that appear at time t. Represents R t,text Orthogonal.

[0088] 3.6) The text data distribution in the public real-valued space is connected to the label through S = W(L + D⊙A), and the text data distribution loss function in the public real-valued space is converted into a text label loss function loss text2 :

[0089]

[0090] Since W is an orthogonal matrix, loss text2 Equivalent to the loss function loss text3 :

[0091]

[0092] 3.7) The final loss function loss is:

[0093] loss=loss2+loss image3 +loss text3 .

[0094] 4) If Figure 2 As shown, the present invention calculates the weight of the hash function based on the obtained hash function using the image-text similarity relationship weight, hash code variance weight, and hash code pairing weight. The image-text similarity relationship weight is calculated using the similarity matrix, using the Hamming distance as an indicator to minimize the Hamming distance between similar image features and text features and maximize the Hamming distance between dissimilar image features and text features. The hash code variance weight is calculated by separately calculating the variance of the image data hash code and the variance of the text data hash code to maximize the hash code variance, highest entropy, and the most information, thereby obtaining the optimal hash function. The hash code pairing weight calculates the similarity between the image data hash code and the text data hash code. The higher the hash code similarity, the better the hash function effect. The weights of the NUM saved hash functions are then calculated using the image-text similarity relationship weight, hash code variance weight, and hash code pairing weight. Among the NUM+1 hash functions, the first NUM hash functions with the highest weights and their hash codes are retained. These NUM hash functions and their hash codes constitute a multi-hash table mechanism.

[0095] The multi-hash table mechanism solves the imbalance problem of new and old information in a single hash table and the catastrophic forgetting problem caused by continuous accumulation. It can also better deal with the concept drift problem. The specific operation steps are as follows:

[0096] 4.1) Based on the hash function obtained from the loss function, the image-text similarity weight is calculated so that the Hamming distance between similar image-text pairs is as close as possible, and the Hamming distance between dissimilar image-text pairs is as far as possible. The image-text similarity weight γ is expressed as follows:

[0097]

[0098] Where x j is the jth image feature, x k is the kth text feature, f(x j ,x k ) represents the Hamming distance between image features and text features in the hash space, and N represents the image feature x j With text feature x k With different label information, P represents the image feature x j With text feature x k Have the same tag information.

[0099] 4.2) Based on the hash function obtained from the loss function, calculate the hash code variance weight to maximize the variance and make the hash code contain the most information. The hash code variance weight δ is expressed as follows:

[0100]

[0101]

[0102] Where, δ image is the image variance, δ text is the text variance, B represents the number of hash code bits, σ b is the b-th hash code variance.

[0103] 4.3) Based on the hash function obtained from the loss function, calculate the hash code pairing weight so that the hash codes of image-text pairs with the same label are more similar. The hash code pairing weight λ is expressed as follows:

[0104]

[0105] Where H image Represents the hash code of the image data calculated by the hash function, H text Represents the hash code of the text data calculated by the hash function, xor is the exclusive OR operation, and Hsize is the hash code size.

[0106] 4.4) Finally, the weight formula is:

[0107] weight=γ+δ image +δ text +λ

[0108] 4.5) At time t, the newly arrived data X is trained using the loss function loss t,image and X t,text Get the hash function and hash code, then perform the weight formula weight calculation to get the weight weight1, then calculate the weights of the NUM hash functions saved at time [1, t-1] using the weight formula weight, compare all the NUM+1 weights at time t, eliminate the hash function with the smallest weight, and save the remaining NUM hash functions and their hash codes.

[0109] 5) Calculate the similarity based on the obtained NUM hash codes, sort them, and select the top n candidate set instances as the final query results.

[0110] The above-described embodiments are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, any changes made based on the shape and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A dynamic cross-modal hash retrieval method for concept drift, characterized by: The following steps are involved: S1: Acquire multimodal data, including image data and text data, perform feature extraction on the multimodal data, extract corresponding image features and text features, and perform centralization operations on the image features and text features; S2: Image features and text features after the centering operation use the radial basis kernel function to capture the potential nonlinear features in their feature space, and obtain image features and text features with richer semantic information. The image features and text features are divided into Batch, data appears in batches, thus simulating a dynamic data environment. As image features and text features appear in batches, the distribution of data changes and concept drift occurs; S3: For each batch of image and text features, label regression learning, label drag learning, and semantic embedding learning are performed to reduce the information imbalance between image and text data, strengthen multimodal information interaction, and map image and text features to a hash space to train hash functions and hash codes. S4: Based on the obtained hash function, the weight of the hash function is calculated by the image-text similarity relationship weight, the hash code variance weight, and the hash code pairing weight; The image-text similarity relationship weight is calculated by the similarity matrix, and the Hamming distance is used as an indicator to make the Hamming distance between similar image features and text features as small as possible, and the Hamming distance between dissimilar image features and text features as large as possible; the hash code variance weight is calculated by respectively calculating the variance of the image data hash code and the variance of the text data hash code, so that the hash code variance is maximized, the entropy is highest, and the most information is contained, thereby obtaining the optimal hash function; the hash code pairing weight is calculated to calculate the similarity between the image data hash code and the text data hash code. The higher the hash code similarity, the better the hash function effect; and then the saved The weight of the hash function, Among the hash functions, keep the top one with the highest weight. A hash function and its hash code; this A hash function and its hash code constitute a multi-hash table mechanism; S5: According to the obtained Calculate the similarity of the hash codes, sort them, and select the top candidate set instances as the final query results.

2. The dynamic cross-modal hash retrieval method for concept drift according to claim 1 is characterized in that: In step S1, the VGG19 neural network is used to extract image features for image data, and the BOW model is used to extract text features for text data.

3. The dynamic cross-modal hash retrieval method for concept drift according to claim 1 is characterized in that: The specific operation steps of step S3 are as follows: S31: Using label regression learning, hash codes can learn label information, so that image data and text data with more similar labels have more similar hash codes, thereby reducing the information imbalance between image data and text data, strengthening multimodal information interaction, and calculating the loss function of label regression learning : ； ； Where, represents a hash code, Indicates a label, represents a public real-valued space-hash space mapping function and is orthogonal, is the identity matrix, represents the F-norm; S32: Using label drag learning to increase the difference between data with different labels, thereby learning more discriminative hash codes and calculating the loss function of label drag learning : ； Where, Indicates the direction in which the label is dragged. Indicates the extent of label dragging; S33: Using semantic embedding learning, the image features processed by the radial basis kernel function are mapped to a common real-valued space. The image data distribution loss function in the common real-valued space is : ； ； Where, Indicates time , Represents image related parameters, is the data distribution in the public real-valued space, is the image feature-public real-valued space mapping function, is the radial basis kernel function, Indicates time the image features that appear; express orthogonal; S34: Image data distribution in the public real-valued space is achieved by Establish a connection with the label and convert the image data distribution loss function in the public real-valued space into the image data label loss function : ； because is an orthogonal matrix, Equivalent to the loss function : ； S35: Using semantic embedding learning, the text features processed by the radial basis kernel function are mapped to a common real-valued space. The text data distribution loss function in the common real-valued space is : ； ； Where, Indicates text related parameters, is the text feature-public real-valued space mapping function, Indicates time the textual features that appear; express orthogonal; S36: Text data distribution in public real-valued space is achieved through Establish a connection with the label and convert the text data distribution loss function in the public real-valued space into a text label loss function : ； because is an orthogonal matrix, Equivalent to the loss function : ； S37: Final loss function for: 。 4. The dynamic cross-modal hash retrieval method for concept drift according to claim 1 is characterized in that: The specific operation steps of step S4 are as follows: S41: Calculate the image-text similarity weight based on the hash function obtained from the loss function , so that the Hamming distance between similar image-text pairs is as close as possible, and the Hamming distance between dissimilar image-text pairs is as far as possible, and the image-text similarity relationship weight It is expressed as follows: ； Where, For the image features, For the text features, represents the Hamming distance between image features and text features in the hash space, Representing image features With text features With different label information, Representing image features With text features Have the same label information; S42: Calculate the hash code variance weight based on the hash function obtained from the loss function , which maximizes the variance and makes the hash code contain the most information. The hash code variance weight It is expressed as follows: ； ； Where, is the image variance, is the text variance, Indicates the number of hash code bits, For the Bit hash code variance; S43: Calculate the hash code pairing weight based on the hash function obtained by the loss function , so that the hash codes of image-text pairs with the same label are more similar, the hash code pairing weight It is expressed as follows: ； Where, Represents the hash code of the image data calculated by the hash function, Represents the hash code of text data calculated by the hash function. is the XOR operation, is the hash code size; S44: Finally, the weight formula for: ； S45: At the moment , through the loss function Training on newly arrived data and Get the hash function and hash code, and then perform the weight formula Calculate and get the weight , then Always saved A hash function, through the weight formula Calculate the weights separately and divide the moments All Compare the weights, eliminate the hash function with the smallest weight, and save the remaining A hash function and its hash code.

Images