Streaming modal data-oriented cross-modal hash retrieval method

Abstract

The invention discloses a streaming modal data-oriented cross-modal hash retrieval method. The method specifically comprises the following steps of: obtaining a feature data set and a tag data set of a modal; a knowledge base is constructed for each mode and used for storing semantic information of each mode; projecting modal features to a public Hamming space by using the sub-network of each modal, performing knowledge base mining, hash code alignment and knowledge base migration, and optimizing the sub-networks and the knowledge base; and repeatedly optimizing the sub-network and the knowledge base until convergence. The streaming media Hash learning normal form is utilized, the Hash code of each mode can be learned in parallel to process the streaming media data, all historical modes do not need to be trained again, and therefore the training complexity is reduced; the Hash technology is adopted to map high-dimensional features into compact binary Hash codes, the storage requirement is greatly reduced, the similarity of the Hash codes is measured through simple bitwise XOR operation, and the calculation speed is greatly increased.

Description

Technical Field

[0001] The present application belongs to the field of multimodal learning technology, and in particular relates to a cross-modal hash retrieval method for streaming modal data. Background Art

[0002] Cross-modal hash retrieval is an efficient data indexing and query method designed to address retrieval across different modalities, such as images, text, audio, and video. By mapping high-dimensional features to compact binary hash codes, cross-modal hash retrieval enables efficient computation of similarity metrics based on the Hamming distance, significantly reducing computational and storage overhead. Furthermore, hashing methods enable rapid indexing, enabling near-real-time matching and retrieval of cross-modal data in large-scale databases. This has important practical implications for building efficient, low-storage multimodal retrieval systems.

[0003] Traditional cross-modal retrieval methods use a joint learning approach to learn a common representation of multimodal data. Although these methods have demonstrated good performance, they rely heavily on the implicit assumption that data is fully available. In practical application scenarios, it is difficult to collect data from all modalities simultaneously, such as emergency medical assistance. Due to the asynchronous nature of multimodal data acquisition, these data from different modalities are often collected and processed continuously at different time points, forming streaming data. When processing streaming data, the above methods must frequently retrain all sub-networks of all modalities, which undoubtedly increases the learning cost and limits the scalability and flexibility of cross-modal retrieval. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a cross-modal hash retrieval method for streaming modal data, which uses the streaming hash learning paradigm to learn the hash code of each modality in parallel to process streaming data without retraining all historical modalities, thereby reducing training complexity.

[0005] In order to solve the above technical problems, the present invention is implemented in the following ways:

[0006] A cross-modal hash retrieval method for streaming modal data specifically includes the following steps:

[0007] S1. Obtain the feature dataset and label dataset of the modality;

[0008] S2. Build a knowledge base for each modality to store the semantic information of each modality;

[0009] S3. Use the sub-network of each modality to project the modal features into a common Hamming space, and perform knowledge base mining, hash code alignment, and knowledge base migration to optimize the sub-network and knowledge base.

[0010] S4. Repeat step S3 to optimize the sub-network and knowledge base until convergence.

[0011] Furthermore, the specific method of step S1 is as follows:

[0012] Collect the data set and assume that the data set has M modes, and its expression is as follows:

[0013]

[0014] in, represents the feature dataset on the mth modality, Represents the label dataset on the mth modality;

[0015] The feature dataset expression on the mth mode is as follows:

[0016]

[0017] Among them, n m Indicates the number of samples in the feature dataset of the mth modality;

[0018] The label dataset expression on the mth modality is as follows:

[0019]

[0020] Among them, n m Indicates the number of labeled dataset samples for the mth modality;

[0021] If the i-th sample on the m-th mode Belongs to the Cth category, then the Cth element of its semantic label ,otherwise .

[0022] Furthermore, the specific method of step S2 is as follows:

[0023] The class prototype is used as the abstract knowledge base to retain the semantic information of the stream data. The knowledge base expression of the mth mode is as follows:

[0024]

[0025] Where C represents the number of categories, Represents the class prototype set of category C;

[0026] Then for category c, the K class prototype expressions are as follows:

[0027]

[0028] Where K represents the number of class prototypes and L represents the bit length.

[0029] Furthermore, the specific method of step S3 is as follows:

[0030] First, use the sub-network of each modality to project the modal features into a common Hamming space, and obtain the hash code of each sample, setting the feature dataset sample The hash code is , in order to ensure that the hash code and its closest prototype of the same category are close to each other, the knowledge base loss is mined:

[0031] First, the sample The similarity expression with the k-th corresponding category prototype is as follows:

[0032]

[0033] in, Represents the cosine similarity calculation function, which is used to measure the similarity between two vectors;

[0034] Then, the loss expression of mining knowledge base is as follows:

[0035]

[0036] To ensure semantic consistency between prototypes within a class, this paper develops a knowledge base regularization mechanism to enhance the distinction between prototypes between classes. The knowledge base regularization loss expression is as follows:

[0037]

[0038] To ensure semantic consistency between streaming modal data, a knowledge base migration module is designed to extract knowledge from new media data and migrate its semantic information to the baseline knowledge base. This allows the proposed streaming hashing method to process new media without retraining the entire historical media data. For the mth (m ≥ 2) modality, the knowledge base migration loss is expressed as follows:

[0039]

[0040] Among them, σ represents the preset similarity boundary, which is used to control the degree of semantic similarity that the new and old class prototypes need to maintain;

[0041] In order to enhance the compactness of intra-class samples and the dispersion of inter-class samples, a discriminative hash learning module is proposed to constrain the learning of all sub-networks. The discriminative hash learning module strives to maximize the similarity of hash codes within the same class while minimizing the similarity of hash codes of different classes. For m (m ≥ 1) modes, the discriminative hash learning loss is expressed as follows:

[0042]

[0043] in, Indicates the similarity index of two samples. If the sample and samples For samples of the same type, , otherwise 0;

[0044] The total loss function expression is as follows:

[0045]

[0046] Among them, α and β represent the hyperparameters of the balance loss contribution, under the guidance of which the sub-network parameters and knowledge base of each modality are updated.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] Unlike existing static cross-modal retrieval methods that require all modalities to be prepared before processing, the present invention utilizes a streaming hash learning paradigm to learn the hash code of each modality in parallel to process streaming data without the need to retrain all historical modalities, thereby reducing training complexity. The hashing technology is used to map high-dimensional features into compact binary hash codes, significantly reducing storage requirements. The hash code also measures similarity through a simple bitwise XOR operation, significantly improving the calculation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0050] The specific implementation of the present invention will be further described in detail below with reference to the accompanying drawings and specific examples.

[0051] like Figure 1 As shown, a cross-modal hash retrieval method for streaming modal data specifically includes the following steps:

[0052] S1. Obtain the modality feature dataset and label dataset. The specific method is as follows:

[0053] Collect the data set and assume that the data set has M modes, and its expression is as follows:

[0054]

[0055] in, represents the feature dataset on the mth modality, Represents the label dataset on the mth modality;

[0056] The feature dataset expression on the mth mode is as follows:

[0057]

[0058] Among them, n m Indicates the number of samples in the feature dataset of the mth modality;

[0059] The label dataset expression on the mth modality is as follows:

[0060]

[0061] Among them, n m Indicates the number of labeled dataset samples for the mth modality;

[0062] If the i-th sample on the m-th mode Belongs to the Cth category, then the Cth element of its semantic label ,otherwise .

[0063] S2. Build a knowledge base for each modality to store the semantic information of each modality. The specific method is as follows:

[0064] Due to the lack of modal interaction, training a specific modality subnetwork for streaming media data alone is prone to semantic inconsistency. This paper uses class prototypes as an abstract knowledge base to preserve the semantic information of streaming data while reducing the computational burden of maintaining historical data. The knowledge base expression for the mth modality is as follows:

[0065]

[0066] Where C represents the number of categories, Represents the class prototype set of category C;

[0067] Then for category c, the K class prototype expressions are as follows:

[0068]

[0069] Among them, K represents the number of class prototypes, L represents the bit length, Represents the set of class prototypes of category c.

[0070] S3. Use the sub-network of each modality to project the modal features into a common Hamming space, and perform knowledge base mining, hash code alignment, and knowledge base migration to optimize the sub-network and knowledge base. The specific method is as follows:

[0071] First, use the sub-network of each modality to project the modal features into a common Hamming space, and obtain the hash code of each sample, setting the feature dataset sample The hash code is , in order to ensure that the hash code and its closest prototype of the same category are close to each other, the knowledge base loss is mined:

[0072] First, the sample The similarity expression with the k-th corresponding category prototype is as follows:

[0073]

[0074] in, Represents the cosine similarity calculation function, which is used to measure the similarity between two vectors;

[0075] Then, the loss expression of mining knowledge base is as follows:

[0076]

[0077] The above loss is designed to encourage each training sample to approach the nearest prototype with the same category. This loss may lead to a simple solution where all prototypes collapse to a single point, thus weakening the discriminative ability of the hash code. To this end, a knowledge base regularization mechanism is developed to enhance the distinguishability between inter-class prototypes while ensuring semantic consistency between intra-class prototypes. The knowledge base regularization loss expression is as follows:

[0078]

[0079] To ensure semantic consistency between streaming modal data, a knowledge base migration module is designed to extract knowledge from new media data and migrate its semantic information to the baseline knowledge base. This allows the proposed streaming hashing method to process new media without retraining the entire historical media data. For the mth (m ≥ 2) modality, the knowledge base migration loss is expressed as follows:

[0080]

[0081] Among them, σ represents the preset similarity boundary, which is used to control the degree of semantic similarity that the new and old class prototypes need to maintain;

[0082] In order to enhance the compactness of intra-class samples and the dispersion of inter-class samples, a discriminative hash learning module is proposed to constrain the learning of all sub-networks. The discriminative hash learning module strives to maximize the similarity of hash codes within the same class while minimizing the similarity of hash codes of different classes. For m (m ≥ 1) modes, the discriminative hash learning loss is expressed as follows:

[0083]

[0084] in, Indicates the similarity index of two samples. If the sample and samples For samples of the same type, , otherwise 0;

[0085] For the current data with mode m, the total loss function expression is as follows:

[0086]

[0087] Among them, α and β represent the hyperparameters of the balance loss contribution, under the guidance of which the sub-network parameters and knowledge base of each modality are updated.

[0088] S4. Repeat step S3 to optimize the sub-network and knowledge base until convergence.

[0089] The above description is merely an embodiment of the present invention. It is stated again that, for a person skilled in the art, several improvements can be made to the present invention without departing from the principles of the present invention, and these improvements are also included in the scope of protection of the claims of the present invention.

Claims

1. A cross-modal hash retrieval method for streaming modal data, characterized by: The specific steps include: S1. Obtain the feature dataset and label dataset of the modality; S2. Build a knowledge base for each modality to store the semantic information of each modality; S3. Use the sub-network of each modality to project the modal features into a common Hamming space, and perform knowledge base mining, hash code alignment, and knowledge base migration to optimize the sub-network and knowledge base. S4. Repeat step S3 to optimize the sub-network and knowledge base until convergence.

2. The cross-modal hash retrieval method for streaming modal data according to claim 1, characterized in that: The specific method of step S1 is as follows: Collect the data set and assume that the data set has M modes, and its expression is as follows: in, represents the feature dataset on the mth modality, Represents the label dataset on the mth modality; The feature dataset expression on the mth mode is as follows: Among them, n m Indicates the number of samples in the feature dataset of the mth modality; The label dataset expression on the mth modality is as follows: Among them, n m Indicates the number of labeled dataset samples for the mth modality; If the i-th sample on the m-th mode Belongs to the Cth category, then the Cth element of its semantic label ,otherwise .

3. The cross-modal hash retrieval method for streaming modal data according to claim 1, characterized in that: The specific method of step S2 is as follows: The class prototype is used as the abstract knowledge base to retain the semantic information of the stream data. The knowledge base expression of the mth mode is as follows: Where C represents the number of categories, Represents the class prototype set of category C; Then for category c, the K class prototype expressions are as follows: Where K represents the number of class prototypes and L represents the bit length.

4. The cross-modal hash retrieval method for streaming modal data according to claim 1, characterized in that: The specific method of step S3 is as follows: First, use the sub-network of each modality to project the modal features into a common Hamming space, and obtain the hash code of each sample, setting the feature dataset sample The hash code is , in order to ensure that the hash code and its closest prototype of the same category are close to each other, the knowledge base loss is mined: First, the sample The similarity expression with the k-th corresponding category prototype is as follows: in, Represents the cosine similarity calculation function, which is used to measure the similarity between two vectors; Then, the loss expression of mining knowledge base is as follows: A knowledge base regularization mechanism is developed to enhance the discrimination between inter-class prototypes. The knowledge base regularization loss expression is as follows: To ensure semantic consistency between streaming modal data, a knowledge base migration module is designed to extract knowledge from new media data and migrate its semantic information to the baseline knowledge base. This allows the proposed streaming hashing method to process new media without retraining the entire historical media data. For the mth (m ≥ 2) modality, the knowledge base migration loss is expressed as follows: Among them, σ represents the preset similarity boundary, which is used to control the degree of semantic similarity that the new and old class prototypes need to maintain; In order to enhance the compactness of intra-class samples and the dispersion of inter-class samples, a discriminative hash learning module is proposed to constrain the learning of all sub-networks. The discriminative hash learning module strives to maximize the similarity of hash codes within the same class while minimizing the similarity of hash codes of different classes. For m (m ≥ 1) modes, the discriminative hash learning loss is expressed as follows: in, Indicates the similarity index of two samples. If the sample and samples For samples of the same type, , otherwise 0; The total loss function expression is as follows: Among them, α and β represent the hyperparameters of the balance loss contribution, under the guidance of which the sub-network parameters and knowledge base of each modality are updated.

Images