A Dynamic Multimedia Data Hash Retrieval Method and System Based on Scalable Increments

Abstract

This invention discloses a dynamic multimedia data hash retrieval method and system based on scalable incremental hashing, relating to the field of multimedia data retrieval technology. The method includes the following steps: acquiring the dynamic multimedia data to be retrieved; constructing a pre-trained hash retrieval model, using a scalable hash center as global supervision information and label cosine similarity as local supervision information to train the hash retrieval model. Specifically, forward propagation is used to obtain a generalized feature representation of the new multimedia data that maintains the discriminative power between new and old classes; a linear mapping relationship is constructed between the generalized feature representation and the hash code; auxiliary variables are introduced to continuously update the hash function without replay; the trained hash retrieval model is used to generate a query hash code for the dynamic multimedia data to be retrieved, and the query hash code is used for retrieval. This invention achieves low memory consumption, high update efficiency, and forgetting-free retrieval of dynamic multimedia data streams in an open environment.

Description

Technical Field

[0001] This invention relates to the field of multimedia data retrieval technology, and in particular to a dynamic multimedia data hash retrieval method and system based on scalable increments. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of internet technology, the scale of multimedia data is growing exponentially. Hash learning technology, by mapping high-dimensional features into compact binary hash codes, has become a mainstream solution for large-scale retrieval due to the fast computation advantage of Hamming distance. However, existing hash methods have significant drawbacks in open and dynamic environments, specifically in the following aspects:

[0004] (1) Traditional methods generally use the classification features at the end of the deep network (such as the cls token of the last layer of ViT) as the sample feature representation. This makes it easy to ignore the transfer knowledge of shallow features, making deep features prone to overfitting to new categories, resulting in insufficient discrimination ability for old categories, thereby exacerbating catastrophic forgetting, and ultimately causing frequent retrieval mismatches and a significant decrease in accuracy.

[0005] (2) Since discrete hash codes differ from real-valued features, according to channel coding theory, the size of the Hamming space that a hash code of a specific bit length can cover is fixed. However, the fixed bit length used in traditional hash learning lacks flexibility in open and dynamic scenarios. If the initial bit length is too short, hash collisions will inevitably occur as the category space continues to expand, leading to a decrease in retrieval accuracy; if the initial bit length is set too long to avoid collisions, it will increase the complexity of Hamming distance calculation and significantly reduce retrieval efficiency.

[0006] (3) Existing methods require caching and replaying historical samples to alleviate catastrophic forgetting, which increases memory burden and violates privacy protection and streaming storage constraints that data is only seen once and cannot be reused.

[0007] (4) Existing methods generally adopt backpropagation for end-to-end training. Each batch of new data requires multiple forward-backward iterations. The training time increases exponentially with the amount of data and parameters, which is difficult to meet the needs of rapid training of large-scale data and deployment of models on terminal devices with limited computing resources. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the purpose of this invention is to provide a dynamic multimedia data hash retrieval method and system based on scalable incremental hashing, achieving low memory usage, high update efficiency, and forget-free retrieval of dynamic multimedia data streams in an open environment.

[0009] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0010] The first aspect of this invention provides a dynamic multimedia data hash retrieval method based on scalable increments, comprising the following steps:

[0011] Retrieve dynamic multimedia data to be searched, wherein the multimedia data conforms to the fuzzy class incremental setting;

[0012] A pre-trained hash retrieval model is constructed, using the bit-extendable hash center as global supervision information and the label cosine similarity as local supervision information to train the hash retrieval model. Specifically, a generalized feature representation of new multimedia data that maintains the discriminative power between new and old classes is obtained through forward propagation. A linear mapping relationship between the generalized feature representation and the hash code is constructed, and auxiliary variables are introduced to continuously update the hash function without replay.

[0013] The trained hash retrieval model is used to generate query hash codes for the dynamic multimedia data to be retrieved, and the query hash codes are used for retrieval.

[0014] A second aspect of the present invention provides a dynamic multimedia data hash retrieval system based on scalable increments, comprising:

[0015] The data acquisition module is configured to acquire dynamic multimedia data to be retrieved, wherein the multimedia data conforms to the fuzzy incremental setting;

[0016] The model pre-training module is configured to build a pre-trained hash retrieval model. The model is trained using bit-scalable hash centers as global supervision information and label cosine similarity as local supervision information. Specifically, a generalized feature representation of new multimedia data that maintains the discriminative power between new and old classes is obtained through forward propagation. A linear mapping relationship between the generalized feature representation and the hash code is constructed, and auxiliary variables are introduced to continuously update the hash function without replay.

[0017] The hash retrieval module is configured to generate query hash codes from the dynamic multimedia data to be retrieved using the trained hash retrieval model, and then use the query hash codes to perform the retrieval.

[0018] A third aspect of the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and to execute steps in the dynamic multimedia data hash retrieval method based on scalable increments as described in the first aspect of the present invention.

[0019] A fourth aspect of the present invention provides a computer device comprising:

[0020] A processor, adapted to execute computer programs;

[0021] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the dynamic multimedia data hash retrieval method based on scalable increments as described in the first aspect of the present invention.

[0022] The above one or more technical solutions have the following beneficial effects:

[0023] This invention discloses a dynamic multimedia data hash retrieval method and system based on scalable incremental hashing. Based on a generalized feature extraction unit, it integrates shallow transferable knowledge and deep specific knowledge processes from a pre-trained large model, avoiding the limitations of single-end features. It fully utilizes the value of features at each layer while enhancing the fine-grained distinction between new and old categories, effectively reducing the risk of overfitting to new categories, minimizing retrieval errors, and significantly improving retrieval accuracy. Based on a bit-scalable hash center generation unit, a self-replication strategy extends historical hash centers and historical hash codes during code length expansion, eliminating the need to recalculate historical sample hash codes. The hash code length adaptively adjusts with the number of categories in powers of 2, avoiding hash collisions caused by short code lengths and computational redundancy caused by long code lengths, achieving a dynamic balance between semantic expressiveness and retrieval efficiency. Based on an online incremental hash encoding unit and an online hash function update unit, by introducing auxiliary variables, it directly solves analytical solutions relying only on key information such as labels and hash centers, without caching or replaying historical samples. This satisfies the privacy protection and streaming storage constraints of data only seeing it once, significantly reduces memory burden, and ensures that the update effect is equivalent to joint learning. This invention abandons the traditional multi-round iterative mode of backpropagation, and directly solves the hash code and projection matrix through analytical solution. The model update can be completed in only one forward propagation, achieving second-level update, effectively reducing computational complexity, meeting the rapid training requirements of large-scale dynamic data streams, adapting to terminal device deployment scenarios with limited computing resources, and improving the accuracy and efficiency of continuous retrieval of dynamic multimedia data in open environments.

[0024] This invention designs a feature extractor with powerful feature extraction and knowledge transfer capabilities, which can extract generalized feature representations from dynamic multimedia data, both generalizing to new categories and avoiding the forgetting of old knowledge.

[0025] This invention uses an adaptive hash code mechanism to dynamically adjust the bit length based on the number of categories and the data scale, achieving a dynamic balance between retrieval accuracy and efficiency. The adaptive adjustment of the hash code length matches the incremental categories, avoiding the accuracy loss or inefficiency caused by a fixed length.

[0026] This invention achieves the same effect as joint learning through a no-replay update mechanism, enabling incremental hash model updates without replaying historical data, thus balancing privacy protection and update efficiency.

[0027] This invention achieves low computational complexity training through a fast optimization algorithm, balancing training efficiency and model accuracy without end-to-end backpropagation.

[0028] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of the dynamic multimedia data hash retrieval method based on scalable increments in Embodiment 1 of the present invention. Detailed Implementation

[0031] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0032] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0034] Example 1:

[0035] Embodiment 1 of the present invention provides a dynamic multimedia data hash retrieval method based on scalable increments, such as... Figure 1As shown, dynamic multimedia data conforming to the fuzzy incremental setting is obtained. Based on coding theory and the Hadamard matrix, a bit-expandable hash center is generated. The bit-expandable hash center is used as global supervision information, and label cosine similarity as local supervision information. An auxiliary variable is introduced to achieve continuous hash code updates without replay. Generalized features of the newly added multimedia data are extracted, and a linear mapping relationship between the generalized feature representation and the hash code is constructed. An auxiliary variable is introduced to achieve a continuous hash function updates without replay. Query hash codes are generated for the samples to be retrieved. The Hamming distance between the query hash code and the hash codes in the database is calculated, and the TOP-K retrieval results are output.

[0036] Specifically, the following steps are included:

[0037] S1: Obtain the dynamic multimedia data to be retrieved.

[0038] In one specific implementation, the multimedia data conforms to a fuzzy class increment setting, where the fuzzy class increment setting is... It can include both old and new category samples. In an open environment, dynamic multimedia data follows a fuzzy class increment setting, collecting data block sequences in the form of a data stream. The newly added data in round t is denoted as... , To add multimedia data, the multimedia data format can be images, text, video, audio, etc. Indicates the corresponding category label, This is the number of samples in the new data block. It is the total number of all categories that have appeared. It is the old category number. The number of new categories that new data blocks may introduce leads to complex situations where task boundaries become blurred. This represents the number of new categories introduced by the new data block in the j-th round.

[0039] S2: Construct a pre-trained hash retrieval model, using bit-extendable hash centers as global supervision information and label cosine similarity as local supervision information to train the hash retrieval model.

[0040] The hash retrieval model in this embodiment is an incremental dynamic hash retrieval architecture, including:

[0041] Basic encoder layer: adopts a frozen pre-trained large model (such as CLIP-ViT-L / 14, LLaMA), the core function of which is to generate the initial block features of multimedia data. All original parameters are frozen throughout the process and do not participate in the update.

[0042] Feature processing module: Contains three units: feature fusion, expansion, and smoothing. It is used to transform the initial block features into generalized features, updating only the weights and random projection matrices within each unit. and Sigmoid layer parameters;

[0043] The hash function learning module contains three units: supervised signal, projection matrix learning, and auxiliary variables. Its core function is to construct a semantic mapping between generalized features and hash codes, and only updates the hash center, projection matrix, and auxiliary variables.

[0044] In one specific implementation, historical multimedia data is used to train the hash retrieval model, enabling... Represents historical multimedia data. This represents the historical tag matrix. All accumulated multimedia data can then be represented as... The corresponding label matrix can be represented as Obviously, , , In round t, model training and hash code updates only apply the new data block. All historical data It is non-reusable, enabling continuous updates of hash codes and hash functions without replay, and its effect is equivalent to using all data. Joint training. Among them, This represents the number of new multimedia data samples added in the j-th round.

[0045] Specifically, the following steps are included:

[0046] S2.1: Obtain a generalized feature representation of new multimedia data that maintains the discriminative power between the old and new classes through forward propagation.

[0047] In one specific implementation, the hash retrieval large model consists of a frozen pre-trained large model encoder. Forward propagation is performed on the frozen pre-trained large model encoder to extract features from each block, followed by feature fusion, expansion, and smoothing. The output is a generalized feature representation of the new multimedia data that maintains the discriminative power between the old and new classes. .

[0048] In this embodiment, all original parameters of the pre-trained large model (such as LIP-ViT-L / 14, LLaMA, etc.) are frozen, including the weights and biases of each block. These parameters remain fixed throughout the training process and do not participate in training updates. The training parameters are the parameters corresponding to the newly added feature fusion, expansion, and smoothing processes in the hash retrieval large model. Specifically, these include: the weight parameters corresponding to the summation and averaging of features in each block during the feature fusion stage; the random projection matrix parameters during the feature expansion stage; and the Sigmoid layer parameters during the feature smoothing stage.

[0049] S2.1.1: Select a pre-trained large model based on the differences in data types and generate the initial feature representation of the corresponding samples.

[0050] Specifically, it needs to be based on Due to differences in data types, a suitable pre-trained large model is selected. In this embodiment, CLIP-ViT-L / 14 is used for image data, LLaMA for text data, VideoMAE for video data, and AudioMAE for audio data. A single forward propagation is performed on the encoder of all models, and the original multimedia data is input into the encoder to generate the initial feature representation of the corresponding sample, laying the foundation for the subsequent feature fusion stage.

[0051] S2.1.2: Perform feature fusion on the obtained initial feature representations to obtain fused features.

[0052] Specifically, for each block feature of the pre-trained large model The fused features are obtained by performing summation and averaging operations:

[0053] .

[0054] in, To fuse features, N represents the number of model blocks, and l represents the model block index. This operation, through the collaborative integration of the initial features of each block, fully preserves the transferable general knowledge of shallow features (such as cross-category common information like image edges and text grammar) and the category-specific discriminative knowledge of deep features (such as fine-grained category semantics and sample-specific features). This avoids the problems of overfitting to new categories and weakening the discriminative ability of old categories caused by relying solely on deep features at the end, and also eliminates the problem of wasting the value of shallow features. It effectively enhances the fine-grained discrimination ability of new categories in the incremental stage, laying a feature foundation for mitigating catastrophic forgetting in subsequent incremental learning.

[0055] S2.1.3: Extend the fusion features to obtain extended features.

[0056] Specifically, construct the random projection matrix. ,in, Represents the set of real numbers. For fusion features Dimensions For high-dimensional space, random projection matrix After being generated and fixed through a random normal distribution, this embodiment uses the fused feature dimension d and a preset high-dimensional dimension. Determine the random projection matrix Shape, according to independent standard normal distribution N Each element is sampled at (0,1), and the generated data is reused in a fixed manner. No multimedia sample data (features, labels, historical data) is involved throughout the process, and no updates are required with each round. The fused features will be used. The extended feature matrix is ​​obtained by projecting it onto a higher-dimensional space through a linear mapping. :

[0057] .

[0058] In this embodiment, the projection matrix P is constructed as a linear transformation carrier to map the low-dimensional fusion features of multimedia data to a high-dimensional space. The core is to expand the feature expression dimension through the high-dimensional space to solve the problem of insufficient discrimination of low-dimensional features. Subsequently, the analytical solution is solved based on the expanded features. Essentially, this allows the model to learn the semantic mapping rules from fusion features to hash codes (matching the category semantics of hash codes), thereby obtaining the hash code association knowledge corresponding to the features required for hash retrieval.

[0059] S2.1.4: Perform feature smoothing operation on the extended features.

[0060] Specifically, the Sigmoid function is applied to the extended features to compress the feature values ​​to the (0,1) interval. This suppresses the interference of extreme feature values ​​such as abnormal noise and obscure semantics in the new category samples on the subsequent learning process of hash codes and hash functions, ensuring the stability of the feature distribution. The final output is a generalized feature representation of the new multimedia data.

[0061] .

[0062] in, Features after smoothing operations refer to the generalized features of newly arrived multimedia data after processing. The Sigmoid function enables the fusion of shallow transferability and deep specific features, mitigating fine-grained knowledge forgetting in incremental learning.

[0063] S2.2: Construct a linear mapping relationship between the generalized feature representation and the hash code.

[0064] In one specific implementation, this embodiment constructs generalized features for multimedia data. With hash code The linear mapping relationship is explained by introducing auxiliary variables. and Using only new data Solving the projection matrix Generate a hash function. The sum of generalization features, including new generalization features. It also includes historical generalization characteristics.

[0065] Specifically, the following steps are included:

[0066] S2.2.1: Define the objective function for joint learning.

[0067] The objective function is:

[0068] .

[0069] in, This is the generalized feature matrix of multimedia data. The corresponding hash code is the hash code of all accumulated data, including newly arrived hash codes. And historical hash codes, This is the regularization coefficient, used to balance fitting accuracy and model complexity. Here, Frobenius norm is used to measure the overall difference between matrices. The projection matrix to be learned in turn By minimizing this function, the generalized feature projection result of multimedia data is made as close as possible to the hash code to ensure the accuracy of semantic mapping. For example, the same type of image or text features can correspond to the same hash code after projection. At the same time, regularization avoids the model only adapting to the current round of multimedia data (for example, only remembering the local features of the current image), and ensures that when new multimedia data is input in the future, it can also be accurately mapped to the corresponding hash code.

[0070] S2.2.2: Introduce auxiliary variables to solve the projection matrix without playback of the objective function. .

[0071] Specifically, introduce auxiliary variables and :

[0072] ,

[0073] .

[0074] In this embodiment Both represent the hash adaptive code length in the t-th round, and correspondingly, This represents the adaptive hash code length for the (t-1)th round.

[0075] The analytical solution of the projection matrix can be obtained directly through matrix operations:

[0076] .

[0077] in, It is the cumulative cross statistic of features, used to record the autocorrelation of each round of feature expansion, and its role is to replace the feature distribution information of historical samples in no-replay updates; This is the cumulative correlation statistic between features and hash codes, recording the mutual correlation between features and hash codes in each round of expansion. Its function is to replace the feature-to-hash-code mapping information of historical samples in no-replay updates. The auxiliary variable is designed to achieve the goal of solving the analytical solution of the projection matrix solely through the cumulative statistics without caching historical samples, thereby meeting the needs of efficient and privacy-preserving updates in fuzzy incremental scenarios. for The solution for the identity matrix of order 1 / 2 can be obtained without relying on historical sample features, using only new data to achieve a solution equivalent to joint learning, depending only on the current round's data. , Compared to the previous round , This approach avoids the risk of historical data privacy leaks and eliminates the need for multiple forward-backward iterations in traditional reverse propagation, perfectly adapting to the high-efficiency update requirements of dynamic data streams in open environments.

[0078] S2.2.3: Generate the corresponding hash function based on the projection matrix obtained from the solution.

[0079] Specifically, the first The hash function for the round is:

[0080] .

[0081] in, This represents the generalization feature of the sample to be retrieved in round t. For the sign function, The real-valued projection result is converted into a binary hash code encoded in {-1,1}.

[0082] S2.3: Introduce auxiliary variables to continuously update the hash function without replay.

[0083] S2.3.1: Based on encoding theory and the Hadamard matrix, generate a bit-scalable hash center that fits the number of categories.

[0084] In one specific implementation, based on coding theory and the row orthogonality of the Hadamard matrix, a method is generated to adapt to the current cumulative number of categories. The bit-scalable hash center, the hash center follows Growth Adaptive Adjustment of Hash Code Length And stretch.

[0085] Specifically, the following steps are included:

[0086] S2.3.1.1: Determine the adaptive code length based on the cumulative number of categories.

[0087] Specifically, calculate the adaptive hash code length for the t-th round. Ensure that the code length is not less than the cumulative number of categories, and is a power of 2. This avoids the problems of storage redundancy due to too few categories and too long code, and insufficient semantic expression due to too many categories and too short code. Specifically, This represents the equality comparison operator, used to determine whether the expressions on both sides are equal. Here, it means checking whether the remainder is equal to 0.

[0088] S2.3.1.2: Construct the adaptive Hadamard matrix.

[0089] Specifically, define the adaptive Hadamard matrix for the t-th round. It satisfies the condition that any two rows are orthogonal (maximum Hamming distance), ensures the discriminative power of subsequent class centers, serves as the template for generating all higher-order Hadamard matrices, and is a higher-order matrix generated by recursively stretching the fundamental matrix. This construction method ensures that the stretched matrix still maintains orthogonality, without the need to retrain the historical hash center.

[0090] S2.3.1.3: Select the hash center of the new category based on the adaptive Hadamard matrix.

[0091] Specifically, from the adaptive Hadamard matrix Selected from Arriving at the The row serves as the hash center for the newly added category, i.e. .

[0092] S2.3.1.4: Extend the historical hash center according to the code length change.

[0093] Specifically, for historical hash centers, if If the code length remains unchanged, then the historical hash center remains unchanged, that is... If the code length in round t is... , then the first Wheel's Historical Hash Center Extend the historical hash center through a self-replication strategy, i.e. .

[0094] This embodiment relies on the row orthogonality of the Hadamard matrix. When the code length remains unchanged, the historical hash center remains unchanged (which already satisfies the requirement for inter-class differentiation). When the code length doubles, the historical hash center is self-replicated (utilizing the property that the orthogonal matrix remains orthogonal after self-replication). This eliminates the need to recalculate the hash code of historical samples, reduces update costs, and completely preserves the semantic features of the old category hash center. At the same time, it adapts to the dynamic growth of categories in the fuzzy class incremental scenario, solves the redundancy or collision problems of traditional fixed code length, and improves the adaptability and effectiveness of hash retrieval.

[0095] S2.3.1.5: Construct a global hash center by merging historical hash centers with newly added category hash centers.

[0096] Specifically, the merged and extended historical hash center and the newly added category hash center form the first... Round complete hash center ,in, It is a newly added category hash center. It is the center of historical hashes. It is a global hash center; by merging historical hash centers with newly added category hash centers, a global hash center is constructed to ensure that both the old and new category hash centers satisfy orthogonality and maximize the Hamming distance between classes.

[0097] S2.3.2: Using the bit-extendable hash center as global supervision information and the label cosine similarity as local supervision information, we learn the hash code of semantic embedding and introduce auxiliary variables to achieve continuous update of hash code without replay.

[0098] In one specific implementation, using the bit-extendable hash center as global supervision information and the label cosine similarity as local supervision information, auxiliary variables are introduced to construct the objective function, utilizing only... Generate hash code for new data , and according to Stretch strategy update history hash code The cumulative hash code matrix is ​​obtained. .

[0099] Specifically, the following steps are included:

[0100] S2.3.2.1: Reconstruct the objective function of joint learning based on global and local supervision information.

[0101] Specifically, a scalable hash center Label cosine similarity serves as global supervisory information. As local supervisory information, learn the hash code of semantic embedding.

[0102] in, Represents the cumulative label matrix. This is the label normalization function, which transforms the category label matrix (taking values ​​of 0 / 1) into a real-valued semantic vector matrix with a 2-norm of 1. This is used in the formula for calculating label cosine similarity and hash code solving, providing semantic supervision for hash code updates and supporting semantic alignment between old and new hash codes. The objective function for joint learning is shown below:

[0103] .

[0104] in, It is the hash code of all accumulated data. It is the hash code of historical data. It is the hash code of the new data. This is a balancing coefficient with positive real values, used to adjust the importance ratio between global and local monitoring items. is the Frobenius norm, used to measure the overall difference between matrices.

[0105] S2.3.2.2: Objective function that is continuously updated using new data labels.

[0106] Specifically, keep the historical hash codes unchanged and only use the new data tags. Learning the hash code of new data transforms the joint objective function into:

[0107] .

[0108] S2.3.2.3: Solve the hash code of new data without replay based on the continuously updated objective function.

[0109] Specifically, introduce auxiliary variables To find the solution to the continuously updated objective function is Using only new data, an equivalent solution to joint learning can be achieved. For new data labels, For the first Round complete hash center, For balance coefficient, This is the hash code length for the current round.

[0110] S2.3.2.4: Merge all hash codes to obtain the overall hash code.

[0111] Specifically, first, the historical data hash code is stretched. If the code length in round t is... ,but If the code length in round t is... Then the hash code of historical data Perform column self-copy stretch operation Then, the stretched historical data hash code is merged with the new hash code to obtain the first hash code. Round database hash code .

[0112] In this embodiment, step S2.1, generating generalized feature representations, relies on the model's basic encoder layer and feature processing module. Initial features are generated through a pre-trained encoder and transformed into generalized features via fusion, expansion, and smoothing units. Step S2.2, constructing the linear mapping and hash function, relies on the hash function learning module. The projection matrix is ​​optimized through the objective function, and the hash function is generated by solving for auxiliary variables. Step S2.3, continuously updating the hash function without replay, relies on the supervisory signal unit and auxiliary variable unit of the hash function learning module to generate a bit-scalable hash center, update the hash code library, and achieve incremental updates of the model without replay.

[0113] The three-layer architecture of the hash retrieval model provides a clear execution vehicle for the processing method in this step. The S2 processing method executes the functions of each module of the model to achieve the complete transformation of multimedia data extraction of generalized features and hash code, thus achieving the goal of dynamic retrieval with no playback, low memory usage, and high accuracy.

[0114] S3: Use the trained hash retrieval model to generate query hash codes for the dynamic multimedia data to be retrieved, and use the query hash codes to perform retrieval.

[0115] In one specific implementation, the sample to be retrieved Generate query hash code Calculate the query hash code Hash codes in the database Calculate the Hamming distance and output the Top-K search results in ascending order of Hamming distance. calculate.

[0116] Specifically, the following steps are included:

[0117] S3.1: Encoded query sample.

[0118] Specifically, a dedicated pre-trained large model is selected based on the query sample type (completely consistent with the model selection rules in S2.1.1). A single forward propagation is performed on the model encoder to extract the initial features of each block, preserving shallow transfer knowledge and deep category-specific knowledge. Generalized features of the query samples are generated through "sum-average" fusion, random projection expansion, and sigmoid smoothing. ;Will Input the hash function of S2.2.3 The binary hash code of the query sample is calculated. .

[0119] S3.2: Calculate the Hamming distance.

[0120] Specifically, calculate the hash code of the query sample. With the complete database hash code up to round t Hamming distance :

[0121] .

[0122] S3.3: Output the search results.

[0123] Specifically, for global hash codes The corresponding database samples are sorted in ascending order of Hamming distance. The smaller the distance, the higher the semantic matching degree between the sample and the query requirement. The top-K samples (K is the preset number of search results, such as K=10, 20, etc.) are selected as the final search results output.

[0124] Example 2:

[0125] Embodiment 2 of the present invention provides a dynamic multimedia data hash retrieval system based on scalable increments, comprising:

[0126] The data acquisition module is configured to acquire dynamic multimedia data to be retrieved, wherein the multimedia data conforms to the fuzzy incremental setting;

[0127] The model pre-training module is configured to build a pre-trained hash retrieval model. The model is trained using bit-scalable hash centers as global supervision information and label cosine similarity as local supervision information. Specifically, a generalized feature representation of new multimedia data that maintains the discriminative power between new and old classes is obtained through forward propagation. A linear mapping relationship between the generalized feature representation and the hash code is constructed, and auxiliary variables are introduced to continuously update the hash function without replay.

[0128] The hash retrieval module is configured to generate query hash codes from the dynamic multimedia data to be retrieved using the trained hash retrieval model, and then use the query hash codes to perform the retrieval.

[0129] Example 3:

[0130] Embodiment 3 of the present invention provides a computer-readable storage medium storing a computer program adapted for loading by a processor and executing the steps of the dynamic multimedia data hash retrieval method based on scalable increments as described in Embodiment 1 of the present invention.

[0131] Example 4:

[0132] Embodiment 4 of the present invention provides a computer device, the device comprising:

[0133] A processor, adapted to execute computer programs;

[0134] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps of the dynamic multimedia data hash retrieval method based on scalable increments as described in Embodiment 1 of the present invention.

[0135] The steps and methods involved in Examples 2, 3 and 4 above correspond to those in Example 1. For specific implementation details, please refer to the relevant description section of Example 1.

[0136] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0137] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc.

[0138] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A dynamic multimedia data hash retrieval method based on scalable incremental hashing, characterized in that, Includes the following steps: Retrieve dynamic multimedia data to be searched, wherein the multimedia data conforms to the fuzzy class incremental setting; A pre-trained hash retrieval model is constructed, using the bit-extendable hash center as global supervision information and the label cosine similarity as local supervision information to train the hash retrieval model. Specifically, a generalized feature representation of new multimedia data that maintains the discriminative power between new and old classes is obtained through forward propagation. A linear mapping relationship between the generalized feature representation and the hash code is constructed, and auxiliary variables are introduced to continuously update the hash function without replay. The specific steps for obtaining generalized feature representations of new multimedia data that maintain the discriminative power between old and new classes through forward propagation are as follows: First, select a pre-trained large model based on the differences in data types and generate initial feature representations for the corresponding samples. Specifically, CLIP-ViT-L / 14 is used for image data, LLaMA for text data, VideoMAE for video data, and AudioMAE for audio data. Perform a single forward propagation on the encoders of all models, inputting the original multimedia data into the encoders to generate initial feature representations for the corresponding samples, laying the foundation for subsequent feature fusion. Then, perform feature fusion on the obtained initial feature representations to obtain fused features. Specifically, for each block feature of the pre-trained large model... The fused features are obtained by performing summation and averaging operations: The fused features are expanded to obtain expanded features; feature smoothing is then performed on the expanded features. in, For feature fusion, N is the number of model blocks, and l is the index of the model block; The specific steps for introducing auxiliary variables to continuously update the hash function without replay are as follows: Based on encoding theory and the Hadamard matrix, a bit-expandable hash center with an appropriate number of categories is generated; using the bit-expandable hash center as global supervision information and the label cosine similarity as local supervision information, the hash code of semantic embedding is learned; auxiliary variables are introduced to achieve continuous updating of the hash code without replay, specifically: reconstructing the joint learning objective function based on global and local supervision information; continuously updating the objective function using new data labels; solving the hash code of new data without replay based on the continuously updated objective function; merging all hash codes to obtain the overall hash code; The objective function for reconstructing the joint learning based on global and local supervision information specifically involves using a bit-scalable hash center. Label cosine similarity serves as global supervisory information. As local supervisory information, the hash code of the semantic embedding is learned; where, Represents the cumulative label matrix. The label normalization function transforms the category label matrix (with values ​​of 0 / 1) into a real-valued semantic vector matrix with a 2-norm of 1. This matrix is ​​used in the formula for calculating label cosine similarity and hash code solving, providing semantic supervision for hash code updates and supporting semantic alignment between old and new hash codes. The objective function for joint learning is shown below: in, It is the hash code of all accumulated data. It is the hash code of historical data. It is the hash code of the new data. This is a balancing coefficient with positive real values, used to adjust the importance ratio between global and local monitoring items. The Frobenius norm is used to measure the overall difference between matrices. Specifically, to solve for the hash code of new data without replay based on the continuously updated objective function, auxiliary variables are introduced. To find the solution to the continuously updated objective function is Using only new data, a solution equivalent to joint learning can be achieved; where For new data labels, For the first Round complete hash center, For balance coefficient, The current round's hash code length is given; all hash codes are merged to obtain the overall hash code. The trained hash retrieval model is used to generate query hash codes for the dynamic multimedia data to be retrieved, and the query hash codes are used for retrieval.

2. The dynamic multimedia data hash retrieval method based on scalable incremental hashing as described in claim 1, characterized in that, The specific steps for constructing a linear mapping relationship between generalized feature representations and hash codes are as follows: Define the objective function for joint learning; Auxiliary variables are introduced to solve the projection matrix without playback based on the objective function; Generate the corresponding hash function based on the obtained projection matrix.

3. The dynamic multimedia data hash retrieval method based on scalable incremental hashing as described in claim 1, characterized in that, Based on coding theory and the Hadamard matrix, the specific steps for generating a bit-scalable hash center that fits the number of categories are as follows: The adaptive code length is determined based on the cumulative number of categories; Construct an adaptive Hadamard matrix; Select the hash center for the new category based on the adaptive Hadamard matrix; The historical hash center is extended according to the code length change; A global hash center is constructed by merging historical hash centers with newly added category hash centers.

4. The dynamic multimedia data hash retrieval method based on scalable incremental hashing as described in claim 1, characterized in that, The specific steps for retrieving data using query hash codes are as follows: Calculate the Hamming distance between the query hash code and the hash codes in the database, and output the Top-K search results in ascending order of Hamming distance.

5. A dynamic multimedia data hash retrieval system based on scalable incremental hashing, characterized in that, include: The data acquisition module is configured to acquire dynamic multimedia data to be retrieved, wherein the multimedia data conforms to the fuzzy incremental setting; The model pre-training module is configured to build a pre-trained hash retrieval model. The model is trained using bit-scalable hash centers as global supervision information and label cosine similarity as local supervision information. Specifically, a generalized feature representation of new multimedia data that maintains the discriminative power between new and old classes is obtained through forward propagation. A linear mapping relationship between the generalized feature representation and the hash code is constructed, and auxiliary variables are introduced to continuously update the hash function without replay. The specific steps for obtaining generalized feature representations of new multimedia data that maintain the discriminative power between old and new classes through forward propagation are as follows: First, select a pre-trained large model based on the differences in data types and generate initial feature representations for the corresponding samples. Specifically, CLIP-ViT-L / 14 is used for image data, LLaMA for text data, VideoMAE for video data, and AudioMAE for audio data. Perform a single forward propagation on the encoders of all models, inputting the original multimedia data into the encoders to generate initial feature representations for the corresponding samples, laying the foundation for subsequent feature fusion. Then, perform feature fusion on the obtained initial feature representations to obtain fused features. Specifically, for each block feature of the pre-trained large model... The fused features are obtained by performing summation and averaging operations: The fused features are expanded to obtain expanded features; feature smoothing is then performed on the expanded features. in, For feature fusion, N is the number of model blocks, and l is the index of the model block; The specific steps for introducing auxiliary variables to continuously update the hash function without replay are as follows: Based on encoding theory and the Hadamard matrix, a bit-expandable hash center with an appropriate number of categories is generated; using the bit-expandable hash center as global supervision information and the label cosine similarity as local supervision information, the hash code of semantic embedding is learned; auxiliary variables are introduced to achieve continuous updating of the hash code without replay, specifically: reconstructing the joint learning objective function based on global and local supervision information; continuously updating the objective function using new data labels; solving the hash code of new data without replay based on the continuously updated objective function; merging all hash codes to obtain the overall hash code; The objective function for reconstructing the joint learning based on global and local supervision information specifically involves using a bit-scalable hash center. Label cosine similarity serves as global supervisory information. As local supervisory information, the hash code of the semantic embedding is learned; where, Represents the cumulative label matrix. The label normalization function transforms the category label matrix (with values ​​of 0 / 1) into a real-valued semantic vector matrix with a 2-norm of 1. This matrix is ​​used in the formula for calculating label cosine similarity and hash code solving, providing semantic supervision for hash code updates and supporting semantic alignment between old and new hash codes. The objective function for joint learning is shown below: in, It is the hash code of all accumulated data. It is the hash code of historical data. It is the hash code of the new data. This is a balancing coefficient with positive real values, used to adjust the importance ratio between global and local monitoring items. The Frobenius norm is used to measure the overall difference between matrices. Specifically, to solve for the hash code of new data without replay based on the continuously updated objective function, auxiliary variables are introduced. To find the solution to the continuously updated objective function is Using only new data, a solution equivalent to joint learning can be achieved; where For new data labels, For the first Round complete hash center, For balance coefficient, The current round's hash code length is given; all hash codes are merged to obtain the overall hash code. The hash retrieval module is configured to generate query hash codes from the dynamic multimedia data to be retrieved using the trained hash retrieval model, and then use the query hash codes to perform the retrieval.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-4, the dynamic multimedia data hash retrieval method based on scalable increments.

7. A computer device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, implements the dynamic multimedia data hash retrieval method based on scalable increments as described in any one of claims 1-4.

Images