A video hash large model, a video hash model construction method and device

Abstract

This invention relates to the field of video retrieval technology, specifically to a large video hash model, a method for constructing the video hash model, and an apparatus. In this invention, the base model utilizes Video-LlaMA to extract deep features from both visual and audio data, ensuring semantic quality. The cross-modal attention layer in the hash model innovatively achieves semantic-level interaction and fusion of audio and video features, making the generated fused features more discriminative. Finally, after further integration by the multimodal encoder and binarization processing by the hash layer, a compact hash code is output. Thus, the large video hash model, including the base model and the hash model, not only significantly improves the semantic representation accuracy of video content, thereby achieving higher accuracy in large-scale retrieval, but also greatly improves storage and retrieval efficiency through the generated binary hash code, providing a reliable technical foundation for accurate and rapid video retrieval and analysis in resource-constrained environments.

Description

Technical Field

[0001] This invention relates to the field of video retrieval technology, specifically to a large video hash model, a method for constructing the video hash model, and an apparatus. Background Technology

[0002] With the increasingly widespread application of video retrieval technology in various fields, higher demands are being placed on retrieval efficiency, storage costs, and computing resources. Video hashing technology, by mapping high-dimensional video features into compact binary hash codes, significantly reduces storage requirements and accelerates similarity searches, becoming a key technology for solving these challenges.

[0003] In recent years, machine learning methods based on pre-trained large models have demonstrated significant advantages in various tasks, but there has been no relevant research in the field of video hashing. Furthermore, the high storage and computational costs of large models limit their application in resource-constrained environments such as mobile devices and embedded systems. Therefore, how to reduce resource consumption while maintaining the excellent performance of large models has become a key problem that urgently needs to be solved in the field of video retrieval. Summary of the Invention

[0004] This invention provides a large video hash model, a method for constructing a video hash model, and an apparatus to address the problem in the prior art of lacking the application of pre-trained large models in the field of video hashing.

[0005] In a first aspect, the present invention provides a method for constructing a video hash model, the method comprising: a base model for acquiring video frame data and audio data of video data, and using a Video-LlaMA large model to extract features from the video frame data and audio data respectively to obtain video modal features and audio modal features; a hash model comprising a cross-modal attention layer, a multimodal encoder and a hash layer, wherein the cross-modal attention layer is used to capture the semantic interaction between the video modal features and audio modal features using cross-attention to obtain fused features after interaction, the multimodal encoder encodes the fused features to obtain an encoded output, and the hash layer is used to perform hash processing on the encoded output to obtain a hash code.

[0006] This invention constructs a large-scale video model comprising a base model and a hash model to achieve video hashing. The base model utilizes powerful Video-LlaMA to extract deep features from both visual and audio sources, ensuring richness of information sources and semantic quality. The cross-modal attention layer in the hash model innovatively achieves semantic-level interaction and fusion of audio and video features, making the generated fused features more discriminative. Finally, after further integration by the multimodal encoder and binarization processing by the hash layer, a compact hash code is output. This design not only significantly improves the semantic representation accuracy of video content, thus achieving higher accuracy in large-scale retrieval, but also greatly enhances storage and retrieval efficiency through the generated binary hash code, providing a reliable technical foundation for accurate and rapid video retrieval and analysis in resource-constrained environments.

[0007] In one optional implementation, the base model includes: an input layer for acquiring a video frame sequence containing preset frames and a preset number of audio segments corresponding to each frame; a modality-specific encoder for mapping each frame in the video frame sequence to an image embedding vector using an image encoder to obtain a video frame representation, adding positional encoding to the video frame representation, aggregating the positionally encoded frame representations to obtain a video representation of a preset length, and performing a linear transformation projection on the video representation to obtain video modal features; the modality-specific encoder is further used to convert audio segments into spectrograms, mapping each spectrogram to a dense vector using an audio encoder to obtain an audio representation, adding positional encoding to the audio representation, aggregating the positionally encoded audio representations to obtain an audio representation of a preset length, and performing a linear transformation projection on the audio representation to obtain audio modal features.

[0008] In this invention, modality-specific encoders are used to perform independent and structured deep processing on the visual and audio signals of a video. First, pre-trained dedicated encoders (image encoder and audio encoder) are used to extract high-quality frame-level visual features and audio spectral features, ensuring the richness and discriminative power of the original information for each modality. Then, by adding positional encoding to the serialized features, the model can effectively capture the dynamic changes of the video in the temporal dimension and the temporal structure of the audio signal. Finally, after feature aggregation and projection transformation, the model outputs fixed-length, semantically aligned video and audio modal features, providing high-quality, high-level input representations for subsequent cross-modal deep fusion, thus laying a solid foundation for improving the accuracy and robustness of video understanding and retrieval tasks.

[0009] In an optional implementation, the cross-modal attention layer is further used to generate cross-modal video features and audio features using a multi-head cross-attention mechanism with a linear fully connected layer, and to fuse the cross-modal video features and audio features to obtain fused features; The multimodal encoder is also used to encode the fusion features with added CLS tokens and location information using multiple Transformer layers, and to use the output vector corresponding to CLS in the encoded output sequence as the encoded output. The hash layer is also used to convert the encoded output into a binary real-valued vector using a fully connected network with a hyperbolic tangent function, and to process the binary real-valued vector using a sign function to obtain a binary hash code.

[0010] In this invention, a structured cross-modal information deep integration and compression mechanism significantly improves the semantic expressiveness and generation efficiency of video hash codes. Specifically, the cross-modal attention layer achieves deep semantic alignment and complementary fusion of video and audio features through multi-head cross-attention, enabling the fused features to more comprehensively characterize the video content. The multimodal encoder, leveraging the global modeling capabilities of the Transformer and specialized CLS markers, distills a highly condensed single vector representation containing complete video semantics from the fused features. Finally, the hash layer, by introducing a smooth binarization process using the hyperbolic tangent function, effectively reduces the information loss and gradient optimization difficulties caused by direct quantization of the sign function, thereby generating more discriminative and stable compact binary hash codes.

[0011] Secondly, the present invention provides a method for constructing a video hash model, the method comprising: constructing a large video hash model as described in the first aspect and any one of the claims of the first aspect of the present invention, and training the large video hash model to obtain a teacher model; quantizing the parameters of the base model using a first quantization method, quantizing the parameters of the hash model using a second quantization method, and introducing a Propulsion parameter into the hash model to obtain a student model, wherein the quantization error of the first quantization method is less than the quantization error of the second quantization method; and fine-tuning the student model based on the teacher model using a knowledge distillation method to obtain a video hash model.

[0012] This invention achieves significant lightweighting and acceleration while maintaining model accuracy through a strategy combining hierarchical hybrid quantization and knowledge distillation. The hierarchical hybrid quantization strategy differentiates the quantization precision of each layer, achieving an optimized balance between computational efficiency and retrieval performance. Simultaneously, by introducing learnable propulsion parameters and utilizing a pre-trained teacher model for knowledge distillation guidance, the student model can effectively inherit the powerful semantic representation of the teacher model and recover the accuracy lost due to quantization during fine-tuning with lower parameter update costs.

[0013] In one optional implementation, the first quantization method is the NF4 quantization method, and the second quantization method is the NF2 quantization method.

[0014] In this invention, low-bit quantization methods such as NF4 quantization and NF2 quantization are used, which not only reduces the computational burden but also achieves a higher compression ratio.

[0015] In one alternative implementation, training the teacher model includes: freezing the parameters of the base model and training and adjusting the parameters of the hash model based on the contrastive learning loss function to obtain the teacher model.

[0016] In one optional implementation, the student model is fine-tuned using a knowledge distillation method based on the teacher model to obtain a video hash model, including: constructing a distillation loss function based on the output of the base model in the teacher model and the output of the hash model; constructing a total loss function based on the distillation loss function and the contrastive learning loss function; and training and adjusting the parameters of the student model based on the total loss function to obtain the video hash model.

[0017] In this invention, the powerful feature representation capabilities of the full-precision teacher model are utilized, and a distillation loss mechanism enables the student model to mimic its deep semantic output, thereby quickly compensating for the information loss caused by quantization. Simultaneously, a contrastive learning loss function is used for further optimization, ensuring that the hash codes generated by the student model not only inherit the rich semantics of the teacher model but also possess excellent instance discrimination characteristics. This dual-supervision mechanism enables the final video hash model to significantly improve retrieval accuracy and feature quality while maintaining its lightweight advantage, achieving a highly efficient balance between model efficiency and performance.

[0018] Thirdly, the present invention provides a device for constructing a video hash model, the device comprising: a teacher model construction module, configured to construct a large video hash model as described in the first aspect and any one of the claims of the first aspect of the present invention, and train the large video hash model to obtain a teacher model; a student model construction module, configured to quantize the parameters of the base model using a first quantization method, quantize the parameters of the hash model using a second quantization method, and introduce a Propulsion parameter into the hash model to obtain a student model, wherein the quantization error of the first quantization method is less than the quantization error of the second quantization method; and a fine-tuning module, configured to fine-tune the student model based on the teacher model using a knowledge distillation method to obtain a video hash model.

[0019] Fourthly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the video hash model construction method of the second aspect above or any corresponding embodiment thereof.

[0020] Fifthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the method for constructing a video hash model according to the second aspect above or any corresponding embodiment thereof.

[0021] In a sixth aspect, the present invention provides a computer program product, including computer instructions for causing a computer to execute the method for constructing a video hash model according to the second aspect above or any corresponding embodiment thereof. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a structural block diagram of the video hashing big model according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a large video hash model according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating a method for constructing a video hash model according to an embodiment of the present invention; Figure 4 This is a schematic diagram comparing Propulsion with the hierarchical Propulsion of this invention embodiment; Figure 5 This is a structural block diagram of a device for constructing a video hash model according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0026] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0027] This embodiment provides a large video hash model, such as Figure 1 As shown, the video hashing big model includes: a base model for acquiring video frame data and audio data, and using the Video-LlaMA big model to extract features from the video frame data and audio data respectively to obtain video modal features and audio modal features; and a hashing model including a cross-modal attention layer, a multimodal encoder, and a hashing layer. The cross-modal attention layer is used to capture the semantic interaction between the video modal features and audio modal features using cross-attention to obtain the fused features after interaction. The multimodal encoder encodes the fused features to obtain the encoded output. The hashing layer is used to perform hashing processing on the encoded output to obtain a hash code.

[0028] Specifically, video hashing is used to generate semantic binary vectors for each video, thereby significantly reducing storage space and improving computational speed. Related technologies rely on pre-trained convolutional neural networks to extract features from keyframes of the video, average them to generate an overall representation, and then use an image hashing model to generate a binary vector as a hash code. Subsequently, end-to-end deep hashing methods specifically designed for video have been developed. For example, methods based on 3DCNN (3D-Convolutional Neural Networks) directly extract spatiotemporal features from the video. Methods based on MLP (Multilayer Perceptron) obtain the overall representation of the video through techniques such as fully connected networks. Methods based on LSTM (Long Short-Term Memory) model the sequence structure of the video to capture spatiotemporal content. Transformer-based methods utilize advanced attention mechanism architectures or contrastive learning for hash learning.

[0029] However, the research direction of combining pre-trained large models with video hashing tasks has not yet been explored. Based on this, this embodiment constructs a large video model comprising a base model and a hash model to achieve video hashing processing. The base model utilizes powerful Video-LlaMA to extract deep features from both visual and audio sources, ensuring the richness of information sources and semantic quality. The cross-modal attention layer in the hash model innovatively achieves semantic-level interaction and fusion of audio and video features, making the generated fused features more discriminative. Finally, after further integration by the multimodal encoder and binarization processing by the hash layer, a compact hash code is output. This design not only significantly improves the semantic representation accuracy of video content, thus achieving higher accuracy in large-scale retrieval, but also greatly improves storage and retrieval efficiency through the generated binary hash code, providing a reliable technical foundation for accurate and rapid video retrieval and analysis in resource-constrained environments.

[0030] The Video-LlaMA large-scale model is a pre-trained version. Specifically, the Video-LlaMA large-scale model is a multimodal large-scale model designed specifically for video understanding. It integrates pre-trained visual and audio encoders with a large-scale language model through efficient Q-Former (Querying Transformer) bridging technology, thereby endowing the model with the ability to deeply analyze visual and audio information in videos. The model adopts a two-branch architecture: the visual branch processes video frame sequences and extracts spatiotemporal features through Video Q-Former; the audio branch analyzes audio segments and captures sound semantics through Audio Q-Former. Finally, the multimodal features are projected onto the language model space for joint understanding and text generation. Therefore, this embodiment directly uses the pre-trained Video-LlaMA large-scale model as the base model to process video frame data and audio data in the video data to obtain features of both modalities.

[0031] In addition to the basic model, the video hashing model constructed in this embodiment further includes a hashing model. This hashing model comprises a cross-modal attention layer, a multimodal encoder, and a hashing layer. The cross-modal attention layer employs a cross-attention mechanism, allowing the features of the two modalities output by the basic model to interact deeply, capturing their semantic relationships and obtaining cross-modal fused features. The multimodal encoder further fuses the cross-modal features through a Transformer. The hashing layer compresses the encoded output of the multimodal encoder into a compact binary hash code.

[0032] In one optional implementation, the base model includes: an input layer for acquiring a video frame sequence containing preset frames and a preset number of audio segments corresponding to each frame; a modality-specific encoder for mapping each frame in the video frame sequence to an image embedding vector using an image encoder to obtain a video frame representation, adding positional encoding to the video frame representation, aggregating the positionally encoded frame representations to obtain a video representation of a preset length, and performing a linear transformation projection on the video representation to obtain video modal features; the modality-specific encoder is further used to convert audio segments into spectrograms, mapping each spectrogram to a dense vector using an audio encoder to obtain an audio representation, adding positional encoding to the audio representation, aggregating the positionally encoded audio representations to obtain an audio representation of a preset length, and performing a linear transformation projection on the audio representation to obtain audio modal features.

[0033] Specifically, in the input layer, this embodiment inputs a sequence of N video keyframes and N audio segments corresponding to each frame. Here, a keyframe refers to a representative set of frames used to characterize the main semantics of the entire video, rather than concepts such as I-frames in video coding standards. Keyframes can be selected from the original video at fixed intervals using a time-uniform sampling method; alternatively, existing scene segmentation or frame selection algorithms based on inter-frame differences can be used. This embodiment does not impose specific limitations on this approach.

[0034] like Figure 2 As shown, in the modality-specific encoder, the visual and audio branches of Video-LlaMA are used to extract features from audio and images, respectively. Specifically, in the visual branch, given a feature derived from... Video-LlaMA first uses a pre-trained image encoder (including Vision Transformer (ViT)) to map each frame into a sequence of keyframes. From the image embedding vectors, we obtain the video frame representation:

[0035] in Indicates the first Frames A set of 3D image embedding vectors, totaling Several vectors. To inject temporal information into the video frame representation, the model introduces a location embedding layer, adding location encoding to the representation of different frames:

[0036] in It is the first The frames are then positionally encoded. Next, the positionally encoded frame representations are fed into a Video Q-former (with the same structure as the Query Transformer in BLIP-2) for aggregation, generating fixed-length video features.

[0037] Thus obtain Each dimension is Video embedding vector As a representation of the video, since video features and LLM need to interact in the same embedding space, a linear transformation layer is required for projection:

[0038] in , For bias. Input features representing image modalities.

[0039] In the audio branch, the video is first evenly divided into... We first generate short audio clips and convert them into 128-dimensional Mel spectrograms. Then, we use a pre-trained audio encoder to map each spectrogram into a dense vector, obtaining the audio representation:

[0040] in, It is the first Features of each audio segment. Similarly, temporal information is injected into the audio segments through a location embedding layer:

[0041] Then, the audio Q-former is used to aggregate the segment features to obtain a fixed-length audio representation. ,in The number of audio embedding vectors. Dimensions for each vector:

[0042] Finally, a linear layer is used to represent the audio. Mapping to the same dimension as the text embedding in LLM yields the audio feature vector. :

[0043] in , For bias. The input features represent the audio modality.

[0044] Additionally, it should be noted that, in Figure 2 In this context, the superscript 't' indicates that 'v' and 'a' represent the video modality and audio modality, respectively. Video Queries and Audio Queries are sets of learnable query vectors designed for video and audio modalities. These vectors interact with the features output by the base model through attention, extracting high-level semantic representations suitable for hash encoding.

[0045] In an optional implementation, the cross-modal attention layer is further configured to generate cross-modal video and audio features using a multi-head cross-attention mechanism with a linear fully connected layer, and to fuse the cross-modal video and audio features to obtain fused features; the multimodal encoder is further configured to encode the fused features with added CLS tokens and location information using multiple Transformer layers, and to use the output vector corresponding to CLS in the encoded output sequence as the encoded output; the hash layer is further configured to convert the encoded output into a binary real-valued vector using a fully connected network with a hyperbolic tangent function, and to process the binary real-valued vector using a sign function to obtain a binary hash code.

[0046] Specifically, such as Figure 2 As shown, in the cross-modal attention layer, a multi-head cross-attention mechanism with a linear fully connected layer is used to process the two single-modal features. The specific processing procedure is expressed by the following formula:

[0047] in Indicates the target mode, The source mode is represented by the MCA calculation formula, which is given below:

[0048]

[0049] as well as

[0050]

[0051]

[0052] in , , ,and It is a learnable matrix. Equivalent to or Therefore, a representation rich in cross-modal information is obtained through the following formula. , :

[0053]

[0054] The data from the two modalities are concatenated to obtain the final fused feature:

[0055] To obtain the final video representation after fusing the features of the two modalities, this embodiment introduces a multi-modal encoder into the hash model. This encoder consists of... It consists of 10 Transformer layers, each of which has a multi-head self-attention (MSA) function (the number of multi-heads is 100). It consists of an MLP layer and an MLP layer. In the input... Previously, in A cls token, or CLS tag, is added to the beginning of the video, and its corresponding output serves as the final representation of the video. Specifically, this process is represented by the following formula:

[0056]

[0057]

[0058] in , This represents the learnable position embedding matrix corresponding to the audio / video multimodal token sequence. It is used to add positional information to the CLS tags and modal features before inputting them into the multimodal Transformer, enabling the model to distinguish tokens at different positions and time steps, thereby encoding temporal and sequence structure information. By following the above process, video... Potential representation It should be noted that the output of the last layer of the multimodal Transformer is a sequence of tokens of length 1+N. The first token is the global semantic representation corresponding to the predefined CLS marker, and the subsequent N tokens correspond to N multimodal feature vectors (e.g., representations of keyframes or time slices). Therefore, The output sequence contains (1+N) parts corresponding to the preset label CLS and N multimodal features. It consists of one CLS vector and N multimodal vectors, totaling 1+N parts. During hash encoding, the output vector corresponding to the CLS is... This serves as the global representation of the entire video, inputting it into subsequent hash layers.

[0059] In the hash layer, a fully connected network with a hyperbolic tangent function is used to convert the encoded output into a binary real-valued vector, i.e., for a given video... The output of its multimodal encoder Convert to 3D binary real-valued vector pass:

[0060] Then forward it to the final binary code. The Sign(·) function, i.e.

[0061] In related technologies, the research direction of combining pre-trained large models with video hashing tasks has not yet been explored. Furthermore, existing video hashing algorithms have not explored quantization, remaining focused on full-precision applications. Quantization methods can convert high-precision weights and activation values ​​into lower precision, significantly reducing computational resources and memory consumption, and improving inference efficiency and energy efficiency. Quantization methods can be divided into two categories: Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ).

[0062] Specifically, QAT maintains model performance through quantization-aware training. However, retraining large models is costly, and QAT methods require significant GPU time. PTQ is an efficient alternative. Furthermore, some research focuses on combining architecture compression and low-bit quantization to achieve greater compression ratios. However, existing PTQ methods typically use a fixed bit width (8-bit, 4-bit) for quantization of the entire model, failing to adequately consider the quantization sensitivity of different network layers, making it difficult to achieve an ideal balance between performance and efficiency. Experiments show that, at the same bit width, shallow networks often have larger quantization errors than deep networks, which may lead to more severe information loss in some critical layers.

[0063] Based on this, this embodiment provides a method for constructing a video hash model. By introducing quantization strategies with different quantization errors into the base model and hash model of the large video hash model, not only can storage and computational overhead be significantly reduced, but it may also achieve or even surpass the effect of a full-precision model in some cases.

[0064] According to an embodiment of the present invention, a method for constructing a video hash model is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0065] This embodiment provides a method for constructing a video hash model. Figure 3 This is a flowchart of a method for constructing a video hash model according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps: Step S101: Construct the large video hash model described in the above embodiments, and train the large video hash model to obtain a teacher model. Specifically, the constructed large video hash model includes a base model and a hash model, the specific structure of which is described in the above embodiments and will not be repeated here. In addition, this embodiment uses two video datasets, ActivityNet and FCVID, for training the large video hash model. The ActivityNet dataset covers a wide variety of human activities, and the videos are classified into 200 different categories. During model training, 10,023 videos were randomly selected as the training set and the video retrieval database, respectively. Furthermore, 2,512 videos were used for validation, and 2,413 videos were used for testing. The FCVID dataset contains 91,016 videos, labeled with 239 categories. This dataset is divided into three subsets: a training set containing 45,508 videos, a validation set, and a test set each containing 22,754 videos. The training set is also used as the video retrieval database.

[0066] Step S102 involves quantizing the parameters of the base model using a first quantization method and the parameters of the hash model using a second quantization method. A Propulsion parameter is introduced into the hash model to obtain the student model. The quantization error of the first quantization method is less than that of the second quantization method. Specifically, according to relevant technologies, using bit-level quantization can achieve a higher compression ratio. Therefore, both the first and second quantization methods used in this embodiment are low-bit quantization methods, but the quantization error of the first quantization method is less than that of the second quantization method.

[0067] Specifically, in the video hashing model, the base model is used to extract key features from the input data, where quantization errors can be amplified, leading to a significant performance degradation. The hashing model, however, focuses more on extracting and making decisions based on high-level semantic information, where quantization errors have a relatively smaller impact. Therefore, the first quantization method used in this embodiment has a smaller quantization error than the second quantization method; that is, the hashing model uses lower bit quantization than the base model to reduce computational burden while maintaining model expressive power.

[0068] In addition, the Propulsion method in related technologies adds a fine-tuning module to each Linear, specifically, such as... Figure 4 As shown, for the output of a module Introducing a trainable Propulsion matrix Propulsion adjusts its feature representation by element-wise scaling of the output of the linear layer. Specifically, let... The Each token is represented as:

[0069] Perform an element-wise transformation on it:

[0070] Right now:

[0071] in This represents element-wise multiplication over the entire linear layer's output. The propulsion transformation can be expressed as:

[0072] The representation of each token is adjusted by the Propulsion parameters, allowing the output of the linear layer to more flexibly adapt to task requirements. To further enhance flexibility, Propulsion introduces a multinomial scaling transformation, which involves adjusting the Propulsion parameters... Perform a power transformation:

[0073] hyperparameters Control the degree of nonlinearity of the Propulsion transform.

[0074] As can be seen, the Propulsion method in related technologies adds a fine-tuning module to each linear layer. While this method offers greater flexibility, it may introduce redundant parameters. This embodiment addresses this by using a multimodal encoder in a hash model... Adding a fine-tuning module to each Transformer layer significantly reduces the number of parameters and computational overhead. Specifically, for One Transformer layer:

[0075] Let the first The input of the layer Transformer is The formula for calculating the hierarchical propulsion is:

[0076] in, yes The pre-trained weights of the frozen Transformer layer. It is the first The output of the layer, It is the first The propulsion parameters of the layer transformer.

[0077] Step S103: Based on the teacher model, the student model is fine-tuned using a knowledge distillation method to obtain a video hash model. Specifically, to mitigate the accuracy loss caused by quantization, this embodiment employs a knowledge distillation method, using the unquantized original model as the teacher model and the quantized hierarchical Propulsion adaptation model as the student model. The goal is to make the student model mimic the behavior of the teacher model as closely as possible, thereby reducing their differences in the feature space.

[0078] This embodiment provides a method for constructing a video hash model, which includes the following steps: Step S201: Construct the video hash big model as described in the above embodiment, and train the video hash big model to obtain the teacher model.

[0079] Specifically, training the teacher model involves freezing the parameters of the base model and training and adjusting the parameters of the hash model based on the contrastive learning loss function. This training process to obtain the teacher model can be called the full-precision pre-training stage. In this stage, the lower-level feature extraction layer (i.e., the base model) of the large video hash model uses a pre-trained model of Video-LlaMA and remains frozen during training; the upper-level hash encoding layer (i.e., the hash model) uses random initialization and is optimized during training. The optimization goal of this stage is to use the InfoNCE loss function for hash encoding learning, so that similar samples have similar encodings in the hash space, while dissimilar samples have encodings that are as far apart as possible. This ensures that the model can learn good semantic representations.

[0080] The InfoNCE loss consists of three types of samples: an anchor sample. A positive sample ,and negative samples Anchor samples and positive samples belong to the same class, while anchor samples and negative samples belong to different classes. This InfoNCE loss maximizes the consistency between positive pairs and the inconsistency between positive and negative samples, and its formula is:

[0081] in , and These represent the embedding vectors of the anchor sample, positive sample, and negative sample, respectively. For similarity functions (such as cosine similarity). It is a hyperparameter that controls the discrimination of negative samples.

[0082] Step S202: The parameters of the basic model are quantized using a first quantization method, and the parameters of the hash model are quantized using a second quantization method. A Propulsion parameter is introduced into the hash model to obtain a student model. The quantization error of the first quantization method is less than that of the second quantization method. The first quantization method is the NF4 quantization method, and the second quantization method is the NF2 quantization method.

[0083] Specifically, NF4 (4-bit Normal Float) is an optimized data type based on quantile quantization, specifically designed for quantizing normally distributed parameters with a mean of zero. Its goal is to ensure that each quantization region contains the same number of input values, thereby reducing information loss. The core idea of ​​quantile quantization is: first, to calculate the empirical cumulative distribution function of the input tensor; then, by estimating the tensor's 2-1... 4 +1 = 17 quantiles, dividing the input data into 16 intervals; since calculating precise quantiles is expensive, approximate algorithms are usually used to estimate quantiles. However, the main problem with quantile quantization is that approximate quantile estimation algorithms may introduce large quantization errors, especially when there are many outliers.

[0084] To overcome the above problems, the NF4 quantization method proposes a more efficient solution: assuming that the neural network weights follow a standard normal distribution (i.e., zero mean and standard deviation). ); by scaling This ensures that the parameter distribution fits perfectly within the [-1,1] interval, thus matching the range of the quantized data type. The 17 quantiles of the theoretical normal distribution N(0,1) are calculated, resulting in 16 discrete quantized values:

[0085] in It is the quantile function of the standard normal distribution.

[0086] NF2 is an extremely compressed version of NF4 quantization. Its basic idea is the same as NF4, but it further reduces the number of bits: NF2 quantization uses only 4 quantization levels (2-bit), that is:

[0087] Because NF2 quantization has fewer levels, it has a larger quantization error. However, it significantly reduces storage footprint and computational overhead, making it ideal for computationally less demanding parts of the model, such as hash encoding in later stages.

[0088] Specifically, in this embodiment, the NF4 quantization method is used for the lower-layer base model. This layer is based on the Transformer structure and has a large number of parameters. Using NF4 quantization can effectively reduce quantization errors while maintaining the strong representational capability of the base model. For the upper-layer hash model, the more aggressive NF2 quantization method is used. This layer has relatively fewer parameters, and using NF2 quantization can further compress the parameters. This hybrid quantization strategy fully considers the characteristics and roles of different layers of the model. By using a lower-precision quantization method in the lower layer, key feature information is preserved; while using an even lower-precision quantization method in the upper layer, the model size is further reduced. This layered quantization design reduces storage and computational overhead while maximizing the model's performance. The NF4 quantization formula for the lower-layer base model is as follows:

[0089] in These are the 16 predefined quantization center values ​​of NF4. During the calculation process, the quantized weights need to be adjusted. Inverse quantization can be performed. Since NF4 quantization is based on quantile quantization, a predefined lookup table (LUT) can be used to perform inverse quantization. Mapping to floating-point values:

[0090] LUT is recovered via lookup table. To approximate floating-point values.

[0091] For the hash model layer, NF2 quantization is used, and its quantization and dequantization methods are similar to those of NF4.

[0092] Assume the parameters of the base model layer are The parameters of the hash model layer are First, respectively and Perform NF4 and NF2 quantization to obtain and :

[0093]

[0094] For a containing Keyframes and audio clip For the video data as a whole, the formula for its forward propagation process is as follows:

[0095]

[0096] in Represents the feature extraction layer. These are visual features and audio features, respectively; This represents the hash coding layer, used to calculate hash features. During the forward propagation process, the quantized parameters are first... and The data is dequantized to floating-point numbers and then input into the corresponding network layers for computation. This significantly reduces the model size during the inference phase while maintaining accuracy comparable to floating-point models.

[0097] Step S203: Based on the teacher model, the student model is fine-tuned using the knowledge distillation method to obtain the video hash model.

[0098] Specifically, step S203 includes: Step S2031: Construct a distillation loss function based on the output of the base model in the teacher model and the output of the hash model. In this embodiment, when constructing the distillation loss function, not only are the final output features of the teacher model and the student model aligned, but their intermediate hidden state features are also aligned. Let the hidden state of the teacher model be represented as... The hidden state of the student model is represented as The output of the teacher model is represented as The output of the student model is represented as The distillation loss function is then defined as:

[0099] in It is the first The alignment weights of the hidden states of each layer control the contribution of different layers to the total loss. These are the alignment weights of the final output features. It is a distance metric function; in this embodiment, the mean squared error (MSE) is used.

[0100] Additionally, it should be noted that the hidden state output refers to the feature output extracted by the basic model, while the model output refers to the binary hash code output after passing through the hash model.

[0101] Step S2032: Construct a total loss function based on the distillation loss function and the contrastive learning loss function; specifically, the total loss function is expressed as:

[0102] in These are hyperparameters that control the relative weights of the two losses. Using these two losses allows for simultaneous optimization of the model's task adaptability and hash encoding quality. In actual training, these can be... Set it to 0.1.

[0103] Step S2033: Based on the total loss function, the parameters of the student model are trained and adjusted to obtain the video hash model. Specifically, based on the distillation loss function in the total loss function, the intermediate features and final output of the student model (Model S) can be made as close as possible to the teacher model (Model T). Simultaneously, by combining the InfoNCE loss function in the total loss function, the hash encoding can be further optimized, giving it good retrieval performance in downstream tasks. This stage explores a combination scheme of the two loss functions to achieve better fine-tuning results.

[0104] As one or more specific application embodiments of the present invention, the method for constructing the video hash model is implemented using the following process:

[0105] Furthermore, Table 1 below presents the mAP@100 results of different video hashing methods on the ActivityNet and FCVID datasets. By comparing the performance of the hybrid quantization and hierarchical propulsion fine-tuning method proposed in this embodiment with existing methods, their performance in video retrieval tasks can be comprehensively evaluated. It can be seen that the method in this embodiment achieves significant performance improvements on both datasets. These results demonstrate that by combining a hybrid quantization strategy and efficient parameter fine-tuning techniques, the accuracy of video retrieval can be significantly improved while reducing computational and storage overhead.

[0106] Table 4.1 mAP@100 results of different methods on ActivityNet and FCVID datasets

[0107] Among them, MCMSH (Multi-Constraint Multi-modal Supervised Hashing), BTH (Binary Tree Hashing), DKPH (Deep Kernel Preserving Hashing), SSTH (Semantic Structure-Preserving Video Hashing), ConMH (Contrastive Multi-view Hashing), DSVH (Deep Spatiotemporal Video Hashing), SRH (Semantic-Regularized Hashing), AVH (Attention-based Video Hashing), AVHash (Audio-Visual Hashing), and Video-HLM (Video Hash Large Model) are all video hashing methods used in related technologies, and will not be elaborated further here.

[0108] By introducing learnable projection matrices at each level of the Transformer block, hierarchical propulsion only needs to update these newly added parameters during the fine-tuning phase, while keeping the original model parameters unchanged. This significantly reduces the number of parameters during training and inference. This parameter-efficient characteristic makes hierarchical propulsion more practical in resource-constrained scenarios, significantly reducing computational and storage overhead while maintaining high performance. Therefore, hierarchical propulsion offers an attractive option when balancing efficiency and accuracy, especially in applications with strict limitations on model size and inference speed.

[0109] This embodiment proposes a video hashing method (i.e., a method for constructing a video hashing model) based on low-bit mixed quantization and hierarchical propulsion fine-tuning, and verifies its significant effect through a series of experiments. Experimental results show: First, low-bit mixed quantization technology maintains high retrieval performance while significantly reducing computational and storage costs. This fully demonstrates that low-bit mixed quantization achieves a superior balance between efficiency and accuracy.

[0110] Secondly, hierarchical propulsion is an efficient parameter fine-tuning method. Compared to propulsion, this method significantly reduces the number of training parameters, achieving higher parameter efficiency and demonstrating a good trade-off between performance and efficiency.

[0111] In summary, the low-bit mixed quantization and hierarchical propulsion fine-tuning method proposed in this embodiment has achieved significant performance improvement in video hash retrieval tasks, providing an innovative solution for efficient and accurate video retrieval.

[0112] This embodiment also provides a video hash model construction apparatus, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0113] This embodiment provides a device for constructing a video hash model, such as... Figure 5 As shown, it includes: The teacher model construction module 51 is used to construct the video hash large model as described in any of the above embodiments, and to train the video hash large model to obtain the teacher model; The student model construction module 52 is used to quantize the parameters of the basic model using a first quantization method, quantize the parameters of the hash model using a second quantization method, and introduce the Propulsion parameter into the hash model to obtain the student model. The quantization error of the first quantization method is smaller than the quantization error of the second quantization method. The fine-tuning module 53 is used to fine-tune the student model based on the teacher model using a knowledge distillation method to obtain a video hash model.

[0114] The video hash model construction apparatus provided in this embodiment of the invention can execute the video hash model construction method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution. Further functional descriptions of the above modules and units are the same as in the corresponding embodiments described above, and will not be repeated here.

[0115] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0116] The following is a detailed reference. Figure 6 This diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 11, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 12 or a program loaded from memory 18 into random access memory (RAM) 13. The RAM 13 also stores various programs and data required for the operation of the electronic device. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0117] Typically, the following devices can be connected to I / O interface 15: input devices 16 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 17 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 18 including, for example, magnetic tapes, hard disks, etc.; and communication devices 19. Communication device 19 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0118] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 19, or installed from a memory 18, or installed from a ROM 12. When the computer program is executed by the processor 11, it performs the functions defined in the method for constructing the video hash model according to embodiments of the present invention.

[0119] Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0120] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the method for constructing the video hash model shown in the above embodiments is implemented.

[0121] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0122] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A large video hashing model, characterized in that, The large video hash model includes: The basic model is used to acquire video frame data and audio data of video data. The Video-LlaMA large model is used to extract features from the video frame data and audio data respectively to obtain video modal features and audio modal features. The hash model includes a cross-modal attention layer, a multimodal encoder, and a hash layer. The cross-modal attention layer is used to capture the semantic interaction between the video modal features and the audio modal features using cross-attention to obtain the fused features after interaction. The multimodal encoder encodes the fused features to obtain the encoded output. The hash layer is used to perform hash processing on the encoded output to obtain the hash code.

2. The video hashing big model according to claim 1, characterized in that, The basic model includes: The input layer is used to acquire video data, which includes a video frame sequence containing preset frames and a preset number of audio segments corresponding to each frame; A modality-specific encoder is used to map each frame in a video frame sequence to an image embedding vector using an image encoder to obtain a video frame representation. Position encoding is added to the video frame representation. The position-encoded frame representations are aggregated to obtain a video representation of a preset length. The video representation is then subjected to a linear transformation and projection to obtain video modality features. The modality-specific encoder is also used to convert audio segments into spectrograms, map each spectrogram into a dense vector using an audio encoder to obtain an audio representation, add positional encoding to the audio representation, aggregate the position-encoded audio representations to obtain an audio representation of a preset length, and perform a linear transformation projection on the audio representation to obtain audio modal features.

3. The video hashing big model according to claim 1, characterized in that, The cross-modal attention layer is also used to generate cross-modal video features and audio features using a multi-head cross-attention mechanism with a linear fully connected layer, and to fuse the cross-modal video features and audio features to obtain fused features; The multimodal encoder is also used to encode the fusion features with added CLS tokens and location information using multiple Transformer layers, and to use the output vector corresponding to CLS in the encoded output sequence as the encoded output. The hash layer is also used to convert the encoded output into a binary real-valued vector using a fully connected network with a hyperbolic tangent function, and to process the binary real-valued vector using a sign function to obtain a binary hash code.

4. A method for constructing a video hash model, characterized in that, The method includes: Construct a large video hash model as described in any one of claims 1-3, and train the large video hash model to obtain a teacher model; The parameters of the basic model are quantized using a first quantization method, and the parameters of the hash model are quantized using a second quantization method. A Propulsion parameter is introduced into the hash model to obtain a student model. The quantization error of the first quantization method is smaller than that of the second quantization method. Based on the teacher model, the student model is fine-tuned using the knowledge distillation method to obtain the video hash model.

5. The method according to claim 4, characterized in that, The first quantization method is the NF4 quantization method, and the second quantization method is the NF2 quantization method.

6. The method according to claim 4, characterized in that, The teacher model obtained through training includes: The parameters of the base model are frozen, and the parameters of the hash model are trained and adjusted based on the contrastive learning loss function to obtain the teacher model.

7. The method according to claim 4, characterized in that, Based on the teacher model, the student model is fine-tuned using a knowledge distillation method to obtain a video hash model, including: A distillation loss function is constructed based on the output of the base model in the teacher model and the output of the hash model. Construct a total loss function based on the distillation loss function and the contrastive learning loss function; Based on the total loss function, the parameters of the student model are trained and adjusted to obtain the video hash model.

8. A device for constructing a video hash model, characterized in that, The device includes: A teacher model building module is used to build a large video hash model as described in any one of claims 1-3, and to train the large video hash model to obtain a teacher model; The student model construction module is used to quantize the parameters of the basic model using a first quantization method, quantize the parameters of the hash model using a second quantization method, and introduce the Propulsion parameter into the hash model to obtain the student model. The quantization error of the first quantization method is smaller than the quantization error of the second quantization method. The fine-tuning module is used to fine-tune the student model based on the teacher model using a knowledge distillation method to obtain a video hash model.

9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the method for constructing the video hash model according to any one of claims 4 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method for constructing the video hash model according to any one of claims 4 to 7.

Images