Self-supervised video hashing method and device and electronic equipment

Abstract

The invention provides a self-supervised video hashing method and device and electronic equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: determining a key frame of a video to be processed; inputting the key frame of the to-be-processed video into a video Hash model to obtain a video Hash code output by the video Hash model; the video Hash model is obtained by carrying out clustering guide learning and comparative learning loop training on the basis of key frames of a sample video to be processed, and the expression ability of video Hash codes can be improved through a multi-task learning strategy of clustering guide learning and comparative learning loop training.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a self-supervised video hashing method, device and electronic equipment. Background Art

[0002] With the explosive growth of video content on social media platforms and search engines, efficiently storing, retrieving, and analyzing large amounts of video content has become a key research topic in information technology. Video hashing, a method for mapping video content into compact binary codes, has attracted widespread attention from both academia and industry due to its advantages in fast similarity search and low storage requirements.

[0003] Existing video hashing methods typically rely on supervised or unsupervised learning for model training. Supervised learning methods use large amounts of labeled data to guide the model in generating discriminative hash codes, but manual labeling is costly and difficult to adapt to diverse video content. Unsupervised learning methods optimize based on the statistical characteristics of data distribution. While they don't require labeling, they often struggle to fully capture the semantic information of videos. Therefore, improving the expressiveness of hash codes is a critical issue currently under investigation in the industry. Summary of the Invention

[0004] In response to the problems existing in the prior art, the present invention provides a self-supervised video hashing method, device and electronic device.

[0005] The present invention provides a self-supervised video hashing method, comprising: Determine the key frames of the video to be processed; The key frames of the video to be processed are input into a video hash model to obtain a video hash code output by the video hash model; the video hash model is obtained by performing clustering guided learning and contrastive learning cycle training based on the key frames of the sample video to be processed.

[0006] According to a self-supervised video hashing method provided by the present invention, the video hashing model includes an encoding module and a decoding module; Before performing clustering guided learning and contrastive learning cyclic training based on the sample to-be-processed video, the method further includes: Performing a first frame mask process on the sample video to be processed to obtain a pre-training sample; Inputting the pre-training sample into the encoding module to obtain a predicted video hash code output by the encoding module; Inputting the predicted video hash code into the decoding module to obtain the predicted video output by the decoding module; A first reconstruction loss is determined based on the predicted video and the video to be processed, and parameters of the original video hash model are pre-adjusted based on the first reconstruction loss; the first frame mask processing of the sample video to be processed is repeated to obtain a pre-trained sample process until the original video hash model converges to obtain a pre-trained video hash model.

[0007] According to a self-supervised video hashing method provided by the present invention, after obtaining a pre-trained video hashing model, the method further includes: Performing a second frame mask process on the sample video to be processed to obtain a non-overlapping first training sample and a second training sample; inputting the first training sample into the encoding module to obtain a first predicted video hash code output by the encoding module; inputting the second training sample into the encoding module to obtain a second predicted video hash code output by the encoding module; determining a cluster-guided learning loss based on each of the first predicted video hash code and the second predicted video hash code, and initially adjusting parameters of the pre-trained video hash model according to the cluster-guided learning loss; Performing a second frame mask process on the sample video to be processed to obtain a non-overlapping first training sample and a second training sample; inputting the first training sample into the encoding module to obtain a third predicted video hash code output by the encoding module; inputting the second training sample into the encoding module to obtain a fourth predicted video hash code output by the encoding module; Determining a contrastive learning loss based on the corresponding third predicted video hash code and the fourth predicted video hash code, and adjusting parameters of the pre-trained video hash model secondary according to the contrastive learning loss; Repeating the second frame mask processing on the sample video to be processed to obtain non-overlapping first training samples and second training samples, and optimizing the process in a loop until the pre-trained video hash model converges to obtain the video hash model.

[0008] According to a self-supervised video hashing method provided by the present invention, before performing the second frame masking process on the sample videos to be processed, the method further includes: performing a first frame mask process on each of the sample videos to be processed to obtain a third training sample and a fourth training sample that do not overlap; inputting the third training sample into the encoding module to obtain a fifth predicted video hash code output by the encoding module; and inputting the fourth training sample into the encoding module to obtain a sixth predicted video hash code output by the encoding module; performing clustering processing on each of the fifth predicted video hash code and the sixth predicted video hash code to obtain a set number of cluster centers; Determining a cluster-guided learning loss based on each of the first predicted video hash code and the second predicted video hash code includes: A cluster-guided learning loss is determined based on the first predicted video hash codes, the second predicted video hash codes, and the hash code cluster centers.

[0009] According to a self-supervised video hashing method provided by the present invention, determining a cluster-guided learning loss based on each of the first predicted video hash codes, each of the second predicted video hash codes, and the hash code cluster center includes: Determining a clustering loss for each predicted video hash code and each of the hash code cluster centers, determining a second reconstruction loss for each of the predicted video hash codes, and determining a cluster-guided learning loss based on the clustering loss and the second reconstruction loss; The determining of the contrastive learning loss based on the corresponding third predicted video hash code and the fourth predicted video hash code includes: Determine a contrastive loss between the corresponding third predicted video hash code and the fourth predicted video hash code, determine a third reconstruction loss for each of the predicted video hash codes, and determine a contrastive learning loss based on the contrastive loss and the third reconstruction loss.

[0010] According to a self-supervised video hashing method provided by the present invention, a first frame mask processing is performed on the sample video to be processed to obtain a pre-training sample, comprising: A mask ratio corresponding to the sample video to be processed is determined, and a first frame mask processing is performed on the sample video to be processed based on the mask ratio to obtain a pre-training sample; the pre-training sample includes 2-3 key frames.

[0011] According to a self-supervised video hashing method provided by the present invention, the encoding module includes a state space encoder and a multi-head attention layer; Obtaining the predicted video hash code output by the encoding module, including: Obtaining conversion features of the encoding module performing layer normalization and wide-dimensional conversion processing on the pre-training samples; Input the conversion feature into the state space encoder to obtain the intermediate feature output by the state space encoder; input the intermediate feature into the multi-head attention layer to obtain the encoded feature output by the multi-head attention layer; Pooling operations and binarization processing are performed on the encoding features to obtain a predicted video hash code.

[0012] The present invention also provides a self-supervised video hashing device, comprising: A video determination module, used to determine the key frames of the video to be processed; The video hash module is used to input the key frames of the video to be processed into a video hash model to obtain the video hash code output by the video hash model; the video hash model is obtained by clustering guided learning and contrast learning cycle training based on the key frames of the sample video to be processed.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the self-supervised video hashing method as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the self-supervised video hashing methods described above.

[0015] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the self-supervised video hashing methods described above.

[0016] The self-supervised video hashing method, device and electronic device provided by the present invention use clustering-guided learning and contrastive learning for cyclic training, which can avoid clustering-guided learning or contrastive learning completely dominating the training process and maintain a balance between learning tasks. This allows the trained video hash model to output a video hash code after receiving a video to be processed, which can globally cluster similar videos together and locally retain fine-grained features of the video, thereby improving the expressive power of the video hash code and achieving more effective video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 It is a flowchart of the self-supervised video hashing method provided by the present invention.

[0019] Figure 2 2 is a training diagram of the self-supervised video hashing method provided by the present invention.

[0020] Figure 3 This is one of the effect schematic diagrams of the self-supervised video hashing method provided by the present invention.

[0021] Figure 4 The second schematic diagram of the effect of the self-supervised video hashing method provided by the present invention.

[0022] Figure 5 It is a structural diagram of the self-supervised video hashing device provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0025] The following combination Figures 1-6 The present invention describes the self-supervised video hashing method, apparatus and electronic device.

[0026] Figure 1 It is a flow chart of the self-supervised video hashing method provided by the present invention, such as Figure 1 As shown, the method includes the following steps: Step 101: Determine the key frames of the video to be processed.

[0027] The video to be processed refers to high-dimensional video data that will be mapped to a video hash code. The video to be processed can be video content from a social media platform or a search engine. For example, the key frames of the video to be processed can be determined from multiple videos in a designated storage area.

[0028] Step 102: Input the key frames of the video to be processed into a video hash model to obtain a video hash code output by the video hash model; the video hash model is obtained by performing clustering guided learning and contrastive learning cycle training based on the key frames of the sample video to be processed.

[0029] Video hash codes are compact binary codes obtained by mapping high-dimensional video data. Video hash codes can be used for fast retrieval by matching similar hash codes.

[0030] Cyclic training means that during the training process, the original video hash model alternates between cluster-guided learning and contrastive learning. For example, if the maximum number of iterations for training the original video hash model is preset to mIter=100, then each cycle will perform cluster-guided learning and contrastive learning in sequence. Each cycle is repeated until the number of iterations reaches 100, completing the cyclic training and obtaining the video hash model.

[0031] Clustering-guided learning can facilitate the establishment of a structured latent space so that semantically similar videos are grouped together; contrastive learning can enhance the ability of the trained video hash model to capture similarities and maintain semantic consistency between enhanced views; in this way, through the clustering-guided learning and contrastive learning cycle training, the trained video hash model outputs a video hash code that can globally cluster similar videos together and locally retain the fine-grained features of the video.

[0032] The self-supervised video hashing method provided by the embodiment of the present invention, clustering-guided learning and contrastive learning cyclic training, can avoid clustering-guided learning or contrastive learning completely dominating the training process, maintain the balance between learning tasks, so that the trained video hash model can output a video hash code after receiving the video to be processed, which can globally cluster similar videos together and locally retain the fine-grained features of the video, thereby improving the expressive power of the video hash code and achieving more effective video retrieval.

[0033] Based on the above embodiment, the video hash model includes an encoding module and a decoding module.

[0034] Specifically, the encoding module is a model component that converts the processed video into a binary video hash code, and the decoding module is a model component that reconstructs the masked video frame from the binary video hash code.

[0035] Before performing clustering guided learning and contrastive learning cyclic training based on the sample to-be-processed video, the method further includes: Performing a first frame mask process on the sample video to be processed to obtain a pre-training sample; Inputting the pre-training sample into the encoding module to obtain a predicted video hash code output by the encoding module; Inputting the predicted video hash code into the decoding module to obtain the predicted video output by the decoding module; A first reconstruction loss is determined based on the predicted video and the video to be processed, and parameters of the original video hash model are pre-adjusted based on the first reconstruction loss; the first frame mask processing of the sample video to be processed is repeated to obtain a pre-trained sample process until the original video hash model converges to obtain a pre-trained video hash model.

[0036] Specifically, frame masking refers to masking certain video frames in the video to be processed to generate pre-training samples. First frame masking refers to frame masking using a first masking standard. It will be appreciated that in this embodiment, the first and second standards are primarily used to distinguish between frame masking using different masking standards, rather than to define a specific order.

[0037] In one embodiment, a first frame mask processing is performed on the sample video to be processed to obtain a pre-training sample, including: determining a mask ratio corresponding to the sample video to be processed, and performing a first frame mask processing on the sample video to be processed based on the mask ratio to obtain a pre-training sample; the pre-training sample includes 2-3 key frames.

[0038] The mask ratio is a hyperparameter that controls the proportion of video frames to be masked during frame masking. For example, a large fixed mask ratio can be determined during the pre-training phase. This fixed mask ratio is used to perform the first frame masking of the processed video, which consists of a mini-training sample of 2-3 key frames.

[0039] In this embodiment, the video category is determined by using several key frames, which shortens the sequence of video frames required for determining the video category, reduces the complexity of temporal changes introduced by the video frame sequence, and thus reduces the difficulty of identifying the video category.

[0040] Moreover, in this embodiment, 2-3 key frames can better represent the essential characteristics of the category of the video to be classified, so that the trained video hash model can learn more generalized semantic features and output video hash codes that can globally cluster similar videos together.

[0041] In one embodiment, the encoding module includes a state-space encoder and a multi-head attention layer.

[0042] Specifically, a state-space encoder is an encoding module component that encodes transformation features to obtain intermediate features. For example, the state-space encoder can be a Mamba sequence model architecture, a Transformer neural network architecture, or a hybrid architecture of Mamba and Transformer. The state-space encoder can encode transformation features to obtain intermediate features through methods such as sequence modeling.

[0043] The multi-head attention layer is an encoding module component that acts on the output of the state-space encoder to obtain encoded features. The multi-head attention layer is used to calculate the multi-head attention mechanism on the intermediate features.

[0044] Obtaining the predicted video hash code output by the encoding module, including: Obtaining conversion features of the encoding module performing layer normalization and wide-dimensional conversion processing on the pre-training samples; Input the conversion feature into the state space encoder to obtain the intermediate feature output by the state space encoder; input the intermediate feature into the multi-head attention layer to obtain the encoded feature output by the multi-head attention layer; Pooling operations and binarization processing are performed on the encoding features to obtain a predicted video hash code.

[0045] Specifically, layer normalization refers to standardizing the key frame set of pre-training samples, which can stabilize the subsequent training process.

[0046] Dimension conversion processing refers to the process of linearly transforming the key frame set of the normalized pre-training samples and projecting them from the original dimension to the hidden layer dimension of the state space encoder. Dimension conversion processing can map the key frame set of the normalized high-dimensional pre-training samples to the low-dimensional state space of the state space encoder. It can be understood that wide dimensional conversion processing means that the hidden layer of the state space encoder is a wider hidden layer than the conventional hidden layer. Exemplarily, the size of the hidden layer corresponding to the wide dimensional conversion processing can be 512, or 1024, etc., which is not further limited in this embodiment.

[0047] The intermediate features can be expressed by the following formula: in, is the intermediate feature, For conversion features.

[0048] For example, the multi-head attention layer can be a multi-head attention layer with residual connections. The residual connections allow the original input information to be directly passed to the next layer, that is, the intermediate features output by the state-space encoder can be directly passed to the next layer. In this way, the encoded features output by the multi-head attention layer are formed by adding the output of the multi-head attention layer itself and the intermediate features output by the state-space encoder, which can reduce the risk of information loss or gradient vanishing.

[0049] In one embodiment, the intermediate features may be normalized before being input into the multi-head attention layer. On this basis, the encoded features can be specifically expressed by the following formula: in, is the encoding feature, is the normalized intermediate feature.

[0050] Pooling operation refers to a processing method that aggregates the encoded features in a specified dimension and compresses high-dimensional data into a low-dimensional representation, which is used to extract key features or generate compact output.

[0051] Binarization refers to the process of mapping continuous-valued features into binary values ​​through a sign function, which is used to generate a compact hash code suitable for fast retrieval.

[0052] Exemplarily, the encoded features can be linearly transformed to obtain k-dimensional features and normalized, and the normalized features can be processed by the tanh activation function to obtain the output result. Then, the output result can be average pooled and further signed by the sign function to map it to a binary value of -1 or 1 to obtain the final binary predicted video hash code.

[0053] The predicted video hash code obtained by pooling and binarizing the encoded features can be expressed by the following formula: in, is the output of the tanh activation function, To predict the video hash code, is the average pooling operation, It is a binary processing.

[0054] For example, the decoding module may first process the predicted video hash code to obtain the aforementioned output result , and then use the trainable parameters to Fill until The length of the original video reaches N to obtain the filling feature, which can be expressed by the following formula: in, is the filling feature, is a trainable parameter.

[0055] The decoding module includes a hidden layer and a multi-head attention layer, where the size of the hidden layer of the decoding module can be the same as that of the encoding module, and the multi-head attention layer can be connected with residuals. The padded features can be transformed to obtain the reconstructed encoding features. , which can be expressed by the following formula: The reconstructed encoding features can be processed by the multi-head attention layer to obtain the reconstructed intermediate features , which can be expressed by the following formula: The predicted video can be obtained by performing dimension conversion on the reconstructed intermediate features through the hidden layer , which can be expressed by the following formula: It is understandable that It is the set of visible video frames in the pre-training sample obtained after the first frame mask processing. is the set of video frames masked in the first frame masking process, and The sum of is the set of video frames of the video to be processed, or yes The complement of .

[0056] Can be calculated The original video frame corresponds to the predicted video The first reconstruction loss is determined by the cosine similarity between the predicted video frames of each video frame in , which can be expressed by the following formula: Among them, cos(,) represents cosine similarity, express Each original video frame, Indicates the corresponding Each predicted video frame for each video frame in .

[0057] In this embodiment, by masking the first frame of the sample video to be processed, the pre-trained video hash model is able to infer the complete video data from the partially masked video data, thereby enabling the pre-trained video hash model to capture the deep semantic features of the video data. The resulting pre-trained video hash model can provide high-quality initial features and basic learning capabilities in subsequent cluster-guided learning and contrastive learning, thereby improving the expressiveness of the video hash code output by the trained video hash model.

[0058] Based on any of the above embodiments, after obtaining the pre-trained video hash model, the method further includes: Performing a second frame mask process on the sample video to be processed to obtain a non-overlapping first training sample and a second training sample; inputting the first training sample into the encoding module to obtain a first predicted video hash code output by the encoding module; inputting the second training sample into the encoding module to obtain a second predicted video hash code output by the encoding module; determining a cluster-guided learning loss based on each of the first predicted video hash code and the second predicted video hash code, and initially adjusting parameters of the pre-trained video hash model according to the cluster-guided learning loss; Performing a second frame mask process on the sample video to be processed to obtain a non-overlapping first training sample and a second training sample; inputting the first training sample into the encoding module to obtain a third predicted video hash code output by the encoding module; inputting the second training sample into the encoding module to obtain a fourth predicted video hash code output by the encoding module; Determining a contrastive learning loss based on the corresponding third predicted video hash code and the fourth predicted video hash code, and adjusting parameters of the pre-trained video hash model secondary according to the contrastive learning loss; Repeating the second frame mask processing on the sample video to be processed to obtain non-overlapping first training samples and second training samples, and optimizing the process in a loop until the pre-trained video hash model converges to obtain the video hash model.

[0059] Specifically, the number of video frames included in the training sample obtained by the first frame masking process is smaller than the number of video frames included in the training sample obtained by the second frame masking process. Exemplarily, the training sample obtained by the first frame masking process may include 2 video frames, and the training sample obtained by the second frame masking process may include 4 video frames.

[0060] It is understandable that the first frame mask processing and the second frame mask processing regulate the mask ratio, rather than the mask order, etc. Therefore, even if the second frame mask processing is performed twice on the same sample video to be processed, the obtained training samples may be different.

[0061] It is understandable that even if the same training sample is input into the encoding module, the predicted video hash code obtained before the parameters of the pre-trained video hash model are initially adjusted according to the clustering guided learning loss is different from the predicted video hash code obtained after the parameters of the pre-trained video hash model are initially adjusted according to the clustering guided learning loss.

[0062] The effect of adjusting the parameters of the pre-trained video hash model twice based on comparative optimization learning on the output of the encoding module is essentially the same as the effect of adjusting the parameters of the pre-trained video hash model initially based on cluster-guided learning loss on the output of the encoding module, and is not further described in this embodiment. The effect of adjusting the parameters of the pre-trained video hash model each time on the output of the decoding module is essentially the same as the effect of adjusting the parameters of the pre-trained video hash model each time on the output of the encoding module, and is not further described in this embodiment.

[0063] Exemplarily, the number of loop iterations may be preset, and when the actual number of loop iterations reaches the preset maximum number of loop iterations, it is determined that the pre-trained video hash model has converged, and the video hash model is obtained.

[0064] In this embodiment, training samples are generated in each cycle, which can reduce the risk of the trained video hash model falling into a local optimum.

[0065] Based on any of the above embodiments, before performing the second frame mask processing on the sample videos to be processed, the method further includes: performing a first frame mask process on each of the sample videos to be processed to obtain a third training sample and a fourth training sample that do not overlap; inputting the third training sample into the encoding module to obtain a fifth predicted video hash code output by the encoding module; and inputting the fourth training sample into the encoding module to obtain a sixth predicted video hash code output by the encoding module; performing clustering processing on each of the fifth predicted video hash code and the sixth predicted video hash code to obtain a set number of cluster centers; Determining a cluster-guided learning loss based on each of the first predicted video hash code and the second predicted video hash code includes: A cluster-guided learning loss is determined based on the first predicted video hash codes, the second predicted video hash codes, and the hash code cluster centers.

[0066] Specifically, the predicted video hash code used for clustering is obtained through masking the first frame. As previously mentioned, the first frame masking process can generate a mini training sample of 2-3 key frames. This mini training sample can better capture the essential characteristics of the video category, avoid the complex temporal changes that may be introduced by long sequences, and improve the stability of the generated cluster centers.

[0067] For example, the generated hash code may be Apply K-means clustering to obtain N c Cluster centers The cluster centers are randomly initialized in the first iteration and then iteratively updated based on the results of the previous round.

[0068] In this embodiment, by iteratively updating the cluster center during each optimization cycle, it is possible to recalculate according to the distribution of the currently predicted video hash code, thereby dynamically adapting to the learning process of the pre-trained video hash model, continuously optimizing the structure of the latent space, and increasing the semantic similarity of video samples assigned to the same cluster, thereby increasing the reliability of the clustering-guided learning loss determined based on the hash code cluster center, and further increasing the expressive power of the video hash code output by the trained video hash model.

[0069] Based on any of the above embodiments, determining a cluster-guided learning loss based on each of the first predicted video hash codes, each of the second predicted video hash codes, and the hash code cluster center includes: Determining a clustering loss for each predicted video hash code and each of the hash code cluster centers, determining a second reconstruction loss for each of the predicted video hash codes, and determining a cluster-guided learning loss based on the clustering loss and the second reconstruction loss; The determining of the contrastive learning loss based on the corresponding third predicted video hash code and the fourth predicted video hash code includes: Determine a contrastive loss between the corresponding third predicted video hash code and the fourth predicted video hash code, determine a third reconstruction loss for each of the predicted video hash codes, and determine a contrastive learning loss based on the contrastive loss and the third reconstruction loss.

[0070] The clustering loss between the predicted video hash code and the hash code cluster center can be determined by the cross entropy loss function , which can be expressed by the following formula: in, Represents the hash code of the i-th predicted video With all cluster centers The cosine similarity between represents the temperature hyperparameter.

[0071] Among them, the function Returns the index of the cluster center with the highest cosine similarity to the i-th sample.

[0072] The working principle and technical effect of determining the second reconstruction loss are substantially the same as those of determining the first reconstruction loss, and are not described in detail in this embodiment.

[0073] In one embodiment, the clustering loss and the second reconstruction loss are weighted to determine the clustering guided learning loss. , which can be expressed by the following formula: in, is the second reconstruction loss, is the first balancing hyperparameter.

[0074] The contrast loss and the third reconstruction loss are weighted to determine the contrast learning loss , which can be expressed by the following formula: in, is the third reconstruction loss, is the second balancing hyperparameter, is the contrast loss.

[0075] In this embodiment, by integrating the reconstruction loss into the clustering-guided learning loss, the trained video hash model can reduce the risk of information loss due to over-compression or clustering bias when forming semantic clusters. The reconstruction loss in the contrastive learning loss ensures that local consistency learning does not deviate from the semantic content of the original video, thereby maintaining stability in fine-grained feature learning, and ultimately improving the accuracy of retrieval using the video hash code output by the video hash model.

[0076] Figure 2 This is a schematic diagram of the architecture of the data query method provided by the present invention. Figure 2 As shown, in order to specifically illustrate the functions of the self-supervised video hashing method provided by this implementation, a specific algorithm outline is provided below.

[0077] The algorithm outline of the encoding module is as follows: Input: video key frame features v, with batch size B, feature sequence length l and feature dimension d, encoding module hidden layer size d en , video hash code size k.

[0078] Output: video hash code sequence y, binary video hash code b.

[0079] 1: / *———— Feature dimension transformation————* / 2: 3: / *———— Time series processing————* / 4: 5: 6: / *———— Video hash code generation————* / 7: 8: 9: Returns: y, b.

[0080] The algorithm outline of the decoding module is as follows: Input: video hash code sequence y with batch size B, video hash code sequence length l and video hash code size k, decoding module hidden layer size d en , the original feature sequence length N and feature dimension d.

[0081] Output: reconstructed frame.

[0082] 1: / *———— Fill to the original feature sequence length————* / 2: 3: / *———— Dimension Transformation————* / 4: 5: / *———— Feature reconstruction————* / 6: 7: 8: Return: .

[0083] The algorithm outline of the training process is as follows: Input: Video dataset , the number of key frames is N, the feature dimension is d, the maximum number of pre-training iterations mIterpre and the maximum number of training iterations mIter, hyper parameters lr, α, β, σ, N c ,γ pre ,γ.

[0084] Output: trained video hashing model parameters θ.

[0085] 1: / *————— Pre-training stage————* / 2: 3: 4: Use γ pre generate and ; 5: 6: 7: 8: 9: 10: 11: 12: 13: / *———— Training phase————* / 14: 15: / *————Generate / update cluster center————* / 16: 17: Use γ pre generate and ; 18: 19: 20: twenty one: twenty two: 23: / *———— Clustering guided learning / reconstruction learning————* / twenty four: 25: Generate with γ and ; 26: 27: 28: 29: 30: 31: 32: 33: 34: 35: / *———— Contrastive Learning / Reconstruction Learning————* / 36: 37: Generate with γ and ; 38: 39: 40: 29: 30: 31: 32: 33: 34: 35: Returns: θ.

[0086] After training the video hash model, for a given new video , after preprocessing, its keyframe Indicates that the video hash code can be obtained by inputting it into the video hash model ,After obtaining the video hash code, its Hamming distance with the database ,video can be calculated and sorted, and the top K nearest neighbors can be ,returned as the retrieval result.

[0087] The self-supervised video hashing method proposed in this paper is compared with several self-supervised video hashing methods, such as SSVH (Song et al., 2018), BTH (Li et al., 2021), DKPH (Li et al., 2022), MCMSH (Hao et al., 2022), and ConMH (Wang et al., 2023). Retrieval performance is evaluated using the mean average precision (mAP@K) of the top K retrieval results, following the same evaluation protocol as in (Zhang et al., 2016). Experimental results are evaluated using hash code lengths of 32, 64, and 128 bits.

[0088] like Figure 3 As shown in the figure, the self-supervised video hashing method provided by the present invention consistently outperforms existing methods across different datasets and various topK settings. Specifically, in the mAP@100 metric on the FCVID dataset, the self-supervised video hashing method provided by the present invention improves upon the strongest baseline, ConMH, by 18.8%, 23.1%, and 27.4% at 32-, 64-, and 128-bit hash code lengths, respectively. The performance improvement on the ActivityNet dataset is even more significant, with the self-supervised video hashing method provided by the present invention exceeding ConMH by 44.2%, 44.8%, and 43.7% at the same bit settings, respectively. These consistent and significant improvements across different datasets and code lengths demonstrate the effectiveness and robustness of our proposed method.

[0089] Figure 4 The precision-recall curves for BTH, DKPH, MCMSH, ConMH, and the self-supervised video hashing method provided by the present invention for all code lengths on the FCVID and ActivityNet datasets are shown. It is clear from the curves that the self-supervised video hashing method provided by the present invention consistently maintains higher precision values ​​across the entire recall range, significantly outperforming all baseline methods.

[0090] In addition, we conducted a comprehensive ablation study using 64-bit hash codes on the FCVID dataset to verify the effectiveness of the training strategy proposed in this paper, as shown in the following table.

[0091] Table 2. Comparison of mAP@K results of different training strategies using 64-bit hash codes on the FCVID dataset.

[0092] In the table, WSCH is the self-supervised video hashing method provided by the present invention.

[0093] See the table above, examining the importance of each training step. By removing cluster-guided learning (WSCH w / o Step 2), performance degrades significantly, especially for larger values ​​of K (e.g., from 0.328 to 0.309 at K = 100). Removing contrastive learning (WSCH w / o Step 3) leads to an even more significant performance drop (e.g., from 0.558 to 0.531 at K = 5), demonstrating the critical role of contrastive learning in our framework. When removing the alternating training strategy (WSCH w / o AT), which combines cluster-guided learning and contrastive learning into a single step and simultaneously optimizes the reconstruction, clustering, and contrastive losses, performance degrades significantly, indicating that alternating optimization contributes to better learning.

[0094] We also investigated the impact of pre-training and cluster center updating. These two components aim to acquire and maintain accurate cluster centers for better clustering performance. When either component (WSCH without CU, WSCH without Pre) is missing, performance degradation is observed. When both components are missing (WSCH with CU and Pre), the cluster centers are completely randomized, and the performance degradation is even more pronounced.

[0095] Furthermore, when the self-supervised video hashing method proposed in this paper is replaced by contrastive pre-training (WSCH with ConPre) with a mask rate of 0.3 (the optimal setting for contrastive learning performance, see (Wang et al., 2023)), the results are inferior to WSCH without CU&Pre—the latter at least ensures a more uniform distribution of hash codes in the feature space. This demonstrates the effectiveness of the self-supervised video hashing training method proposed in this paper.

[0096] Comparing the self-supervised video hashing method provided by this invention with ConMH and its variant (ConMH+WSCH-T) using the training strategy provided by this invention shows that the baseline ConMH performs poorly for all values ​​of K. While applying the training strategy provided by this invention to ConMH (ConMH+WSCH-T) improves somewhat, it still falls short of the self-supervised video hashing method provided by this invention. This comparison demonstrates that both the model architecture and training strategy of the self-supervised video hashing method provided by this invention contribute to its superior performance.

[0097] The self-supervised video hashing device provided by the present invention is described below. The self-supervised video hashing device described below and the self-supervised video hashing method described above can be referenced to each other.

[0098] Figure 5 This is a schematic diagram of the structure of the self-supervisory video hashing device provided by the present invention. Figure 5As shown, the device includes: A video determination module 510 is used to determine key frames of a video to be processed; The video hash module 520 is used to input the key frames of the video to be processed into a video hash model to obtain a video hash code output by the video hash model; the video hash model is obtained by clustering guided learning and contrast learning cycle training based on the key frames of the sample video to be processed.

[0099] Based on any of the above embodiments, the video hash model includes an encoding module and a decoding module; The self-supervised video hashing device includes a model pre-training module for: Performing a first frame mask process on the sample video to be processed to obtain a pre-training sample; Inputting the pre-training sample into the encoding module to obtain a predicted video hash code output by the encoding module; Inputting the predicted video hash code into the decoding module to obtain the predicted video output by the decoding module; A first reconstruction loss is determined based on the predicted video and the video to be processed, and parameters of the original video hash model are pre-adjusted based on the first reconstruction loss; the first frame mask processing of the sample video to be processed is repeated to obtain a pre-trained sample process until the original video hash model converges to obtain a pre-trained video hash model.

[0100] Based on any of the above embodiments, the model pre-training module is specifically used to: determine the mask ratio corresponding to the sample video to be processed, and perform a first frame mask processing on the sample video to be processed based on the mask ratio to obtain a pre-training sample; the pre-training sample includes 2-3 key frames.

[0101] Based on any of the above embodiments, the encoding module includes a state space encoder and a multi-head attention layer; Model pre-training module, specifically used for: Obtaining conversion features of the encoding module performing layer normalization and wide-dimensional conversion processing on the pre-training samples; Input the conversion feature into the state space encoder to obtain the intermediate feature output by the state space encoder; input the intermediate feature into the multi-head attention layer to obtain the encoded feature output by the multi-head attention layer; Pooling operations and binarization processing are performed on the encoding features to obtain a predicted video hash code.

[0102] Based on any of the above embodiments, the self-supervised video hashing device includes a model training module for: Performing a second frame mask process on the sample video to be processed to obtain a non-overlapping first training sample and a second training sample; inputting the first training sample into the encoding module to obtain a first predicted video hash code output by the encoding module; inputting the second training sample into the encoding module to obtain a second predicted video hash code output by the encoding module; determining a cluster-guided learning loss based on each of the first predicted video hash code and the second predicted video hash code, and initially adjusting parameters of the pre-trained video hash model according to the cluster-guided learning loss; determining a contrastive learning loss based on the corresponding first predicted video hash code and the second predicted video hash code, and adjusting parameters of the pre-trained video hash model secondary according to the contrastive learning loss; Repeating the second frame mask processing on the sample video to be processed to obtain non-overlapping first training samples and second training samples, and optimizing the process in a loop until the pre-trained video hash model converges to obtain the video hash model.

[0103] Based on any of the above embodiments, the model training module is specifically used to: Performing clustering processing on each of the first predicted video hash code and the second predicted video hash code to obtain a set number of cluster centers; A cluster-guided learning loss is determined based on each predicted video hash code and the center of the hash code cluster.

[0104] Based on any of the above embodiments, the model training module is specifically used to: Determining a clustering loss for each predicted video hash code and each of the hash code cluster centers, determining a second reconstruction loss for each of the predicted video hash codes, and determining a cluster-guided learning loss based on the clustering loss and the second reconstruction loss; The determining of the contrastive learning loss based on the corresponding first predicted video hash code and the second predicted video hash code includes: Determine a contrastive loss between the corresponding first predicted video hash code and the second predicted video hash code, determine a third reconstruction loss for each of the predicted video hash codes, and determine a contrastive learning loss based on the contrastive loss and the third reconstruction loss.

[0105] Figure 6 The following is a schematic diagram of the structure of an electronic device, such as Figure 6As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communications bus 640. The processor 610 may call logic instructions in the memory 630 to execute a self-supervised video hashing method, which includes: determining key frames of a video to be processed; inputting the key frames of the video to be processed into a video hash model to obtain a video hash code output by the video hash model; the video hash model is obtained by performing cluster-guided learning and contrastive learning cyclic training based on the key frames of a sample video to be processed.

[0106] Furthermore, the logic instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0107] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the self-supervised video hash method provided by the above methods, which includes: determining the key frames of the video to be processed; inputting the key frames of the video to be processed into a video hash model to obtain a video hash code output by the video hash model; the video hash model is obtained by clustering guided learning and contrast learning cycle training based on the key frames of the sample video to be processed.

[0108] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the self-supervised video hash method provided by the above-mentioned methods, the method comprising: determining the key frames of the video to be processed; inputting the key frames of the video to be processed into a video hash model to obtain a video hash code output by the video hash model; the video hash model is obtained by performing clustering-guided learning and contrastive learning cycle training based on the key frames of the sample video to be processed.

[0109] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0110] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A self-supervised video hashing method, characterized in that include: Determine the key frames of the video to be processed; Inputting the key frame of the video to be processed into a video hash model to obtain a video hash code output by the video hash model; The video hash model is obtained by performing clustering guided learning and contrastive learning cyclic training based on key frames of sample videos to be processed.

2. The self-supervised video hashing method according to claim 1, characterized in that The video hash model includes an encoding module and a decoding module; Before performing clustering guided learning and contrastive learning cyclic training based on the sample to-be-processed video, the method further includes: Performing a first frame mask process on the sample video to be processed to obtain a pre-training sample; Inputting the pre-training sample into the encoding module to obtain a predicted video hash code output by the encoding module; Inputting the predicted video hash code into the decoding module to obtain the predicted video output by the decoding module; A first reconstruction loss is determined based on the predicted video and the video to be processed, and parameters of the original video hash model are pre-adjusted based on the first reconstruction loss; the first frame mask processing of the sample video to be processed is repeated to obtain a pre-trained sample process until the original video hash model converges to obtain a pre-trained video hash model.

3. The self-supervised video hashing method according to claim 2, characterized in that After obtaining the pre-trained video hash model, the method further includes: Performing a second frame mask process on the sample video to be processed to obtain a non-overlapping first training sample and a second training sample; inputting the first training sample into the encoding module to obtain a first predicted video hash code output by the encoding module; inputting the second training sample into the encoding module to obtain a second predicted video hash code output by the encoding module; determining a cluster-guided learning loss based on each of the first predicted video hash code and the second predicted video hash code, and initially adjusting parameters of the pre-trained video hash model according to the cluster-guided learning loss; Performing a second frame mask process on the sample video to be processed to obtain a non-overlapping first training sample and a second training sample; inputting the first training sample into the encoding module to obtain a third predicted video hash code output by the encoding module; inputting the second training sample into the encoding module to obtain a fourth predicted video hash code output by the encoding module; Determining a contrastive learning loss based on the corresponding third predicted video hash code and the fourth predicted video hash code, and adjusting parameters of the pre-trained video hash model secondary according to the contrastive learning loss; Repeating the second frame mask processing on the sample video to be processed to obtain non-overlapping first training samples and second training samples, and optimizing the process in a loop until the pre-trained video hash model converges to obtain the video hash model.

4. The self-supervised video hashing method according to claim 3, characterized in that Before performing the second frame mask processing on the sample videos to be processed, the method further includes: performing a first frame mask process on each of the sample videos to be processed to obtain a third training sample and a fourth training sample that do not overlap; inputting the third training sample into the encoding module to obtain a fifth predicted video hash code output by the encoding module; and inputting the fourth training sample into the encoding module to obtain a sixth predicted video hash code output by the encoding module; performing clustering processing on each of the fifth predicted video hash code and the sixth predicted video hash code to obtain a set number of cluster centers; Determining a cluster-guided learning loss based on each of the first predicted video hash code and the second predicted video hash code includes: A cluster-guided learning loss is determined based on the first predicted video hash codes, the second predicted video hash codes, and the hash code cluster centers.

5. The self-supervised video hashing method according to claim 4, characterized in that Determining a cluster-guided learning loss based on each of the first predicted video hash codes, each of the second predicted video hash codes, and the hash code cluster center includes: Determining a clustering loss for each predicted video hash code and each of the hash code cluster centers, determining a second reconstruction loss for each of the predicted video hash codes, and determining a cluster-guided learning loss based on the clustering loss and the second reconstruction loss; The determining of the contrastive learning loss based on the corresponding third predicted video hash code and the fourth predicted video hash code includes: Determine a contrastive loss between the corresponding third predicted video hash code and the fourth predicted video hash code, determine a third reconstruction loss for each of the predicted video hash codes, and determine a contrastive learning loss based on the contrastive loss and the third reconstruction loss.

6. The self-supervised video hashing method according to claim 2, characterized in that Performing a first frame mask process on the sample video to be processed to obtain a pre-training sample, including: A mask ratio corresponding to the sample video to be processed is determined, and a first frame mask processing is performed on the sample video to be processed based on the mask ratio to obtain a pre-training sample; the pre-training sample includes 2-3 key frames.

7. The self-supervised video hashing method according to claim 2, characterized in that The encoding module includes a state-space encoder and a multi-head attention layer; Obtaining the predicted video hash code output by the encoding module, including: Obtaining conversion features of the encoding module performing layer normalization and wide-dimensional conversion processing on the pre-training samples; Input the conversion feature into the state space encoder to obtain the intermediate feature output by the state space encoder; input the intermediate feature into the multi-head attention layer to obtain the encoded feature output by the multi-head attention layer; Pooling operations and binarization processing are performed on the encoding features to obtain a predicted video hash code.

8. A self-supervised video hashing device, characterized in that include: A video determination module, used to determine the key frames of the video to be processed; A video hash module, configured to input the key frames of the video to be processed into a video hash model to obtain a video hash code output by the video hash model; The video hash model is obtained by performing clustering guided learning and contrastive learning cyclic training based on key frames of sample videos to be processed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the self-supervised video hashing method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the self-supervised video hashing method according to any one of claims 1 to 7 is implemented.

Images