Video and text cross-modal hash retrieval method based on prompt embedding

Abstract

The application provides a video and text cross-modal hash retrieval method based on prompt embedding, and relates to the technical field of computers, and comprises the following steps: obtaining to-be-queried modal data, and converting the to-be-queried modal data into to-be-queried hash codes through a corresponding qualified hash learning model; wherein the qualified hash learning model comprises a qualified VIT model and a qualified RoBERTa model, the qualified VIT model is used for hash code conversion of video modal data, and the qualified RoBERTa model is used for hash code conversion of text modal data; the Hamming distance between the to-be-queried hash codes and to-be-matched modal hash codes in a video-text hash code database is calculated in a traversal manner, and the output of cross-modal retrieval is obtained through the Hamming distance. Through the design of an effective cross-modal mapping, the data of different modes is mapped to a common hash space, thereby solving the mode gap problem and improving the accuracy of video and text cross-modal hash retrieval.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a method for cross-modal hash retrieval of video and text based on cue embedding. Background Technology

[0002] In recent years, with the continuous expansion of network data scale, multimodal data such as text, video, images, and audio have grown rapidly, greatly enriching human life. For example, shopping, learning, and entertainment all utilize multiple modalities to comprehensively describe things. However, as the scale and types of data increase, the cost of obtaining valuable information during information retrieval also rises. How to more effectively retrieve valuable information from big data has always been a challenge for industry and academia.

[0003] Hash methods save storage by mapping original features to a binary Hamming space and preserve the similarity information of the original data as much as possible in the Hamming space. Furthermore, during retrieval, the Hamming distance (similarity) between hash codes can be calculated simply by XORing, which greatly speeds up the retrieval process. However, in the process of hash representation, data from different modalities are mapped to a common Hamming space, and information loss is inevitable during this mapping process. Therefore, ensuring recognition accuracy is a key research focus of cross-modal hash retrieval. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, this invention provides a video and text cross-modal hash retrieval method based on cue embedding.

[0005] The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] This invention provides a video and text cross-modal hash retrieval method based on cue embedding, comprising: acquiring query modal data, and converting the query modal data into query hash codes through a corresponding qualified hash learning model; wherein, the qualified hash learning model includes: a qualified VIT model and a qualified RoBERTa model, the qualified VIT model is used for hash code conversion of video modal data, and the qualified RoBERTa model is used for hash code conversion of text modal data;

[0007] The Hamming distance between the hash code to be queried and the modal hash code to be matched in the video-text hash code database is calculated by traversal, and the cross-modal retrieval output is obtained by the Hamming distance.

[0008] Optionally, obtaining the cross-modal retrieval output through the Hamming distance includes:

[0009] If the Hamming distance between the hash code to be queried and the modal hash code to be matched in the video-text hash code database is less than a preset value, then the modal hash codes to be matched that are less than the preset value are output in order of size, forming the output of cross-modal retrieval.

[0010] Optionally, the process of generating the video-text hash code database includes:

[0011] Obtain the dataset;

[0012] The dataset is input into the qualified VIT model and the qualified RoBERTa model to obtain the video-text hash code database.

[0013] Optionally, the generation process of the qualified VIT model and the qualified RoBERTa model includes:

[0014] The parameter information of the pre-trained VIT model and the pre-trained RoBERTa model are updated by using the same loss function to obtain the updated VIT model and the updated RoBERTa model.

[0015] Determine whether the updated VIT model and the updated RoBERTa model meet the corresponding accuracy requirements;

[0016] The updated VIT model that meets the accuracy requirements is taken as the qualified VIT model, and the updated RoBERTa model that meets the corresponding accuracy requirements is taken as the qualified RoBERTa model.

[0017] Optionally, determining whether the updated VIT model and the updated RoBERTa model meet the accuracy requirements includes:

[0018] Determine whether the Hamming distance between the hash codes of the updated VIT model and the updated RoBERTa model for the corresponding modal data in the dataset is less than a preset distance value;

[0019] When the distance is less than the preset distance value, it indicates that the updated VIT model and the updated RoBERTa model meet the accuracy requirements;

[0020] When the distance is greater than the preset distance value, the updated VIT model and the updated RoBERTa model continue to be trained in the direction of decreasing the value of the loss function until a VIT model and a RoBERTa model that meet the conditions are obtained.

[0021] Optionally, the process of generating the loss function includes:

[0022] The global video feature matrix is ​​obtained by using a pre-trained VIT model and video information from the dataset; the text feature matrix is ​​obtained by using a pre-trained RoBERTa model and text information from the dataset.

[0023] Both the text feature matrix and the global video feature matrix are subjected to linear and nonlinear transformations to obtain the deep video feature matrix and the deep text feature matrix.

[0024] A loss function is constructed using the deep video feature matrix, the deep text feature matrix, the global video feature matrix, and the text feature matrix.

[0025] Optionally, obtaining the global video feature matrix using a pre-trained VIT model and video information from the dataset, and obtaining the text feature matrix using a pre-trained RoBERTa model and text information from the dataset, includes:

[0026] The text information from the dataset is input into the pre-trained RoBERTa model to obtain the text feature matrix F. T ;

[0027] Multiple sets of images are extracted frame by frame from the video information in the dataset. These images are then encoded using the pre-trained VIT model to obtain multiple video feature sequences. These multiple video feature sequences are then fused to obtain the global video feature matrix F. V .

[0028] Optionally, constructing the loss function using the deep video feature matrix, the deep text feature matrix, the global video feature matrix, and the text feature matrix includes:

[0029] Through the global video feature matrix F V and the text feature matrix F T Calculate the weighted similarity matrix S u ;

[0030] Through the weighted similarity matrix S u The loss function is constructed from the deep video feature matrix and the deep text feature matrix.

[0031] Optionally, the global video feature matrix F V and the text feature matrix F T Calculate the weighted similarity matrix S u ,include:

[0032] For the global video feature matrix F V and the text feature matrix F TNormalization is performed separately to obtain the normalized global video feature matrix F. V ′ and normalized text feature matrix F T ′;

[0033] For the normalized global video feature matrix F V ′ and the normalized text feature matrix F T The video-text similarity matrix S is obtained by performing cosine similarity calculation. VT and the text-video similarity matrix S TV ;

[0034] S VT =cos(F V ,F T )=F' V (F' T ) T (1)

[0035] S TV =cos(F T ,F V )=F' T (F' V ) T (2)

[0036] Among them, S VT ∈[0,1] n×n S TV ∈[0,1] n×n n represents the dimension of the matrix, and T represents the transpose of the matrix;

[0037] Through the video text similarity matrix S VT and the text-video similarity matrix S TV Obtain the similarity matrix S, and then perform a weighted similarity matrix S' = S''. u ;

[0038]

[0039]

[0040] Among them, s mid and s min Let c and s represent the average and minimum similarity values ​​of each batch S, respectively. The constant c ∈ [0, 10], and s represents the element in the similarity matrix S of each batch.

[0041] Optionally, through the weighted similarity matrix S u The loss function is constructed using the deep video feature matrix and the deep text feature matrix, including:

[0042] In the update phase, an intra-modal similarity preservation term is designed to preserve the loss of both the pre-trained VIT model and the pre-trained RoBERTa model. Represented as,

[0043]

[0044] F represents the Frobenius norm, H V Represents the feature matrix of a depth video;

[0045] Intramodal similarity preservation term for:

[0046]

[0047] H T Represents the deep text feature matrix;

[0048] The video modal similarity preservation term and text modality similarity preservation terms By merging, we obtain the intramodal similarity preservation loss.

[0049]

[0050] In the update phase, an inter-modal similarity preservation term is designed to preserve the losses of the pre-trained VIT model and the pre-trained RoBERTa model. Represented as, Text modality similarity preservation term for,

[0051]

[0052] The video modal similarity preservation term Text modality similarity preservation term By merging the modalities, we obtain the intermodal similarity preservation loss.

[0053]

[0054] The depth video feature matrix H V and deep text feature matrix H T The common features of pre-trained VIT and pre-trained RoBERTa models under unimodal tasks are used to design a consistency loss for unimodal tasks.

[0055] Loss functions of the updated VIT model and the updated RoBERTa model Represented as:

[0056]

[0057] Where λ1, λ2, and λ3 represent the optimization parameters for intramodal similarity preservation loss, intermodal similarity preservation loss, and consistency loss, respectively.

[0058] This invention provides a video and text cross-modal hash retrieval method based on cue embedding, comprising: acquiring query modality data; converting the query modality data into query hash codes using a corresponding qualified hash learning model; wherein the qualified hash learning model includes a qualified VIT model and a qualified RoBERTa model, the qualified VIT model being used for hash code conversion of video modality data, and the qualified RoBERTa model being used for hash code conversion of text modality data; calculating the Hamming distance between the query hash code and the matching modality hash codes in the video-text hash code database by traversal, and obtaining the cross-modal retrieval output through the Hamming distance. By designing an effective cross-modal mapping, data from different modalities are mapped to a common hash space, thereby solving the modality gap problem and improving the accuracy of video and text cross-modal hash retrieval.

[0059] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0060] Figure 1 A flowchart illustrating the video and text cross-modal hash retrieval method based on cue embedding provided in an embodiment of the present invention;

[0061] Figure 2 This is an algorithm framework diagram provided for an embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram of the prompt embedding module provided in an embodiment of the present invention;

[0063] Figure 4 A diagram of a video feature processing module provided in an embodiment of the present invention;

[0064] Figure 5 The diagram shows the structure of the Attn-Hash Net module provided in this embodiment of the invention. Detailed Implementation

[0065] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0066] This invention is applied to the retrieval of cross-modal data, for example: retrieving corresponding videos by inputting text, or retrieving corresponding text by inputting videos.

[0067] To achieve the above technical objectives, embodiments of the present invention provide a video and text cross-modal hash retrieval method based on cue embedding. Figure 1 This is a flowchart illustrating the video and text cross-modal hash retrieval method based on cue embedding provided in an embodiment of the present invention, as shown below. Figure 1 As shown, it includes:

[0068] S101. Obtain the modality data to be queried, and convert the modality data to be queried into a hash code to be queried through a hash learning model that meets the corresponding conditions;

[0069] It should be noted that the eligible hash learning models include: the eligible VIT model and the eligible RoBERTa model. The eligible VIT model is used for hash code conversion of video modal data, and the eligible RoBERTa model is used for hash code conversion of text modal data.

[0070] Specifically, the modal data to be queried can be text or video.

[0071] S102. Calculate the Hamming distance between the hash code to be queried and the modal hash code to be matched in the video-text hash code database by traversal, and obtain the cross-modal retrieval output through the Hamming distance.

[0072] It should be noted that, in this embodiment of the invention, the modality to be matched is the modality data corresponding to the modality data to be queried. When the modality data to be queried is text data, the modality to be matched is the video data corresponding to that text data. For example, when the modality data to be queried is "dog, man, play", the modality to be matched is a video containing "dog, man, play" corresponding to the above text. The length of the video can be adjusted by changing the parameters of the model, and this embodiment of the invention does not limit this. Correspondingly, when the modality data to be queried is a video data, the modality to be matched is the text data corresponding to that video data.

[0073] This invention provides a video and text cross-modal hash retrieval method based on cue embedding, comprising: acquiring query modality data; converting the query modality data into query hash codes using a corresponding qualified hash learning model; wherein the qualified hash learning model includes a qualified VIT model and a qualified RoBERTa model, the qualified VIT model being used for hash code conversion of video modality data, and the qualified RoBERTa model being used for hash code conversion of text modality data; calculating the Hamming distance between the query hash code and the matching modality hash codes in the video-text hash code database by traversal, and obtaining the cross-modal retrieval output through the Hamming distance. By designing an effective cross-modal mapping, data from different modalities are mapped to a common hash space, thereby solving the modality gap problem and improving the accuracy of video and text cross-modal hash retrieval.

[0074] Optionally, S102 includes: when the Hamming distance between the hash code to be queried and the modal hash code to be matched in the video-text hash code database is less than a preset value, the modal hash codes to be matched that are less than the preset value are output in order of size to form the output of cross-modal retrieval.

[0075] In this embodiment of the invention, regardless of whether the query is video or text, it first needs to be converted into the corresponding hash code. This means that the features of the query need to be converted into hash codes using the same hash learning model. If the query is text, a text-to-hash code mapping (i.e., a RoBERTa model that meets the criteria) is used; if the query is video, a video-to-hash code mapping (i.e., a VIT model) is used.

[0076] Once the query hash code is generated, the Hamming distance between the query hash code and the modal hash code to be matched in the video-text hash code database can be calculated. The Hamming distance measures the number of different characters at corresponding positions in two strings of equal length and is suitable for comparing the similarity of hash codes. Specifically, the smaller the Hamming distance, the more similar the two hash codes are.

[0077] The calculated Hamming distances are sorted in ascending order, and the results with the smallest distances (i.e., the most similar) are selected as the output of the cross-modal retrieval. For example... Figure 2 The diagram shown is an algorithm framework diagram provided in an embodiment of the present invention.

[0078] Optionally, the process of generating the video-text hash code database includes:

[0079] Obtain the dataset;

[0080] Input the dataset into a qualified VIT model and a qualified RoBERTa model to obtain a video-text hash code database.

[0081] It should be noted that the dataset in this embodiment of the invention can be information about a company, including text and video information.

[0082] Optionally, the generation process of eligible VIT models and eligible RoBERTa models includes:

[0083] The parameter information of the pre-trained VIT model and the pre-trained RoBERTa model are updated by using the same loss function to obtain the updated VIT model and the updated RoBERTa model.

[0084] Determine whether the updated VIT model and the updated RoBERTa model meet the corresponding accuracy requirements;

[0085] The updated VIT model that meets the accuracy requirements is considered a qualified VIT model, and the updated RoBERTa model that meets the corresponding accuracy requirements is considered a qualified RoBERTa model.

[0086] Optionally, determine whether the updated VIT model and the updated RoBERTa model meet the accuracy requirements, including:

[0087] Determine whether the Hamming distance between the hash codes of the corresponding modal data in the dataset is less than a preset distance value for the updated VIT model and the updated RoBERTa model. If it is less than the preset distance value, it means that the updated VIT model and the updated RoBERTa model meet the accuracy requirements.

[0088] When the distance is greater than the preset value, the updated VIT model and the updated RoBERTa model continue to be trained in the direction of decreasing the value of the loss function until a VIT model and a RoBERTa model that meet the conditions are obtained.

[0089] Optionally, the process of generating the loss function includes:

[0090] The global video feature matrix is ​​obtained by using a pre-trained VIT model and video information from the dataset; the text feature matrix is ​​obtained by using a pre-trained RoBERTa model and text information from the dataset.

[0091] Both the text feature matrix and the global video feature matrix are subjected to linear and nonlinear transformations to obtain the deep video feature matrix and the deep text feature matrix.

[0092] The loss function is constructed using the deep video feature matrix, the deep text feature matrix, the global video feature matrix, and the text feature matrix.

[0093] Optionally, a global video feature matrix is ​​obtained using a pre-trained VIT model and video information from the dataset, and a text feature matrix is ​​obtained using a pre-trained RoBERTa model and text information from the dataset, including:

[0094] The text information from the dataset is input into a pre-trained RoBERTa model to obtain the text feature matrix F. T ;

[0095] Multiple sets of images are extracted frame-by-frame from the video information in the dataset. These images are then encoded using a pre-trained VIT model to obtain multiple video feature sequences. Finally, these multiple video feature sequences are fused to obtain the global video feature matrix F. V .

[0096] In this embodiment of the invention, a prompt embedding module is added to the text feature extraction module during text feature extraction. Figure 3 This is a schematic diagram of a prompt embedding module provided in an embodiment of the present invention. Figure 3 The prompt embedding module shown mainly consists of three parts: the input layer, the prompt embedding layer, and the feature code layer.

[0097] In the process of acquiring video features in this embodiment of the invention, a set of images is first extracted frame by frame from the video segment, denoted as V = (V 1 V 2 , ..., V G ), where G represents the number of images in each group, and then V is encoded by the video encoder module to obtain the video feature sequence. Figure 4 This is a diagram of a video feature processing module provided in an embodiment of the present invention.

[0098] In addition, in order to obtain a global video feature matrix that fully represents the features of the image set extracted from the video, feature fusion operations were also performed on the video sequence.

[0099] It is understood that in the embodiments of the present invention, the feature fusion operation makes the acquired features more comprehensive, resulting in a lower loss function value and ultimately improving the accuracy of cross-modal retrieval.

[0100] Specifically, the Attn-Hash Net module is used to combine the features F output by the pre-trained BLIP model and the pre-trained RoBERTa model. V and F T Perform linear transformations (matrix multiplication) and nonlinear transformations (activation functions) to obtain the depth video feature matrix H. V and deep text feature matrix H T High-level features are extracted. V and H TTogether with the similarity matrix, it is used to generate the loss to guide the training of Attn-Hash Net and is fed into the hash code generation module. Figure 5 The diagram shows the structure of the Attn-HashNet module provided in this embodiment of the invention.

[0101] In this embodiment of the invention, a hash code generation method based on distance judgment is adopted, and the calculation process is as follows:

[0102] B V =f(H V );

[0103] B T =f(H T );

[0104]

[0105]

[0106] Among them, B V B represents the hash code corresponding to the video feature. T Let H represent the hash code corresponding to the text feature, and f(·) be the mapping function. For ease of description, H will be referred to as H in the following text. V and H T It is uniformly denoted as H.

[0107] H m Let H represent the m-th dimension of the latent features learned in the batch, where m = 1, 2, ..., M. For each n-dimensional vector H... m d i Representation vector H m All elements in the i-th row, where i represents vector H m Bank of China's bank number. First, calculate... Find the value of and calculate . The average value is denoted as Then compare each element d i Compared with the average The size of the element if the element is greater than To assign a value to the mapping function, increment it by 1; otherwise, decrement it by 1.

[0108] It should be noted that, in this embodiment of the invention, in order to better extract deep features from the video (modality to be matched) and the embedded cue text (modality data to be queried), they are converted into two-dimensional features and processed using two-dimensional convolution. The feature dimension output from the pre-trained BLIP model is set to 1×512. First, it is converted into 32×16 two-dimensional features. Then, it passes through a two-dimensional convolutional layer with 8 channels (kernel size 3×3, stride 1, padding 1) to convert it into an 8×32×16 two-dimensional feature set. Finally, it is fed into the SE module to obtain the two-dimensional feature weights for each channel and compared with the original two-dimensional feature set. The process involves a two-dimensional max-pooling layer (with a stride of 2) to transform the feature into an 8×16×8 shape. Similarly, a SE module and a two-dimensional convolutional layer with 64 channels are used to obtain a 64×8×4 two-dimensional feature. Finally, the feature is flattened and output according to a pre-set hash code length. In this embodiment of the invention, zero-padding is used for all convolutions.

[0109] Optionally, a loss function is constructed using the deep video feature matrix, the deep text feature matrix, the global video feature matrix, and the text feature matrix, including:

[0110] Through the global video feature matrix F V and the text feature matrix F T Calculate the weighted similarity matrix S u ;

[0111] Through the weighted similarity matrix S u The loss function is constructed from the deep video feature matrix and the deep text feature matrix.

[0112] Optionally, through the global video feature matrix F V and the text feature matrix F T Calculate the weighted similarity matrix S u ,include:

[0113] For the global video feature matrix F V and text feature matrix F T Normalization is performed separately to obtain the normalized global video feature matrix F. V ′ and normalized text feature matrix F T ′;

[0114] For the normalized global video feature matrix F V ′ and normalized text feature matrix F T The video-text similarity matrix S is obtained by performing cosine similarity calculation. VT and the text-video similarity matrix S TV ;

[0115] SVT =cos(F V ,F T )=F' V (F' T ) T (1)

[0116] S TV =cos(F T ,F V )=F' T (F' V ) T (2)

[0117] Among them, S VT ∈[0,1] n×n S TV ∈[0,1] n×n n represents the dimension of the matrix, and T represents the transpose of the matrix;

[0118] Through the video text similarity matrix S VT and the text-video similarity matrix S TV Obtain the similarity matrix S, and then perform a weighted similarity matrix S' = S''. u ;

[0119]

[0120]

[0121] Among them, s mid and s min Let c and s represent the average and minimum similarity values ​​of each batch S, respectively. The constant c ∈ [0, 10], and s represents the element in the similarity matrix S of each batch.

[0122] Optionally, through a weighted similarity matrix S u The loss function is constructed from the deep video feature matrix and the deep text feature matrix, including:

[0123] In the update phase, an intramodal similarity preservation term is designed to preserve the loss of both the pre-trained VIT model and the pre-trained RoBERTa model. Represented as:

[0124]

[0125] F represents the Frobenius norm, H V Represents the feature matrix of a depth video;

[0126] Intramodal similarity preservation term for,

[0127] H T Represents the deep text feature matrix;

[0128] Intramodal similarity preservation term for video and text modality similarity preservation terms By merging, we obtain the intramodal similarity preservation loss.

[0129]

[0130] In the update phase, an inter-modal similarity preservation term is designed to preserve the loss of both the pre-trained VIT model and the pre-trained RoBERTa model. Represented as:

[0131]

[0132] Text modality similarity preservation term for,

[0133]

[0134] Video modal similarity preservation term Text modality similarity preservation term By merging the modalities, we obtain the intermodal similarity preservation loss.

[0135]

[0136] The depth video feature matrix H V and deep text feature matrix H T The common features of pre-trained VIT and pre-trained RoBERTa models under unimodal tasks are used to design a consistency loss for unimodal tasks.

[0137]

[0138] Loss functions of the updated VIT model and the updated RoBERTa model Represented as,

[0139]

[0140] Where λ1, λ2, and λ3 represent the optimization parameters for intramodal similarity preservation loss, intermodal similarity preservation loss, and consistency loss, respectively.

[0141] Understandably, in this embodiment of the invention, by setting λ1, λ2, and λ3 to adjust the optimization parameters of the preservation term, the weights among intra-modal, inter-modal, and consistency losses become adjustable, greatly improving the flexibility of the similarity preservation loss. The loss function can guide network training and update network parameters. Finally, the optimal network parameters are trained, and a video-text hash code database is built based on these parameters, significantly improving the accuracy of cross-modal hash retrieval.

[0142] To verify the accuracy of the video and text cross-modal hash retrieval method based on cue embedding provided in this invention, the MSRVTT dataset was used to test the performance of the method provided in this invention.

[0143] Table 1 shows the MAP values ​​of the method of this invention and the comparative method on the MSRVTT dataset. "T to V" represents the text retrieval video task, and "V to T" represents the video retrieval text task. Optimal data is shown in bold.

[0144] Table 1 - MAP values ​​of the method of the present invention and the comparative method on the MSRVTT dataset.

[0145]

[0146]

[0147] In Table 1, * indicates a non-hashing method without the bit size parameter; - indicates the inventor did not use this method in their experiments. Experimental results show that on the MSRVTT dataset, this invention outperforms the other two hashing methods (DJSRH and JDSH), and its performance significantly improves with increasing bit size. Furthermore, with a 1024-bit length, this invention outperforms non-hashing video-text cross-modal methods (CE and ClipBERT) in R@1, R@5, and R@10 metrics. With a larger hash code length of 2048 bits, this invention outperforms non-hashing video-text retrieval methods in all experimental settings and achieves a 5% performance improvement over the best-performing non-hashing video-text retrieval method, ClipBERT. This demonstrates that with a 2048-bit hash code length, this invention eliminates the drawback of excessive information loss compared to non-hashing methods using binary representation, while using 2048 bits requires hundreds of times less memory to store samples compared to non-hashing methods.

[0148] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0149] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of any of the above-described video and text cross-modal hash retrieval methods based on prompt embedding.

[0150] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0151] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0152] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings and the disclosure, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the description of this invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0153] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A video and text cross-modal hash retrieval method based on prompt-embedded, characterized in that, The method comprises the following steps: acquiring to-be-queried modal data, and converting the to-be-queried modal data into to-be-queried hash codes through a corresponding qualified hash learning model; wherein the qualified hash learning model comprises a qualified VIT model and a qualified RoBERTa model, the qualified VIT model is used for hash code conversion of video modal data, and the qualified RoBERTa model is used for hash code conversion of text modal data; calculating a Hamming distance between the to-be-queried hash codes and to-be-matched modal hash codes in a video-text hash code database in a traversal manner, and obtaining an output of cross-modal retrieval through the Hamming distance; a generation process of the qualified VIT model and the qualified RoBERTa model comprises: updating parameter information of a pre-trained VIT model and a pre-trained RoBERTa model through a same loss function to obtain an updated VIT model and an updated RoBERTa model; judging whether the updated VIT model and the updated RoBERTa model meet corresponding accuracy requirements; taking the updated VIT model that meets the corresponding accuracy requirements as the qualified VIT model, and taking the updated RoBERTa model that meets the corresponding accuracy requirements as the qualified RoBERTa model; a generation process of the loss function comprises: obtaining a global video feature matrix through the pre-trained VIT model and video information in a data set, and obtaining a text feature matrix through the pre-trained RoBERTa model and text information in the data set; performing linear transformation and nonlinear transformation on the text feature matrix and the global video feature matrix to obtain a deep video feature matrix and a deep text feature matrix; constructing a loss function through the deep video feature matrix, the deep text feature matrix, the global video feature matrix and the text feature matrix; obtaining a global video feature matrix through the pre-trained VIT model and video information in a data set, and obtaining a text feature matrix through the pre-trained RoBERTa model and text information in the data set, comprises: inputting text information in the data set into the pre-trained RoBERTa model to obtain the text feature matrix ; The video information in the data set is cut into multiple groups of images frame by frame, the multiple groups of images are encoded by the pre-trained VIT model, multiple video feature sequences are obtained, and the multiple video feature sequences are fused to obtain a global video feature matrix ; the loss function is constructed through the deep video feature matrix, the deep text feature matrix, the global video feature matrix and the text feature matrix, comprises: by said global video feature matrix and said text feature matrix computing a weighted similarity matrix ; by the weighted similarity matrix , the deep video feature matrix and the deep text feature matrix construct a loss function.

2. The prompt-embedding-based video and text cross-modal hash retrieval method according to claim 1, characterized in that, the output of cross-modal retrieval obtained through the Hamming distance comprises: when the Hamming distance between the to-be-queried hash codes and the to-be-matched modal hash codes in the video-text hash code database is less than a preset value, outputting the to-be-matched modal hash codes less than the preset value in a size order to form the output of cross-modal retrieval.

3. The prompt-embedding-based video and text cross-modal hash retrieval method according to claim 1, characterized in that, a generation process of the video-text hash code database comprises: acquiring a data set; inputting the data set into the qualified VIT model and the qualified RoBERTa model to obtain a video-text hash code database.

4. The prompt-embedding-based video and text cross-modal hash retrieval method according to claim 1, characterized in that, the judgment of whether the updated VIT model and the updated RoBERTa model meet the accuracy requirements comprises: determining whether the hamming distance between the hash codes of the updated VIT model and the updated RoBERTa model for the corresponding modal data in the data set is less than a preset distance value; when less than the preset distance value, it indicates that the updated VIT model and the updated RoBERTa model meet the accuracy requirement; when greater than the preset distance value, the updated VIT model and the updated RoBERTa model are continuously trained in the direction of decreasing the value of the loss function until the qualified VIT model and the qualified RoBERTa model are obtained.

5. The prompt-embedding-based video and text cross-modal hash retrieval method according to claim 1, characterized in that, said global video feature matrix and said text feature matrix computing a weighted similarity matrix comprising: For the global video feature matrix and the text feature matrix Normalization is performed separately to obtain the normalized global video feature matrix. and normalized text feature matrix ; a normalized global video feature matrix and the normalized text feature matrix , and a cosine similarity calculation is performed to obtain a video-text similarity matrix and a text-video similarity matrix ; ；(1) ；(2) wherein , , denotes the dimension of a matrix, denotes transposing a matrix; by the video text similarity matrix and the text video similarity matrix a similarity matrix and weighting the similarity matrix to obtain a weighted similarity matrix ; ；(3) ；(4) wherein, and respectively denote the average and minimum value of the similarity per batch, the constant , denotes an element in the similarity matrix per batch.

6. The prompt-embedding-based video and text cross-modal hash retrieval method according to claim 1, characterized in that, by the weighted similarity matrix , the deep video feature matrix and the deep text feature matrix to construct a loss function, comprising: In the update phase, a similarity preservation term in the design modalities is used to maintain the loss of the pre-trained VIT model and the pre-trained RoBERTa model, and a similarity preservation term in the video modalities is represented as, ; (5) denotes the Frobenius norm, denotes the depth video feature matrix; Intra-textual similarity preserving term is: ；(6) denotes the deep text feature matrix; the video modality intra-similarity keeping term and the text modality intra-similarity keeping term are combined to obtain the modality intra-similarity keeping loss , ；(7) In the update phase, the inter-modal similarity preservation term is designed to preserve the loss of the pre-trained VIT model and the pre-trained RoBERTa model, and the inter-video modal similarity preservation term is represented as, ; (8) Inter-textual similarity preserving term For, ； (9) the video inter-modality similarity preservation term and the text inter-modality similarity preservation term are combined to obtain an inter-modality similarity preservation loss , ；(10) The depth video feature matrix And the depth text feature matrix As the common features of the pre-trained VIT model and the pre-trained RoBERTa model under the single-modal task, and a consistency loss under the single-modal task is designed , ; Loss function of the updated VIT model and the updated RoBERTa model is represented as: ； wherein respectively denote the optimization parameters for the intra-modality similarity preservation loss, the optimization parameters for the inter-modality similarity preservation loss, and the optimization parameters for the consistency loss.

Images