Method, system and device for text retrieval video based on instruction-guided GPT

Abstract

The present invention discloses a text retrieval video method, system and device based on instruction-guided GPT, which relates to the field of retrieval technology, including the steps of: obtaining multiple videos and their corresponding multiple descriptive texts; inputting the descriptive text into a cross-modal retrieval model to expand the text, and generating corresponding subtitles for the video; encoding the subtitles, extended text and descriptive text, obtaining the similarity between the encoded descriptive text encoding features and the subtitle and extended text encodings, obtaining enhanced text according to the similarity results, training the enhanced text and pseudo-paired video in a cross-modal retrieval model, obtaining the final cross-modal retrieval model, and performing retrieval. The present invention realizes associating real scenes with descriptive texts, thereby expanding entity information in texts, improving the correlation between video texts, prompting the model to understand common sense information, reducing the impact of enhanced text and video mismatch on retrieval performance, and improving retrieval effects.

Description

Technical Field

[0001] The present invention relates to the field of retrieval technology, and in particular to a text retrieval video method, system and device based on instruction-guided GPT. Background Art

[0002] With the emergence and popularity of video media software such as YouTube and Tik Tok, more and more users record and share their daily lives through videos. Therefore, a huge amount of video, text and other data is generated and spread on the Internet every day. Such complex data has led to traditional single-modal retrieval limiting users' demand for comprehensive and flexible information acquisition, and also increased the cost of data management for enterprises. In order to better solve these problems, text-video cross-modal retrieval has begun to attract widespread attention from researchers. As an important research direction in the field of multimodal vision and semantic understanding, it aims to analyze descriptive text to find the best matching video from the corpus.

[0003] Early text-video cross-modal retrieval mainly adopted a concept-based approach, which extracts concepts from videos and aligns them with text words, and determines the retrieval results by calculating the similarity between text and video concepts. However, this approach has an obvious disadvantage: it is difficult to describe the rich sequence information in videos and texts with only a few selected concepts.

[0004] Existing text-video cross-modal retrieval methods have achieved good retrieval performance. However, these methods all assume that there is a strong semantic relationship between the text description and the video, but ignore the potential weak semantic text description. Specifically, both the current text-video retrieval dataset and real-world scenarios contain a large amount of weak semantic text, that is, the text description contains common sense information or focuses on describing a certain segment of the video, lacking information about the entities contained in the video. The model cannot understand the common sense information, resulting in a decrease in retrieval performance and mismatched retrieval results. Summary of the invention

[0005] The purpose of the present invention is to provide a text retrieval video method, system and device based on instruction-guided GPT to address the deficiencies of the above-mentioned prior art, so as to solve the problem that the prior art has a large amount of weak semantic text, that is, the text description contains common sense information or focuses on describing a certain segment of the video, lacks information about the entities contained in the video, and the model cannot understand the common sense information, resulting in reduced retrieval performance and mismatch of retrieval results.

[0006] The present invention specifically provides the following technical solution: a text retrieval video method based on instruction-guided GPT, comprising the following steps:

[0007] Get multiple videos and their corresponding descriptive texts;

[0008] Input the descriptive text into a cross-modal retrieval model, guide the deep learning model GPT to expand the descriptive text through instructions, obtain extended text, and generate corresponding subtitles for the video;

[0009] The extended text is concatenated with the corresponding descriptive text and then encoded to obtain descriptive text encoding features, and the subtitles and the video are encoded to obtain subtitle encoding features and video encoding features;

[0010] Obtaining the similarity between the descriptive text encoding feature and the video encoding feature, and obtaining the similarity between the descriptive text encoding feature and the subtitle encoding feature;

[0011] When the similarity between the descriptive text encoding feature and the non-matching subtitle is higher than that between the matching subtitle, the non-matching subtitle is used as a pseudo matching subtitle;

[0012] The descriptive text encoding features and the pseudo-paired captions are used as the enhanced text, and the cross-modal retrieval model is trained by the enhanced text and the pseudo-paired video to obtain the final cross-modal retrieval model.

[0013] The data to be retrieved is input into the final cross-modal retrieval model, and the retrieval results are obtained by sorting the cosine similarity of the output feature vectors.

[0014] Preferably, the method of guiding the deep learning model GPT to expand the descriptive text through instructions to obtain the extended text and generate corresponding subtitles for the video includes the following steps:

[0015] Enter a manual setting instruction and use it to guide GPT to expand the input text; the specific expression is:

[0016] E i ~G(·|T i ,I).

[0017] Among them, G(·) is the GPT model, T i is the i-th descriptive text, I is the manual input instruction, E i is the extended text of the i-th descriptive text after being expanded by the GPT model;

[0018] Use the ZeroCap model to generate subtitles for the video and align the generated subtitles with the extended text; the specific expression is:

[0019] C i ~Z(·|V i ).

[0020] Among them, Z(·) represents the subtitle generation model of the video, V i represents the i-th video, C iIndicates the subtitles generated for the i-th video.

[0021] Preferably, the extended text is concatenated with the corresponding descriptive text and then encoded to obtain descriptive text encoding features, and the subtitles and videos are encoded to obtain subtitle encoding features and video encoding features, including the following steps:

[0022] The extended text and the corresponding descriptive text are concatenated and sent to a text encoder for encoding, and the average value of the encoded text features is obtained, and the average value of the encoded text features is used as the descriptive text encoding feature; the specific expression is:

[0023]

[0024] in, is the i-th descriptive text encoding feature, T i is the i-th descriptive text, T i j is the jth extended text corresponding to the ith descriptive text, T(·) is the CLIP text encoder, θ is the hyperparameter corresponding to the CLIP text encoder, N is the number of extended texts, For splicing;

[0025] The subtitles are input into a text editor for encoding, an average value of the encoded subtitle features is obtained, and the average value of the encoded subtitle features is used as the subtitle encoding; the specific expression is:

[0026]

[0027] in, is the encoding feature of the i-th video subtitle, Generate the jth subtitle for the i-th video, T(·) is the CLIP text encoder, θ is its corresponding hyperparameter, and the descriptive text encoding and video subtitle encoding share the same set of hyperparameters;

[0028] The visual encoder is used to encode the video, and the first data structure C1S_Token is taken from the final output layer to represent the video encoding features; the specific expression is:

[0029]

[0030] in, is the i-th video coding feature, F(·) is the CLIP video encoder, is its corresponding hyperparameter, V i is the i-th video.

[0031] Preferably, the obtaining of the similarity between the descriptive text coding feature and the video coding feature, and the obtaining of the similarity between the descriptive text coding feature and the subtitle coding feature, comprises the following steps:

[0032] Cosine similarity is used to measure the similarity between video coding features and descriptive text coding features, as well as the similarity between subtitle coding features and descriptive text coding features; the specific expression is:

[0033]

[0034] in, and denote the i-th video encoding feature, the i-th descriptive text encoding feature, and the subtitle encoding feature corresponding to the i-th video, respectively. vt (·) represents the video and text similarity calculation function, S ct (·) represents the similarity calculation function between video subtitles and text.

[0035] Preferably, the cross-modal retrieval model is trained by enhancing the text and the pseudo-paired video to obtain the final cross-modal retrieval model, specifically:

[0036] N i ~G(·|E i ,C i ).

[0037]

[0038] Among them, N i represents the common description text between the descriptive text encoding features and pseudo-paired subtitles, G(·) is the GPT model, and E i is the text of the i-th descriptive text expanded by the GPT model, C i represents the subtitles generated for the i-th video. represents the i-th encoded feature, T(·) is the CLIP text encoder, θ is the hyperparameter corresponding to the CLIP text encoder, and M i For subtitles, descriptive text or video that needs to be encoded, S vt (·) represents the video and text similarity calculation function, τ represents the temperature hyperparameter, B represents the batch input video-text pair size, is the jth encoding feature, is the jth video encoding feature, is the i-th video coding feature, L N2V represents the enhanced text-to-video retrieval loss, L V2N represents the video to enhanced text retrieval loss, L NV represents the total loss of enhanced text-video retrieval;

[0039] The cross-modal retrieval model is optimized through the total loss to obtain the final cross-modal retrieval model.

[0040] Preferably, the step of inputting the data to be retrieved into the final cross-modal retrieval model and obtaining the retrieval results by sorting the cosine similarity of the output feature vectors comprises the following steps:

[0041] The data to be retrieved is input into the final cross-modal retrieval model, and three feature vectors e are output. t , and

[0042] Among them, e t Represents the descriptive text encoding obtained after GPT expansion and averaging operations.

[0043] Indicates the encoding features of the 1st, 2nd to nth videos, Indicates the encoding features of the first, second to nth subtitles;

[0044] Use the cosine similarity formula to calculate e t and The similarity and e t and The similarity between the two videos is calculated and the similarity results are sorted to retrieve the videos most relevant to the descriptive text.

[0045] The present invention provides a text retrieval video method system based on instruction-guided GPT, comprising:

[0046] A data acquisition module, used to acquire multiple videos and their corresponding multiple descriptive texts;

[0047] A data preprocessing module, used for inputting the descriptive text into a cross-modal retrieval model, guiding the deep learning model GPT to expand the descriptive text through instructions, obtaining extended text, and generating corresponding subtitles for the video;

[0048] An encoding module, used for concatenating the extended text with the corresponding descriptive text and encoding them to obtain descriptive text encoding features, and encoding the subtitles and the video to obtain subtitle encoding features and video encoding features;

[0049] A similarity acquisition module, used to acquire the similarity between the descriptive text coding features and the video coding features, and to acquire the similarity between the descriptive text coding features and the subtitle coding features;

[0050] an optimization module, configured to use the non-matching subtitles as pseudo-matching subtitles when the similarity between the descriptive text encoding feature and the non-matching subtitles is higher than that between the matching subtitles;

[0051] A training module is used to use the descriptive text encoding features and the common description of the pseudo-paired subtitles as enhanced text, and train the cross-modal retrieval model through the enhanced text and the pseudo-paired video to obtain the final cross-modal retrieval model;

[0052] The output module is used to input the data to be retrieved into the final cross-modal retrieval model and obtain the retrieval results by sorting the cosine similarity of the output feature vectors.

[0053] The present invention provides a computer device, including a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of a text retrieval video method based on instruction-guided GPT.

[0054] Compared with the prior art, the present invention has the following significant advantages:

[0055] The present invention uses GPT to generate corresponding extended text for each descriptive text, and generates corresponding subtitles for the video, and uses the video subtitles as a supplement to the video content, and performs text similarity measurement with the extended text, thereby obtaining pseudo-paired subtitles, and uses the common description of the descriptive text encoding features and the pseudo-paired subtitles as enhanced text, and integrates the enhanced text with the pseudo-paired video as training data, thereby improving the quality of the training data. The present invention realizes the association of real scenes with descriptive texts, thereby expanding the entity information in the text, improving the correlation between video texts, prompting the model to understand common sense information, reducing the impact of the mismatch between enhanced text and video on retrieval performance, and improving the retrieval effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a structural schematic diagram of the text retrieval video method based on instruction-guided GPT of the present invention;

[0057] Figure 2 It is a schematic diagram of the training process of the text retrieval video method based on instruction-guided GPT of the present invention;

[0058] Figure 3 It is a schematic diagram of the retrieval process of the text retrieval video method based on instruction-guided GPT of the present invention;

[0059] Figure 4 It is a processing diagram of expanding text in the text retrieval video method based on instruction-guided GPT of the present invention. DETAILED DESCRIPTION

[0060] The following is a clear and complete description of the technical solutions of the embodiments of the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0061] The overall structure of the model of the present invention is as follows Figure 1 As shown in the figure, first, GPT is used to generate corresponding enhanced text for each descriptive text, with the aim of associating the real scene with the descriptive text, thereby expanding the entity information in the text and improving the relevance between the video text; secondly, combined with the idea of ​​Cap4Video, the video extension model of ZeroCap is used to generate corresponding subtitle descriptions for the video, with the aim of using the video subtitles as a supplement to the video content and performing unimodal similarity measurement between texts with the enhanced text; finally, in the iterative self-training process, GPT is used to integrate the overlapping information of the subtitles generated by the enhanced text with the pseudo-paired video to generate new training instances, thereby improving the quality of the training data and reducing the impact of the mismatch between the enhanced text and the video on the decision boundary.

[0062] like Figure 2 and Figure 3 As shown, the present invention proposes a text retrieval video method based on instruction-guided GPT, which specifically includes the following steps:

[0063] Step S1: Obtain multiple videos and their corresponding multiple descriptive texts.

[0064] That is, obtain the data set, select and divide the training set required for the text-video cross-modal retrieval task.

[0065] In this embodiment, common text retrieval video datasets MSR-VTT, DiDeMo and LSM DC are used for training and testing. The MSR-VTT dataset contains more than 10,000 video clips of 10 to 30 seconds, each of which has approximately 20 natural language descriptions. There are 180,000 training data pairs and 1,000 test data pairs. The DiDeMo dataset consists of 10,464 video clips, each of which has 3 to 5 text descriptions. All sentence descriptions of each video are concatenated as the final description of the entire video. There are 8,395 training data pairs and 1,004 test data pairs. The LSMDC dataset contains 128,000 movie clips, each of which is between 2s and 30s long and is described by text titles or subtitles extracted from the movie, and contains 118,081 video-text pairs. There are 109,673 training data pairs and 1,000 test data pairs.

[0066] Define the training set as M = {{V1, T1}, {V2, T2}, ..., {V i ,T i},...}, where V i represents the i-th video, T i Represents the corresponding descriptive text.

[0067] Step S2: Input the descriptive text into the cross-modal retrieval model, guide the deep learning model GPT through instructions to expand the descriptive text, obtain the extended text, and generate corresponding subtitles for the video.

[0068] S2 specifically includes: guiding the deep learning model GPT to expand the descriptive text through instructions, obtaining the expanded text, and generating corresponding subtitles for the video. It includes the following steps:

[0069] Step S21: Descriptive text expansion: When the input descriptive text is short or contains common sense phrases, the existing text video retrieval model may not be able to infer the entity information contained in the video from the text description, thereby reducing the model retrieval performance. Since the performance of large-scale GPT-based models in natural language processing tasks has been significantly improved, can the GPT model be used to expand the original input text in combination with real-world scenarios so that the expanded text description contains information about the entities in the video?

[0070] Specifically, a manually set instruction I is input: "Please expand on the above text with real-reality scenarios", and the instruction is used to guide the GPT model (such as Alpaca7B) to expand the input text T. The descriptive text expansion is publicized as follows:

[0071] E i ~G(·|T i ,I).

[0072] Where: G(·) is the GPT model, T i is the i-th descriptive text, I is the manual input instruction, E i is the extended text of the i-th descriptive text after being expanded by the GPT model.

[0073] Step S22: Video zero-shot subtitle generation: Generate subtitles for the video using the ZeroCap model and align the generated subtitles with the extended text.

[0074] The Cap4Video paper has demonstrated that using the ZeroCap model to generate zero-shot subtitles for videos, where the generated subtitles are used as feature representations of the video and aligned with descriptive text to aid text video retrieval, can significantly improve model performance. On this basis, the present invention also adopts a similar idea to generate subtitles for videos, and aligns the generated subtitles with the expanded text to assist the model in retrieval. The specific calculation process for generating subtitles is as follows:

[0075] C i ~Z(·|V i ).

[0076] Where: Z(·) represents the subtitle generation model of the video, V i represents the i-th video, C i Indicates the subtitles generated for the i-th video.

[0077] Step S3: concatenate the extended text with the corresponding descriptive text and encode them to obtain the descriptive text encoding features, and encode the subtitles and videos to obtain subtitle encoding features and video encoding features, and encode the two modal data into the same dimension.

[0078] S3 is specifically as follows: inputting the original data and the data processed by step S2 into the CLIP4Clip model, and loading the pre-trained weight parameters.

[0079] The CLIP text encoder is a Transformer structure consisting of 12 layers and 8 attention heads, with query, key and value feature dimensions of 512 dimensions; the CLIP video encoder (ViT-B / 16) adopts the VisionTransformer model architecture, with a patch size of 16*16, a Transformer module consisting of 12 layers and 12 attention heads, and an output feature dimension of 512 dimensions.

[0080] Step S31: Descriptive text encoding: The extended text expanded by the GPT model is concatenated with the corresponding descriptive text and sent to the text encoder T(·) for encoding. The average value of the encoded text features is calculated, and the average value of the encoded text features is used as the descriptive text encoding feature to obtain the final text enhancement feature embedding. The calculation formula is as follows:

[0081]

[0082] Where: is the i-th descriptive text encoding feature, T i is the i-th descriptive text, T i jis the jth extended text corresponding to the ith descriptive text, T(·) is the CLIP text encoder, θ is its corresponding hyperparameter, N is the number of extended texts, For splicing.

[0083] Step S32: Video subtitle encoding: Input the subtitles generated by the ZeroCap model into the text encoder T(·) for encoding, obtain the average value of the subtitle features after encoding, use the average value of the subtitle features after encoding as the subtitle encoding, and obtain the final subtitle feature embedding. The calculation formula is as follows:

[0084]

[0085] Where: is the encoding feature of the i-th video subtitle, Generate the jth subtitle for the i-th video. T(·) is the CLIP text encoder and θ is its corresponding hyperparameter. The descriptive text encoding and video subtitle encoding share the same set of hyperparameters.

[0086] Step S33: Video encoding: The visual encoder uses VisionTransformer, whose pre-trained weights come from 400 million high-quality image-text pairs in CLIP. In the final output layer, the first data structure C1S_Token is taken to represent the video encoding feature. The calculation formula is as follows:

[0087]

[0088] Where: is the i-th video coding feature, F(·) is the CLIP video encoder, is its corresponding hyperparameter, V i is the i-th video.

[0089] Step S4: obtaining the similarity between the descriptive text coding features and the video coding features, and obtaining the similarity between the descriptive text coding features and the subtitle coding features.

[0090] Specifically: similarity calculation is performed on cross-modal features, including the similarity between the expanded descriptive text and the video to be matched; the similarity between the expanded descriptive text and the subtitles of the video to be matched.

[0091] Mapping the high-dimensional semantic features of data from different modalities to the same common space and measuring the similarity of different modalities can capture the relevant information between different modalities and further eliminate the heterogeneity problem between data to optimize the alignment process.

[0092] The present invention uses cosine similarity, which is good at processing high-dimensional data, to measure the similarity between video coding features and descriptive text coding features, as well as the similarity between subtitle coding features and descriptive text coding features. The specific calculation process is as follows:

[0093]

[0094]

[0095] Where: and They represent the i-th video coding feature, the i-th descriptive text coding feature, and the subtitle coding feature corresponding to the i-th video. vt (·) represents the video and text similarity calculation function, S ct (·) represents the similarity calculation function between subtitles and text.

[0096] Step S5: When the similarity between the descriptive text encoding feature and the non-matching subtitle is higher than that between the matching subtitle, the non-matching subtitle is used as a pseudo matching subtitle.

[0097] Step S6: The descriptive text encoding features and the common description of the pseudo-paired subtitles are used as the enhanced text, and the cross-modal retrieval model is trained through the enhanced text and the pseudo-paired video to obtain the final cross-modal retrieval model.

[0098] S5 and S6 are specifically: model and dataset optimization.

[0099] When using command-guided GPT to expand input text, there is a problem that needs to be explored: how to avoid the correlation between the expanded text and the unmatched videos being higher than the correlation between the matched videos. When the descriptive text is short, it is easy to cause one text description to correspond to multiple videos. At this time, the expanded text description may be more biased towards the unmatched video. When the text and video are not matched correctly, the boundary line between the videos may be blurred, causing unnecessary deviations in the model. To solve these problems, the GPT model is used to extract the common description between the expanded descriptive text and the pseudo-paired video subtitles as enhanced text, and the enhanced text and pseudo-paired videos are used as new training instances for model training.

[0100] The specific expression of the training process is:

[0101] N i ~G(·|E i ,C i ).

[0102]

[0103]

[0104] Where: N i represents the common description text between the descriptive text encoding features and pseudo-paired subtitles, G(·) is the GPT model, and E i is the text of the i-th descriptive text expanded by the GPT model, C i represents the subtitles generated for the i-th video. represents the i-th encoded feature, T(·) is the CLIP text encoder, θ is the hyperparameter corresponding to the CLIP text encoder, and M i For subtitles, descriptive text or video that needs to be encoded, S vt (·) represents the video and text similarity calculation function, τ represents the temperature hyperparameter, B represents the batch input video-text pair size, is the jth encoding feature, is the jth video encoding feature, is the i-th video coding feature, L N2V represents the enhanced text-to-video retrieval loss, L V2N represents the video to enhanced text retrieval loss, L NV represents the total loss for enhanced text-video retrieval.

[0105] The cross-modal retrieval model is optimized through the total loss to obtain the final cross-modal retrieval model.

[0106] The specific process of optimizing the cross-modal retrieval model is as follows:

[0107] According to the results of the two similarities, contrastive learning is used between the extended text and the video to be matched, as well as between the extended text and the subtitles to be matched, to obtain the loss between the extended text and the video to be matched, and the loss between the extended text and the subtitles to be matched. The two losses are combined to optimize the cross-modal retrieval model.

[0108] Specifically: the network is optimized through the loss function, which mainly includes the loss between the descriptive text and the video modality, and the loss within the descriptive text and the video subtitle modality.

[0109] Text-Video Inter-Modality Loss: Use a symmetric cross-entropy loss to maximize the similarity between matching query-video pairs and minimize the similarity between other pairs:

[0110]

[0111] Where: S vt (·) represents the video and text similarity calculation function, and τ represents the temperature hyperparameter.

[0112] Text-video subtitle intra-modal loss: The subtitles generated by the video are used as feature representations of the video. For matching video-text pairs, the present invention hopes to narrow the distance between the subtitles generated by the video and the extended text, and for mismatched video-text pairs, it hopes to alienate the distance between the two. Therefore, a symmetric cross entropy loss is used to maximize the similarity between matching query-title pairs and minimize the similarity between other pairs:

[0113]

[0114] Where: S ct (·) represents the similarity calculation function between video subtitles and text, and τ represents the temperature hyperparameter.

[0115] Step S7: input the data to be retrieved into the final cross-modal retrieval model, and obtain the retrieval results by sorting the cosine similarity of the output feature vectors.

[0116] Specifically:

[0117] The user's descriptive text and the video to be paired are obtained as the data to be retrieved.

[0118] In this embodiment, the descriptive text input by the user is defined as t, and the video set to be queried is defined as V = {v1, v2, ..., v n}.

[0119] The data to be retrieved is input into the trained cross-modal retrieval model to obtain its feature vector.

[0120] The text-video cross-modal retrieval model outputs three feature vectors e t , and Among them, e t Represents the descriptive text encoding obtained after GPT expansion and averaging operations. Indicates the encoding features of the 1st, 2nd to nth videos, Indicates the encoding features of the 1st, 2nd to nth subtitles.

[0121] Use the cosine similarity formula to calculate e t and The similarity and e t and The similarity between the two videos is calculated and the similarity results are sorted to retrieve the videos most relevant to the descriptive text.

[0122] The present invention uses the standard recall rate (Recall@rank-k, R@k) as an evaluation index, which is widely used in the field of text and video retrieval. R@k represents the proportion of results related to the query sample in the top K results of the query return similarity ranking to all returned results. The higher the value, the better the retrieval effect.

[0123] The present invention conducts a large number of experiments on three video benchmarks, and the R@1 index reaches 52.0% in the MSR-VTT dataset, which is 5.6% higher than the baseline model Clip4Clip and 0.8% higher than the current advanced model Cap4Video. The R@1 index reaches 52.4% in the MSVD dataset and 52.8% in the DiDeMo dataset, which are 0.6% and 0.8% higher than the current advanced model Cap4Video, respectively.

[0124] like Figure 4 As shown in (a), the user inputs the descriptive text "a birthday party is going on", where "birthday party" is a common sense information. The present invention can combine the common sense concept to infer that the video should have the following information: "cake", "ribbons", "balloons", etc. However, the current text video retrieval model cannot reason about descriptive text like humans, resulting in the inability to match videos suitable for weak semantic text.

[0125] The present invention uses instructions to guide GPT to expand the descriptive text, and the result is as follows Figure 4 As shown in (b), the text description after GPT expansion can be highly matched with the video in the video, which can enhance the overall retrieval effect of the model and is suitable for applications in real scenarios.

[0126] Based on the above method, the present invention provides a text retrieval video method system based on instruction-guided GPT, including: a data acquisition module, a data preprocessing module, an encoding module, a similarity acquisition module, an optimization module, a training module and an output module.

[0127] Among them, the data acquisition module is used to obtain multiple videos and their corresponding multiple descriptive texts; the data preprocessing module is used to input the descriptive text into the cross-modal retrieval model, guide the deep learning model GPT to expand the descriptive text through instructions, obtain the extended text, and generate corresponding subtitles for the video; the encoding module is used to concatenate the extended text with the corresponding descriptive text and encode it to obtain the descriptive text encoding features, and encode the subtitles and videos to obtain the subtitle encoding features and the video encoding features; the similarity acquisition module is used to obtain the similarity between the descriptive text encoding features and the video encoding features, and obtain the similarity between the descriptive text encoding features and the subtitle encoding features; the optimization module is used to use the non-matching subtitles as pseudo-paired subtitles when the similarity between the descriptive text encoding features and the non-matching subtitles is higher than that between the matching subtitles; the training module is used to use the common description of the descriptive text encoding features and the pseudo-paired subtitles as the enhanced text, and train the cross-modal retrieval model through the enhanced text and the pseudo-paired video to obtain the final cross-modal retrieval model; the output module is used to input the data to be retrieved into the final cross-modal retrieval model, and obtain the retrieval results by sorting the cosine similarity of the output feature vectors.

[0128] The present invention also provides a computer device, including a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of a text retrieval video method based on instruction-guided GPT.

[0129] In accordance with the disclosed embodiments, a computing device may communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth communications, etc.), or with any device (e.g., routers, modems, etc.) that enables a computing device to communicate with one or more other computing devices.

[0130] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of a text retrieval video method based on instruction-guided GPT are implemented.

[0131] According to the disclosed embodiments, the storage medium may be a non-volatile computer-readable storage medium, such as but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0132] The above content is a further detailed description of the present invention in combination with a specific preferred embodiment. For technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A text retrieval video method based on instruction-guided GPT, characterized in that: include: Get multiple videos and their corresponding descriptive texts; Input the descriptive text into a cross-modal retrieval model, guide the deep learning model GPT to expand the descriptive text through instructions, obtain extended text, and generate corresponding subtitles for the video; The extended text is concatenated with the corresponding descriptive text and then encoded to obtain descriptive text encoding features, and the subtitles and the video are encoded to obtain subtitle encoding features and video encoding features; Obtaining the similarity between the descriptive text encoding feature and the video encoding feature, and obtaining the similarity between the descriptive text encoding feature and the subtitle encoding feature; When the similarity between the descriptive text encoding feature and the non-matching subtitle is higher than that between the matching subtitle, the non-matching subtitle is used as a pseudo matching subtitle; The descriptive text encoding features and the pseudo-paired captions are used as the enhanced text, and the cross-modal retrieval model is trained by the enhanced text and the pseudo-paired video to obtain the final cross-modal retrieval model. The data to be retrieved is input into the final cross-modal retrieval model, and the retrieval results are obtained by sorting the cosine similarity of the output feature vectors; The method of guiding the deep learning model GPT to expand the descriptive text through instructions to obtain the expanded text and generate corresponding subtitles for the video includes the following steps: Enter a manual setting instruction and use it to guide GPT to expand the input text; the specific expression is: E i ~G(·|T i ,I); Among them, G(·) is the GPT model, T i is the i-th descriptive text, I is the manual input instruction, E i is the extended text of the i-th descriptive text after being expanded by the GPT model; Use the ZeroCap model to generate subtitles for the video and align the generated subtitles with the extended text; the specific expression is: C i ~Z(·|V i ); Among them, Z(·) represents the subtitle generation model of the video, V i represents the i-th video, C i Indicates the subtitles generated for the i-th video; The extended text is concatenated with the corresponding descriptive text and then encoded to obtain descriptive text encoding features, and the subtitles and the video are encoded to obtain subtitle encoding features and video encoding features, including the following steps: The extended text and the corresponding descriptive text are concatenated and sent to a text encoder for encoding, and the average value of the encoded text features is obtained, and the average value of the encoded text features is used as the descriptive text encoding feature; the specific expression is: in, is the i-th descriptive text encoding feature, T i is the i-th descriptive text, T i j is the jth extended text corresponding to the ith descriptive text, T(·) is the CLIP text encoder, θ is the hyperparameter corresponding to the CLIP text encoder, N is the number of extended texts, For splicing; The subtitles are input into a text editor for encoding, an average value of the encoded subtitle features is obtained, and the average value of the encoded subtitle features is used as the subtitle encoding; the specific expression is: in, is the encoding feature of the i-th video subtitle, Generate the jth subtitle for the i-th video, T(·) is the CLIP text encoder, θ is its corresponding hyperparameter, and the descriptive text encoding and video subtitle encoding share the same set of hyperparameters; The visual encoder is used to encode the video, and the first data structure C1S_Token is taken from the final output layer to represent the video encoding features; the specific expression is: in, is the i-th video coding feature, F(·) is the CLIP video encoder, is its corresponding hyperparameter, V i is the i-th video.

2. A text retrieval video method based on instruction-guided GPT as claimed in claim 1, characterized in that: The obtaining of the similarity between the descriptive text coding feature and the video coding feature, and the obtaining of the similarity between the descriptive text coding feature and the subtitle coding feature, comprises the following steps: Cosine similarity is used to measure the similarity between video coding features and descriptive text coding features, as well as the similarity between subtitle coding features and descriptive text coding features; the specific expression is: in, and denote the i-th video encoding feature, the i-th descriptive text encoding feature, and the subtitle encoding feature corresponding to the i-th video, respectively. vt (·) represents the video and text similarity calculation function, S ct (·) represents the similarity calculation function between subtitles and text.

3. A text retrieval video method based on instruction-guided GPT as claimed in claim 1, characterized in that: The cross-modal retrieval model is trained by enhancing text and pseudo-paired video to obtain the final cross-modal retrieval model, which is specifically: The specific expression of the training process is: N i ~G(·|E i ,C i ); Among them, N i represents the common description text between the descriptive text encoding features and pseudo-paired subtitles, G(·) is the GPT model, and E i is the text of the i-th descriptive text expanded by the GPT model, C i represents the subtitles generated for the i-th video. represents the i-th encoded feature, T(·) is the CLIP text encoder, θ is the hyperparameter corresponding to the CLIP text encoder, and M i For subtitles, descriptive text or video that needs to be encoded, S vt (·) represents the video and text similarity calculation function, τ represents the temperature hyperparameter, B represents the batch input video-text pair size, is the jth encoding feature, is the jth video encoding feature, is the i-th video coding feature, L N2V represents the enhanced text-to-video retrieval loss, L V2N represents the video to enhanced text retrieval loss, L NV represents the total loss of enhanced text-video retrieval; The cross-modal retrieval model is optimized through the total loss to obtain the final cross-modal retrieval model.

4. A text retrieval video method based on instruction-guided GPT as claimed in claim 1, characterized in that: The method of inputting the data to be retrieved into the final cross-modal retrieval model and obtaining the retrieval results by sorting the cosine similarity of the output feature vectors comprises the following steps: The data to be retrieved is input into the final cross-modal retrieval model, and three feature vectors e are output. t , and Among them, e t Represents the descriptive text encoding obtained after GPT expansion and averaging operations. Represents the encoding features of the 1st, 2nd to nth videos, Indicates the encoding features of the first, second to nth subtitles; Use the cosine similarity formula to calculate e t and The similarity and e t and The similarity between the two videos is calculated and the similarity results are sorted to retrieve the videos most relevant to the descriptive text.

5. A text retrieval video system based on command-guided GPT, used to implement a text retrieval video method based on command-guided GPT as claimed in any one of claims 1 to 4, characterized in that: include: A data acquisition module, used to acquire multiple videos and their corresponding multiple descriptive texts; A data preprocessing module, used for inputting the descriptive text into a cross-modal retrieval model, guiding the deep learning model GPT to expand the descriptive text through instructions, obtaining extended text, and generating corresponding subtitles for the video; An encoding module, used for concatenating the extended text with the corresponding descriptive text and encoding them to obtain descriptive text encoding features, and encoding the subtitles and the video to obtain subtitle encoding features and video encoding features; A similarity acquisition module, used to acquire the similarity between the descriptive text coding features and the video coding features, and to acquire the similarity between the descriptive text coding features and the subtitle coding features; an optimization module, configured to use the non-matching subtitles as pseudo-matching subtitles when the similarity between the descriptive text encoding feature and the non-matching subtitles is higher than that between the matching subtitles; A training module is used to use the descriptive text encoding features and the common description of the pseudo-paired subtitles as enhanced text, and train the cross-modal retrieval model through the enhanced text and the pseudo-paired video to obtain the final cross-modal retrieval model; The output module is used to input the data to be retrieved into the final cross-modal retrieval model and obtain the retrieval results by sorting the cosine similarity of the output feature vectors.

6. A computer device, characterized in that: It comprises a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of a text retrieval video method based on instruction-guided GPT as described in any one of claims 1 to 4.

Images