Video cross-modal retrieval method, device and equipment and computer medium

Abstract

The invention discloses a video cross-modal retrieval method, device and equipment and a computer medium, and relates to the field of video retrieval. The method comprises the following steps: performing cue word enhancement on a user query text through a pre-training embedding model to obtain a retrieval text; performing cross-modal retrieval on the retrieval text through the large audio and video model to obtain candidate video clips corresponding to the retrieval text; performing object recognition on the candidate video clips according to the retrieval text through an object recognition depth model to obtain a to-be-replaced label and a replacement label; generating a mask according to the to-be-replaced label through a mask decoder; and adjusting the candidate video clip according to the mask and the replacement label through the text video model to obtain a target video clip. Therefore, the user query intention can be understood by utilizing prompt word enhancement, the to-be-replaced part related to the user query prompt word part is extracted from the existing candidate video clips, and the new video generated after replacement is taken as the target video clip expected by the user.

Description

Technical Field

[0001] The present invention relates to the field of video retrieval, and in particular to a video cross-modal retrieval method, device, equipment and computer medium. Background Art

[0002] The current video retrieval system uses a method to retrieve videos by improving the ability to learn multimodal representations. Although the videos retrieved in this way have a certain degree of relevance, they are not the results that users want. In other words, the existing retrieval technology can only continuously improve the retrieval recall rate of existing video content in media resources, but cannot improve the ability to understand intent. Summary of the invention

[0003] In view of this, the purpose of the present invention is to overcome the deficiencies in the prior art and to provide a video cross-modal retrieval method, apparatus, device and computer medium, which are used to significantly improve the cross-modal retrieval intent understanding ability by utilizing prompt word enhancement, and at the same time edit the objects in the video clips obtained by cross-modal retrieval according to the user prompt words, and generate a new video clip as the target video desired by the user.

[0004] The present invention provides the following technical solutions:

[0005] In a first aspect, the present invention provides a video cross-modal retrieval method, comprising:

[0006] The user query text is enhanced with prompt words through the pre-trained embedding model to obtain the retrieval text;

[0007] Performing cross-modal retrieval on the search text through the audio and video big model to obtain candidate video clips corresponding to the search text;

[0008] Performing object recognition on the candidate video clip according to the search text by using an object recognition deep model to obtain a label to be replaced and a replacement label;

[0009] Generate a mask according to the label to be replaced by a mask decoder;

[0010] The candidate video segment is adjusted according to the mask and the replacement label by using the Vincent video model to obtain a target video segment.

[0011] In one embodiment, the step of adjusting the candidate video segment according to the mask and the replacement label using the Vincent video model to obtain the target video segment includes:

[0012] Extracting the motion trajectory of the object to be replaced corresponding to the label to be replaced from the candidate video clips by using the Wensheng video model;

[0013] The candidate video segment is adjusted by the Vincent video model according to the mask, the replacement label and the motion trajectory to obtain the target video segment.

[0014] In one embodiment, the step of adjusting the candidate video segment according to the mask, the replacement label and the motion trajectory by the Vincent video model to obtain the target video segment includes:

[0015] identifying, by the Vincent video model, an adjustment region in the candidate video segment according to the mask;

[0016] The target video segment is obtained by generating a replacement object in the adjustment area according to the replacement tag and the motion trajectory through the Wensheng video model.

[0017] In one embodiment, the method of performing prompt word enhancement on the user query text by using the pre-trained embedding model to obtain the retrieval text includes:

[0018] Inputting the audio and video knowledge base into the pre-trained embedding model for encoding to obtain multiple cross-modal representation vectors;

[0019] Inputting the user query text into the pre-trained embedding model for encoding to obtain a text vector;

[0020] Obtaining a target retrieval vector closest to the text vector from each of the cross-modal representation vectors through cross-modal retrieval;

[0021] The search text is determined according to the target search vector.

[0022] In one embodiment, the object recognition of the candidate video segment is performed according to the search text by using the object recognition deep model to obtain the label to be replaced and the replacement label, including:

[0023] Segmenting the search text to obtain multiple search text labels;

[0024] Embedding each of the search text tags and the candidate video clips into the object recognition deep model;

[0025] Performing object recognition on the candidate video clips according to each of the search text labels using the object recognition deep model to obtain a plurality of image labels;

[0026] The search text labels and the image labels are compared to obtain the labels to be replaced and the replacement labels.

[0027] In one embodiment, the step of generating a mask according to the to-be-replaced label by a mask decoder includes:

[0028] Mapping the to-be-replaced label to a unique identification vector according to a predefined vocabulary;

[0029] Finding the matrix row vector with the shortest distance to the unique identification vector from the embedding matrix;

[0030] Embedding the candidate video segment into the mask decoder via a visual encoder;

[0031] The mask is generated by the mask decoder according to the candidate video segments and the matrix row vectors.

[0032] In one embodiment, the step of generating the mask according to the candidate video segment and the matrix row vector by the mask decoder includes:

[0033] Extracting image features corresponding to the row vectors of the matrix in the candidate video clips through cross attention in the mask decoder;

[0034] The mask is generated according to the image features by a fully connected convolutional network.

[0035] In a second aspect, the present invention provides a video cross-modal retrieval device, comprising:

[0036] An enhancement module is used to enhance the user query text with prompt words through a pre-trained embedding model to obtain a retrieval text;

[0037] A retrieval module, used to perform cross-modal retrieval on the retrieval text through a large audio and video model to obtain candidate video clips corresponding to the retrieval text;

[0038] A recognition module, configured to perform object recognition on the candidate video clip according to the search text by using an object recognition deep model, and obtain a label to be replaced and a replacement label;

[0039] A generating module, configured to generate a mask according to the label to be replaced by using a mask decoder;

[0040] The adjustment module is used to adjust the candidate video segment according to the mask and the replacement label through the Vincent video model to obtain a target video segment.

[0041] In a third aspect, the present invention proposes a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the video cross-modal retrieval method as described in the first aspect is implemented.

[0042] In a fourth aspect, the present invention proposes a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the video cross-modal retrieval method as described in the first aspect.

[0043] The video cross-modal retrieval method, device, equipment and computer medium disclosed in the present invention perform prompt word enhancement on the user query text through a pre-trained embedding model to obtain a retrieval text; perform cross-modal retrieval on the retrieval text through an audio and video large model to obtain a candidate video clip corresponding to the retrieval text; perform object recognition on the candidate video clip according to the retrieval text through an object recognition deep model to obtain a to-be-replaced label and a replacement label; generate a mask according to the to-be-replaced label through a mask decoder; adjust the candidate video clip according to the mask and the replacement label through a vince video model to obtain a target video clip. In this way, the user's query intention can be understood by using prompt word enhancement, and more relevant candidate video clips can be effectively found according to the retrieval text obtained after prompt word enhancement; at the same time, the semantic difference between the prompt word and the retrieval prompt word in the candidate video clip retrieved from the media asset cross-modal is considered, so as to realize the extraction of the to-be-replaced part related to the user's query prompt word part and the corresponding replacement part from the existing candidate video clip, so as to replace the to-be-replaced part and the replacement part, and then in the case that a completely consistent video clip is not retrieved, the new video generated after the replacement is used as the target video clip expected by the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope of protection of the present invention. In each of the drawings, similar components are numbered similarly.

[0045] Figure 1 A first flow chart of the video cross-modal retrieval method proposed in this embodiment is shown;

[0046] Figure 2 A second flow chart of the video cross-modal retrieval method proposed in this embodiment is shown;

[0047] Figure 3 A third flow chart of the video cross-modal retrieval method proposed in this embodiment is shown;

[0048] Figure 4 A fourth flow chart of the video cross-modal retrieval method proposed in this embodiment is shown;

[0049] Figure 5 A schematic diagram showing the first frame image in the candidate video segment proposed in this embodiment;

[0050] Figure 6 A fifth flow chart of the video cross-modal retrieval method proposed in this embodiment is shown;

[0051] Figure 7A schematic diagram showing the comparison between the candidate video segments and the target video segments proposed in this embodiment is shown;

[0052] Figure 8 A sixth flow chart of the video cross-modal retrieval method proposed in this embodiment is shown;

[0053] Fig. 9 A schematic diagram of the structure of the video cross-modal retrieval device proposed in this embodiment is shown.

[0054] Description of the accompanying drawings:

[0055] 900 - video cross-modal retrieval device; 901 - enhancement module; 902 - retrieval module; 903 - identification module; 904 - generation module; 905 - adjustment module. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0057] The components of the embodiments of the present invention generally described and shown in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0058] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present invention, are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items.

[0059] Furthermore, the terms “first”, “second”, “third”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0060] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present invention belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present invention.

[0061] Example 1

[0062] The disclosed embodiments provide a video cross-modal retrieval method for significantly improving the ability to understand cross-modal retrieval intent by utilizing prompt word enhancement, while editing objects in video clips obtained through cross-modal retrieval according to user prompt words to generate new video clips as the target video desired by the user.

[0063] See also Figure 1 ,A video cross-modal retrieval method includes steps S101 to S105 , and each step is described in detail below.

[0064] Step S101, using a pre-trained embedding model to enhance the user query text with prompt words to obtain a retrieval text.

[0065] In this embodiment, the pre-trained embedding model is a model pre-trained on a large-scale data set. The pre-trained embedding model is used to enhance the prompt words of any input user query text to obtain a semantically complete retrieval text. For example, the user query text "the quick brown fox jumped over the lazy dog" is enhanced to the retrieval text "the quick brown fox jumped over the lazy dog". In this way, the user query intention can be understood through the method of prompt word enhancement.

[0066] See also Figure 2 In a specific embodiment, step S101 includes steps S1011 to S1014, and each step is described in detail below.

[0067] Step S1011, input the audio and video knowledge base into the pre-trained embedding model for encoding to obtain multiple cross-modal representation vectors.

[0068] In this embodiment, the audio and video knowledge base including audio, video and text is first encoded into a unified cross-modal representation vector through a pre-trained embedding model, and the cross-modal representation vector can be stored in a vector database. It is recommended to use the image and text contrast learning pre-training model CLIP as a pre-trained embedding model. For Chinese text, in order to ensure the accuracy of semantic alignment of image and text cross-modal representation, it is recommended to retrain CLIP using the Chinese image and text dataset.

[0069] Step S1012: input the user query text into the pre-trained embedding model for encoding to obtain a text vector.

[0070] In this embodiment, any query text input by the user is also encoded into a text vector through the pre-trained embedding model.

[0071] Step S1013, obtaining a target search vector closest to the text vector from each of the cross-modal representation vectors through cross-modal search.

[0072] In this embodiment, a vector closest to the text vector is obtained from each cross-modal representation vector through cross-modal retrieval, and the vector is used as the target retrieval vector. In this way, the semantic difference between the prompt words of the user query text and the prompt words at the video object level in the audio and video knowledge base media assets is considered, thereby realizing the extraction of information related to the prompt words of the user query text from the existing media assets.

[0073] For example, after obtaining the text vector and each cross-modal representation vector, a distance measurement method is needed to calculate the similarity or distance between them. Commonly used distance measurement methods include Euclidean distance, cosine similarity, etc. Among them, cosine similarity is particularly commonly used in cross-modal retrieval because it can measure the degree of similarity between two vectors in direction without being affected by the length of the vector.

[0074] Furthermore, after calculating the distances between all cross-modal representation vectors and the text vector, the vector with the smallest distance is selected as the target retrieval vector. The modal data represented by this vector is the data most relevant to the user's query text and the retrieval text.

[0075] Step S1014: determining the search text according to the target search vector.

[0076] In this embodiment, a large language model is used to generate a search text according to a target search vector. The search text is a semantically complete text based on the user query text.

[0077] Please see again Figure 1 , step S102, performing cross-modal retrieval on the search text through the audio and video large model to obtain candidate video clips corresponding to the search text.

[0078] In this embodiment, the audio and video big model is a type of deep learning model trained on a large amount of audio and video data. They can capture the complex features of audio and video data, including visual content, audio features, and the temporal and spatial relationship between them. Using a loosely coupled method and combining the audio and video big model to perform cross-modal retrieval on the retrieval text, the candidate video clips that are most relevant to the semantics of the retrieval text are obtained, further improving the relevance of the video retrieval results to the user's query intent.

[0079] Please see again Figure 1 , step S103, performing object recognition on the candidate video clip according to the search text through an object recognition deep model to obtain a label to be replaced and a replacement label.

[0080] In this embodiment, the object recognition deep model is used to perform object recognition on the candidate video clip according to the search text, and the to-be-replaced label that needs to be replaced in the candidate video clip and the replacement label corresponding to the to-be-replaced label in the search text are obtained. The to-be-replaced label and the replacement label obtained by matching the candidate video clip with the search text label can be used to subsequently control the generation of the user's desired video.

[0081] See also Figure 3 In a specific embodiment, step S103 includes steps S1031 to S1034, and each step is described in detail below.

[0082] Step S1031, segmenting the search text to obtain a plurality of search text tags.

[0083] In this example, see Figure 4 First, the search text is segmented to obtain multiple search text labels. For example, search text labels include "blue sky" and "white clouds". Before segmentation, the search text may need to be preprocessed, such as removing punctuation marks, numbers, special characters, etc.

[0084] Step S1032: embed each of the search text tags and the candidate video clips into the object recognition deep model.

[0085] In this embodiment, each search text label is embedded into the decoder in the object recognition deep model through position coding, so that each search text label is combined with the corresponding position coding in the form of a numerical vector to form a composite representation containing both text information and position information, and the composite representation is then input into the decoder; the candidate video clip is embedded into the decoder through a visual encoder pre-trained on the ImageNet data set, so that the visual encoder processes the candidate video clip and outputs feature vectors, which are also input into the decoder. Since the objects that appear in the candidate video clip frame by frame are consistent, it is only necessary to identify the objects in the first frame of the candidate video clip, thereby saving computing time and reducing the performance requirements of the object recognition deep model.

[0086] Step S1033: performing object recognition on the candidate video clips according to each of the search text labels using the object recognition deep model to obtain a plurality of image labels.

[0087] In this embodiment, the object recognition deep model decoder uses a cross-attention mechanism to simultaneously focus on the features of the retrieved text labels and the candidate video clips, thereby assigning labels to each object in the candidate video clips and obtaining image labels corresponding to each image object. Benefiting from the image encoding capabilities of the pre-trained visual base model and the cross-attention of the text labels in the transformer decoder, image object recognition can be applied to open set object recognition.

[0088] For example, see Figure 5 , the first frame image includes object 1, object 2, object 3, and object 4, then the object recognition deep model decoder may match the image label "sun" for object 1, the image label "person" for object 2, the image label "column" for object 3, and the image label "cloud" for object 4.

[0089] Step S1034, comparing each of the search text labels and each of the image labels to obtain the label to be replaced and the replacement label.

[0090] In this embodiment, each search text label and each image label are compared according to the order of the labels in the original text, such as the search text label "girl, in, swimming," and the image label "boy, in, swimming," to ensure that the object that needs to be replaced is found.

[0091] Furthermore, the inconsistent tags are used as tags to be replaced and replacement tags. Among them, the tags in the search text that are inconsistent with the image tags are replacement tags, and the tags in the image tags that are inconsistent with the search text tags are tags to be replaced, such as "girl" is the replacement tag, and "boy" is the tag to be replaced.

[0092] Please see again Figure 1 , step S104, generating a mask according to the label to be replaced by a mask decoder.

[0093] In this embodiment, a mask decoder is used to generate a mask of the region to be segmented in the candidate video segment according to the label to be replaced, so as to ensure that only the portion to be replaced can generate the object expected by the user in the subsequent steps.

[0094] In a specific embodiment, step S104 includes: mapping the label to be replaced to a unique identification vector according to a predefined vocabulary; searching for a matrix row vector having the shortest distance to the unique identification vector from an embedding matrix; embedding the candidate video segment into the mask decoder through a visual encoder; and generating the mask according to the candidate video segment and the matrix row vector through the mask decoder.

[0095] In this example, see Figure 6, because the data diversity in the field of video retrieval is still limited, and the Segmentation Large Model (SAM) is aimed at general image object segmentation. Therefore, the mask decoder is designed to convert labels into vectors using word embedding instead of position encoding. This requires mapping the label to be replaced to a unique identification vector according to a predefined vocabulary, and then finding the matrix row vector with the shortest distance to the unique identification vector from an embedding matrix. The vocabulary and embedding matrix provided by the large language model are used here, and the gpt2 version is recommended.

[0096] At the same time, the image of the candidate video segment is also embedded into the mask decoder through the image encoder. Since the image label used to save computing time and reduce the performance requirements of the object recognition deep model is the image label corresponding to the first frame image of the candidate video segment, it is only necessary to embed the first frame image of the candidate video segment into the mask decoder. Further, a mask is generated according to the image of the candidate video segment and the matrix row vector through the mask decoder. The mask indicates the position and outline of the area to be replaced in the candidate video segment.

[0097] In a specific embodiment, generating the mask according to the candidate video segment and the matrix row vector by the mask decoder includes: extracting image features corresponding to the matrix row vector in the candidate video segment by cross-attention in the mask decoder; generating the mask according to the image features by a fully connected convolutional network.

[0098] In this embodiment, the cross attention in the mask decoder includes image-to-label cross attention and label-to-image cross attention. The image-to-label cross attention focuses on the image's attention to the label to be replaced, that is, the row vector of the matrix, and the label-to-image cross attention focuses on the attention of each element in the label to be replaced to each position in the image, so that the extracted image features are more focused on the label to be replaced.

[0099] Furthermore, a fully connected convolutional network with Resnet-152 as the network backbone is used to predict the mask based on the image features. Resnet-152 is used as the network backbone, and its deep convolutional layers can extract high-level features in the image, which are essential for understanding and classifying the image; in the fully connected convolutional network, further convolution and pooling operations can further optimize the feature representation, making it more compact and effective.

[0100] Please see again Figure 1 , step S105, adjusting the candidate video segment according to the mask and the replacement label through the Vincent video model to obtain a target video segment.

[0101] In this embodiment, the candidate video clips are adjusted according to the mask of the replaced part and the replacement label through the Wensheng video model, so as to obtain the target video clip that meets the user's intention, thereby ensuring that the video object in the candidate video clip can be replaced while retaining the other content of the original video clip. In addition, there is no need to use vertical audio and video data for fine-tuning like the large audio and video model, which can save the cost of fine-tuning training.

[0102] For example, see Figure 7 , the candidate video segment includes image labels A and B, and the retrieved text labels are A and C, then the labels B and C are replaced to obtain the target video segment including labels A and C.

[0103] See also Figure 8 In a specific embodiment, step S105 includes steps S1051 to S1052, and each step is described in detail below.

[0104] Step S1051: extracting the motion trajectory of the object to be replaced corresponding to the label to be replaced from the candidate video clips through the Wensheng video model.

[0105] In this embodiment, the Vincent video model is used to extract the motion trajectory of the object to be replaced corresponding to the label to be replaced in the candidate video clip, which can be obtained by an optical flow tracking method. For example, the sparse optical flow is calculated by an optical flow algorithm (Lucas-Kanade, LK), and the corner points of the image to be replaced are used as pixels for LK tracking using corner point detection (Shi-Tomasi).

[0106] Step S1052: adjusting the candidate video segment according to the mask, the replacement label and the motion trajectory through the Vincent video model to obtain the target video segment.

[0107] In this embodiment, the Vincent video model adjusts the candidate video clip according to the mask of the part to be replaced, the motion trajectory of the image to be replaced, and the replacement label, and obtains a target video clip containing a replacement object corresponding to the replacement label, and the motion trajectory of the replacement object is consistent with the motion trajectory of the object to be replaced. In this way, it is ensured that the video object can be replaced accurately while retaining other contents of the original video clip to the greatest extent.

[0108] In a specific embodiment, step S1052 includes: identifying the adjustment area in the candidate video segment according to the mask through the Vincent video model; generating a replacement object in the adjustment area according to the replacement label and the motion trajectory through the Vincent video model to obtain the target video segment.

[0109] In this embodiment, the common parameter efficient fine-tuning method (LoRA) is used to fine-tune the added motion trajectory of the Vincent video model as the input cross-attention layer, and the weights of other layers of the Vincent video model deep network are frozen to identify the adjustment area in the candidate video clip, so as to generate a replacement object in the adjustment area according to the replacement label and the motion trajectory, so as to realize that the Vincent video large model uses the motion trajectory of the object to be replaced as an additional instruction of the text control instruction to control the motion trajectory of the replacement object in the target video clip to be consistent with the motion trajectory of the object to be replaced in the candidate video object. Among them, the Vincent video model can be a Vincent video network structure based on stable diffusion plus temporal attention mechanism.

[0110] It should be noted that a replacement image object appearance enhancement method can be designed to ensure that the replacement object is consistent with the style of the candidate video clip. Specifically, stylized fine-tuning is added to the temporal attention layer, and the image appearance representation outside the mask area is used as an additional input to the temporal attention layer. LoRA fine-tuning is also used to learn the relationship between the appearance of the newly generated replacement image and the additional input, so as to meet the appearance consistency.

[0111] It should be noted that this embodiment can also support the editing of multiple video objects and video generation. At the same time, it can also directly add network sub-blocks to the deep network model of the large audio and video model and retrain it, thus becoming a tightly coupled prompt word enhancement method.

[0112] The video cross-modal retrieval method proposed in this embodiment performs prompt word enhancement on the user query text through a pre-trained embedding model to obtain a retrieval text; performs cross-modal retrieval on the retrieval text through a large audio and video model to obtain a candidate video segment corresponding to the retrieval text; performs object recognition on the candidate video segment according to the retrieval text through an object recognition deep model to obtain a label to be replaced and a replacement label; generates a mask according to the label to be replaced through a mask decoder; and adjusts the candidate video segment according to the mask and the replacement label through a viz-sense video model to obtain a target video segment. In this way, the user's query intention can be understood by using prompt word enhancement, and more relevant candidate video segments can be effectively found according to the retrieval text obtained after prompt word enhancement; at the same time, the semantic difference between the prompt word and the retrieval prompt word in the candidate video segment retrieved from the media asset cross-modal is considered, so as to extract the part to be replaced related to the user query prompt word part and the corresponding replacement part from the existing candidate video segment, so as to replace the part to be replaced and the replacement part, and then, if a completely consistent video segment is not retrieved, the new video generated after replacement is used as the target video segment expected by the user.

[0113] Example 2

[0114] In addition, the present disclosure provides a video cross-modal retrieval device 900, see Fig. 9 , the device comprises:

[0115] An enhancement module 901 is used to enhance the user query text with prompt words through a pre-trained embedding model to obtain a retrieval text;

[0116] A retrieval module 902 is used to perform cross-modal retrieval on the search text through an audio and video macro model to obtain candidate video clips corresponding to the search text;

[0117] The recognition module 903 is used to perform object recognition on the candidate video clip according to the search text through an object recognition deep model to obtain a label to be replaced and a replacement label;

[0118] A generating module 904, configured to generate a mask according to the label to be replaced by using a mask decoder;

[0119] The adjustment module 905 is used to adjust the candidate video segment according to the mask and the replacement label through the Vincent video model to obtain a target video segment.

[0120] Optionally, the adjustment module 905 is also used to extract the motion trajectory of the object to be replaced corresponding to the label to be replaced from the candidate video segment through the Vincent video model; adjust the candidate video segment according to the mask, the replacement label and the motion trajectory through the Vincent video model to obtain the target video segment.

[0121] Optionally, the adjustment module 905 is further used to identify the adjustment area in the candidate video segment according to the mask through the Vincent video model; generate a replacement object in the adjustment area according to the replacement label and the motion trajectory through the Vincent video model to obtain the target video segment.

[0122] Optionally, the enhancement module 901 is also used to input the audio and video knowledge base into the pre-trained embedding model for encoding to obtain multiple cross-modal representation vectors; input the user query text into the pre-trained embedding model for encoding to obtain a text vector; obtain a target retrieval vector that is closest to the text vector from each of the cross-modal representation vectors through cross-modal retrieval; and determine the retrieval text based on the target retrieval vector.

[0123] Optionally, the recognition module 903 is also used to segment the search text to obtain multiple search text labels; embed each of the search text labels and the candidate video clips into the object recognition deep model; perform object recognition on the candidate video clips according to each of the search text labels through the object recognition deep model to obtain multiple image labels; compare each of the search text labels and each of the image labels to obtain the label to be replaced and the replacement label.

[0124] Optionally, the generation module 904 is also used to map the label to be replaced to a unique identification vector according to a predefined vocabulary; search the matrix row vector with the shortest distance to the unique identification vector from the embedding matrix; embed the candidate video segment into the mask decoder through the visual encoder; and generate the mask according to the candidate video segment and the matrix row vector through the mask decoder.

[0125] Optionally, the generation module 904 is further used to extract image features corresponding to the row vectors of the matrix in the candidate video segment through cross-attention in the mask decoder; and generate the mask according to the image features through a fully connected convolutional network.

[0126] The device provided in the embodiment of the present disclosure can execute the steps of the video cross-modal retrieval method provided in Example 1, which will not be described again to avoid repetition.

[0127] The video cross-modal retrieval device proposed in this embodiment performs prompt word enhancement on the user query text through a pre-trained embedding model to obtain a retrieval text; performs cross-modal retrieval on the retrieval text through an audio and video large model to obtain a candidate video segment corresponding to the retrieval text; performs object recognition on the candidate video segment according to the retrieval text through an object recognition deep model to obtain a tag to be replaced and a replacement tag; generates a mask according to the tag to be replaced through a mask decoder; and adjusts the candidate video segment according to the mask and the replacement tag through a viz-sense video model to obtain a target video segment. In this way, the user's query intention can be understood by using prompt word enhancement, and more relevant candidate video segments can be effectively found according to the retrieval text obtained after prompt word enhancement; at the same time, the semantic difference between the prompt word and the retrieval prompt word in the candidate video segment retrieved from the media asset cross-modal is considered, so as to extract the to-be-replaced part related to the user's query prompt word part and the corresponding replacement part from the existing candidate video segment, so as to replace the to-be-replaced part and the replacement part, and then, if a completely identical video segment is not retrieved, the new video generated after replacement is used as the target video segment expected by the user.

[0128] Example 3

[0129] In addition, an embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the video cross-modal retrieval method described in Example 1 is implemented.

[0130] The device provided in the embodiment of the present disclosure can execute the steps of the video cross-modal retrieval method provided in Example 1, which will not be described again to avoid repetition.

[0131] Example 4

[0132] The disclosed embodiment provides a computer-readable storage medium storing a computer program, which implements the video cross-modal retrieval method described in Embodiment 1 when executed by a processor.

[0133] In this embodiment, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0134] The computer-readable storage medium provided in this embodiment can implement the video cross-modal retrieval method provided in Example 1, and will not be described again here to avoid repetition.

[0135] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limiting, and thus other examples of the exemplary embodiments may have different values.

[0136] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0137] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A video cross-modal retrieval method, characterized in that: include: The user query text is enhanced with prompt words through the pre-trained embedding model to obtain the retrieval text; Performing cross-modal retrieval on the search text through the audio and video big model to obtain candidate video clips corresponding to the search text; Performing object recognition on the candidate video clip according to the search text by using an object recognition deep model to obtain a label to be replaced and a replacement label; Generate a mask according to the label to be replaced by a mask decoder; The candidate video segment is adjusted according to the mask and the replacement label by using the Vincent video model to obtain a target video segment.

2. The video cross-modal retrieval method according to claim 1, characterized in that: The step of adjusting the candidate video segment according to the mask and the replacement label by using the Vincent video model to obtain a target video segment includes: Extracting the motion trajectory of the object to be replaced corresponding to the label to be replaced from the candidate video clips by using the Wensheng video model; The candidate video segment is adjusted by the Vincent video model according to the mask, the replacement label and the motion trajectory to obtain the target video segment.

3. The video cross-modal retrieval method according to claim 2, characterized in that: The step of adjusting the candidate video segment according to the mask, the replacement label and the motion trajectory by using the Vincent video model to obtain the target video segment includes: identifying, by the Vincent video model, an adjustment region in the candidate video segment according to the mask; The target video segment is obtained by generating a replacement object in the adjustment area according to the replacement tag and the motion trajectory through the Wensheng video model.

4. The video cross-modal retrieval method according to claim 1, characterized in that: The method of performing prompt word enhancement on the user query text by using the pre-trained embedding model to obtain the retrieval text includes: Inputting the audio and video knowledge base into the pre-trained embedding model for encoding to obtain multiple cross-modal representation vectors; Inputting the user query text into the pre-trained embedding model for encoding to obtain a text vector; Obtaining a target retrieval vector closest to the text vector from each of the cross-modal representation vectors through cross-modal retrieval; The search text is determined according to the target search vector.

5. The video cross-modal retrieval method according to claim 1, characterized in that: The step of performing object recognition on the candidate video clip according to the search text by using the object recognition deep model to obtain the label to be replaced and the replacement label includes: Segmenting the search text to obtain multiple search text labels; Embedding each of the search text tags and the candidate video clips into the object recognition deep model; Performing object recognition on the candidate video clips according to each of the search text labels using the object recognition deep model to obtain a plurality of image labels; The search text labels and the image labels are compared to obtain the labels to be replaced and the replacement labels.

6. The video cross-modal retrieval method according to claim 1, characterized in that: The step of generating a mask according to the label to be replaced by using a mask decoder includes: Mapping the to-be-replaced label to a unique identification vector according to a predefined vocabulary; Finding the matrix row vector with the shortest distance to the unique identification vector from the embedding matrix; Embedding the candidate video segment into the mask decoder via a visual encoder; The mask is generated by the mask decoder according to the candidate video segments and the matrix row vectors.

7. The video cross-modal retrieval method according to claim 6, characterized in that: The step of generating the mask according to the candidate video segment and the matrix row vector by the mask decoder includes: Extracting image features corresponding to the row vectors of the matrix in the candidate video clips through cross attention in the mask decoder; The mask is generated according to the image features by a fully connected convolutional network.

8. A video cross-modal retrieval device, characterized in that: include: An enhancement module is used to enhance the user query text with prompt words through a pre-trained embedding model to obtain a retrieval text; A retrieval module, used to perform cross-modal retrieval on the retrieval text through a large audio and video model to obtain candidate video clips corresponding to the retrieval text; A recognition module, configured to perform object recognition on the candidate video clip according to the search text by using an object recognition deep model, and obtain a label to be replaced and a replacement label; A generating module, configured to generate a mask according to the label to be replaced by using a mask decoder; The adjustment module is used to adjust the candidate video segment according to the mask and the replacement label through the Vincent video model to obtain a target video segment.

9. A computer device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the video cross-modal retrieval method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the video cross-modal retrieval method as described in any one of claims 1 to 7.

Images