Video cross-modal retrieval method, device and equipment and computer medium

Abstract

The application discloses a video cross-modal retrieval method and device, equipment and computer medium, and relates to the field of video retrieval. The method comprises the following steps: enhancing the user query text by a pre-training embedding model to obtain a retrieval text; performing cross-modal retrieval on the retrieval text by an audio-video large model to obtain a candidate video segment corresponding to the retrieval text; performing object recognition on the candidate video segment according to the retrieval text by an object recognition deep model to obtain a label to be replaced and a replacement label; generating a mask according to the label to be replaced by a mask decoder; and adjusting the candidate video segment according to the mask and the replacement label by a text-to-video model to obtain a target video segment. In this way, the user query intention can be understood by using the prompt word enhancement, and the part to be replaced related to the user query prompt word part can be extracted from the existing candidate video segment, and the new video generated after replacement is used as the target video segment expected by the user.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video retrieval, and in particular to a video cross-modal retrieval method, device, equipment and computer medium. BACKGROUND

[0002] The method used by the current video retrieval system retrieves videos by improving the multi-modal representation learning capability, so that the retrieved videos have certain relevance, but are not the results that the user wants. That is, the existing retrieval technology can only continuously improve the retrieval recall rate of the existing video content in the media asset, but cannot improve the intention understanding capability. SUMMARY

[0003] Therefore, the purpose of the present application is to overcome the deficiencies in the prior art, and to provide a video cross-modal retrieval method, device, equipment and computer medium for significantly improving the cross-modal retrieval intention understanding capability by using prompt word enhancement, and for editing objects in the video segments obtained by cross-modal retrieval according to user prompt words to generate new video segments as target videos expected by the user.

[0004] The present application provides the following technical solutions:

[0005] In a first aspect, the present application provides a video cross-modal retrieval method, comprising:

[0006] prompt word enhancement of user query text is performed by a pre-trained embedding model to obtain retrieval text;

[0007] cross-modal retrieval of the retrieval text is performed by an audio-video large model to obtain candidate video segments corresponding to the retrieval text;

[0008] object recognition of the candidate video segments is performed according to the retrieval text by an object recognition deep model to obtain a to-be-replaced label and a replacement label;

[0009] a mask is generated according to the to-be-replaced label by a mask decoder;

[0010] the candidate video segments are adjusted according to the mask and the replacement label by a text-to-video model to obtain a target video segment.

[0011] In an embodiment, the candidate video segments are adjusted according to the mask and the replacement label by the text-to-video model to obtain a target video segment, comprising:

[0012] motion trajectories of a to-be-replaced object corresponding to the to-be-replaced label are extracted from the candidate video segments by the text-to-video model;

[0013] adjust the candidate video segment according to the mask, the replacement label and the motion track by the text-to-video model to obtain the target video segment.

[0014] In an embodiment, the adjusting the candidate video segment according to the mask, the replacement label and the motion track by the text-to-video model to obtain the target video segment comprises:

[0015] identifying an adjustment region in the candidate video segment according to the mask by the text-to-video model;

[0016] generating a replacement object in the adjustment region according to the replacement label and the motion track by the text-to-video model to obtain the target video segment.

[0017] In an embodiment, the prompting word enhancing the user query text by the pre-training embedding model to obtain the retrieval text comprises:

[0018] inputting the audio-video knowledge base into the pre-training embedding model for encoding to obtain a plurality of cross-modal representation vectors;

[0019] inputting the user query text into the pre-training embedding model for encoding to obtain a text vector;

[0020] obtaining a target retrieval vector closest to the text vector from each of the cross-modal representation vectors by cross-modal retrieval;

[0021] determining the retrieval text according to the target retrieval vector.

[0022] In an embodiment, the object recognition of the candidate video segment according to the retrieval text by the object recognition deep model to obtain the to-be-replaced label and the replacement label comprises:

[0023] segmenting the retrieval text to obtain a plurality of retrieval text labels;

[0024] embedding each of the retrieval text labels and the candidate video segment into the object recognition deep model;

[0025] object recognizing the candidate video segment according to each of the retrieval text labels by the object recognition deep model to obtain a plurality of image labels;

[0026] comparing each of the retrieval text labels and each of the image labels to obtain the to-be-replaced label and the replacement label.

[0027] In an embodiment, the generating a mask according to the to-be-replaced label by the mask decoder comprises:

[0028] mapping the label to be replaced to a unique identification vector according to a predefined vocabulary;

[0029] finding a matrix row vector with the shortest distance from the unique identification vector in the embedding matrix;

[0030] embedding the candidate video clip into the mask decoder through a visual encoder;

[0031] generating the mask through the mask decoder according to the candidate video clip and the matrix row vector.

[0032] In an embodiment, the generating the mask through the mask decoder according to the candidate video clip and the matrix row vector comprises:

[0033] extracting image features corresponding to the matrix row vector in the candidate video clip through cross attention in the mask decoder;

[0034] generating the mask through a fully connected convolutional network according to the image features.

[0035] In a second aspect, the present application provides a video cross-modal retrieval device, comprising:

[0036] an enhancement module configured to perform prompt word enhancement on user query text through a pre-trained embedding model to obtain retrieval text;

[0037] a retrieval module configured to perform cross-modal retrieval on the retrieval text through an audio-video large model to obtain candidate video clips corresponding to the retrieval text;

[0038] an identification module configured to perform object identification on the candidate video clips according to the retrieval text through an object identification deep model to obtain a label to be replaced and a replacement label;

[0039] a generation module configured to generate a mask through a mask decoder according to the label to be replaced;

[0040] an adjustment module configured to adjust the candidate video clips according to the mask and the replacement label through a text-to-video model to obtain a target video clip.

[0041] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the video cross-modal retrieval method according to the first aspect.

[0042] In a fourth aspect, the present application provides a computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the video cross-modal retrieval method according to the first aspect.

[0043] The video cross-modal retrieval method, device, equipment and computer medium disclosed by the application, through pre-training of an embedding model, the user query text is prompted and enhanced to obtain a retrieval text; through an audio-video large model, the retrieval text is subjected to cross-modal retrieval to obtain a candidate video segment corresponding to the retrieval text; through an object recognition deep model, the candidate video segment is subjected to object recognition according to the retrieval text to obtain a to-be-replaced label and a replacement label; through a mask decoder, a mask is generated according to the to-be-replaced label; through a text-to-video model, the candidate video segment is adjusted according to the mask and the replacement label to obtain a target video segment. In this way, the user query intention can be understood by using prompt word enhancement, and the retrieval text obtained after prompt word enhancement can effectively find more relevant candidate video segments; meanwhile, the semantic difference between the prompt words in the candidate video segment retrieved from the media asset and the retrieval prompt words is considered, so as to extract a to-be-replaced part and a corresponding replacement part from the existing candidate video segment, which are related to the user query prompt words, and the to-be-replaced part and the replacement part are replaced, and then in the case that a completely consistent video segment is not retrieved, a new video generated after replacement is used as a target video segment expected by the user. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the protection scope of the present application. In each drawing, similar components are marked with similar reference numerals.

[0045] Figure 1 A first flowchart of the video cross-modal retrieval method proposed in the present embodiment is shown;

[0046] Figure 2 A second flowchart of the video cross-modal retrieval method proposed in the present embodiment is shown;

[0047] Figure 3 A third flowchart of the video cross-modal retrieval method proposed in the present embodiment is shown;

[0048] Figure 4 A fourth flowchart of the video cross-modal retrieval method proposed in the present embodiment is shown;

[0049] Figure 5 A schematic diagram of a first frame image in the candidate video segment proposed in the present embodiment is shown;

[0050] Figure 6 A fifth flowchart of the video cross-modal retrieval method proposed in the present embodiment is shown;

[0051] Figure 7A comparison diagram of the candidate video clip and the target video clip proposed in the embodiment is shown.

[0052] Figure 8 A sixth flow diagram of the video cross-modal retrieval method proposed in the embodiment is shown.

[0053] Figure 9 A structure diagram of the video cross-modal retrieval device proposed in the embodiment is shown.

[0054] The accompanying drawings are used to explain the present application.

[0055] 900-video cross-modal retrieval device; 901-enhancement module; 902-retrieval module; 903-identification module; 904-generation module; 905-adjustment module. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application.

[0057] The components of the embodiments of the present application generally described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0058] In the following, the terms "include", "have", and their synonymous words used in various embodiments of the present application are only intended to represent specific features, numbers, steps, operations, elements, components, or combinations of the foregoing, and should not be understood as excluding the existence or possibility of adding one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing.

[0059] In addition, the terms "first", "second", "third", and the like are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0060] Unless specifically defined, all other technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which various embodiments of the present application belong. The terminology used herein (such as the terminology that is defined in commonly used dictionaries) shall be interpreted as having a meaning that is consistent with its meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless clearly defined specifically herein.

[0061] Embodiment 1

[0062] The embodiment of the present disclosure provides a video cross-modal retrieval method, which is used for significantly improving the cross-modal retrieval intention understanding ability by using prompt word enhancement, and editing objects in the video segment obtained by cross-modal retrieval according to the user prompt word to generate a new video segment as the target video expected by the user.

[0063] Please refer to Figure 1 A video cross-modal retrieval method includes steps S101-S105, which are described in detail below.

[0064] Step S101, the user query text is prompted by a pre-trained embedding model to obtain a retrieval text.

[0065] In this embodiment, the pre-trained embedding model is a model pre-trained on a large-scale dataset. The pre-trained embedding model is used to enhance the prompt word of the user query text to obtain a semantic complete retrieval text. For example, the user query text "agile brown fox jumps over lazy" is enhanced to the retrieval text "agile brown fox jumps over lazy dog". In this way, the user query intention can be understood by the method of prompt word enhancement.

[0066] Please refer to Figure 2 In a specific embodiment, step S101 includes steps S1011-S1014, which are described in detail below.

[0067] Step S1011, input the audio-video knowledge base to the pre-trained embedding model for encoding to obtain a plurality of cross-modal representation vectors.

[0068] In this embodiment, the audio-video knowledge base including audio-video and text is first encoded into a unified cross-modal representation vector by a pre-trained embedding model. The cross-modal representation vector can be stored in a vector database. It is recommended to use the image and text contrast learning pre-training model CLIP as the pre-trained embedding model. For Chinese text, to ensure the semantic alignment accuracy of image and text cross-modal representation, it is recommended to retrain CLIP using a Chinese image-text dataset.

[0069] Step S1012, input the user query text into the pre-trained embedding model for encoding to obtain a text vector.

[0070] In this embodiment, the user input arbitrary query text is also encoded into a text vector by the pre-trained embedding model.

[0071] Step S1013, obtain the target retrieval vector closest to the text vector from each cross-modal representation vector through cross-modal retrieval.

[0072] In this embodiment, the vector closest to the text vector is obtained from each cross-modal representation vector through cross-modal retrieval, and this vector is taken as the target retrieval vector. In this way, the semantic difference between the prompt words of the user query text and the prompt words of the video object level in the audio-video knowledge base media asset is considered, so as to extract information related to the prompt word part of the user query text from the existing media asset.

[0073] Demonstratively, after obtaining the text vector and each cross-modal representation vector, a distance measurement method needs to be used to calculate the similarity or distance between them. Common distance measurement methods include Euclidean distance, cosine similarity, etc. Among them, cosine similarity is particularly common in cross-modal retrieval because it can measure the similarity of two vectors in direction without being affected by the length of the vector.

[0074] Further, after calculating the distance between all cross-modal representation vectors and the text vector, the vector with the smallest distance is selected as the target retrieval vector. The modality data represented by this vector is the data most relevant to the user query text and the retrieval text.

[0075] Step S1014, determine the retrieval text according to the target retrieval vector.

[0076] In this embodiment, the retrieval text is generated by the large language model according to the target retrieval vector, and the retrieval text is a semantically complete text based on the user query text.

[0077] Please refer again to Figure 1 , step S102, perform cross-modal retrieval on the retrieval text by the audio-video large model to obtain the candidate video clip corresponding to the retrieval text.

[0078] In this embodiment, the audio-video large model is a kind of deep learning model trained on a large amount of audio-video data, which can capture the complex features of audio-video data, including visual content, audio features and their temporal-spatial relationship. By using the loosely coupled mode and combining the audio-video large model to perform cross-modal retrieval on the retrieval text, the candidate video clip most semantically related to the retrieval text is obtained, further improving the relevance of the video retrieval result to the user query intent.

[0079] Please refer to Figure 1 , step S103, object recognition of the candidate video segment according to the search text is performed by an object recognition deep model to obtain a to-be-replaced label and a replacement label.

[0080] In this embodiment, the object recognition deep model is used to perform object recognition on the candidate video segment according to the search text to obtain a to-be-replaced label in the candidate video segment that needs to be replaced and a replacement label corresponding to the to-be-replaced label in the search text. The to-be-replaced label and the replacement label obtained through the matching of the candidate video segment and the search text label can be used for subsequent control of the generation of the user-desired video.

[0081] Please refer to Figure 3 In a specific embodiment, step S103 includes steps S1031-S1034, which are described in detail below.

[0082] Step S1031, the search text is segmented to obtain a plurality of search text labels.

[0083] In this embodiment, please refer to Figure 4 First, the search text is segmented to obtain a plurality of search text labels. The search text labels are, for example, "blue sky" and "white cloud". Before segmentation, the search text may need to be preprocessed, such as removing punctuation, numbers, special characters, etc.

[0084] Step S1032, each of the search text labels and the candidate video segment is embedded into the object recognition deep model.

[0085] In this embodiment, each search text label is embedded into the decoder in the object recognition deep model through positional encoding, so that each search text label is combined with the corresponding positional encoding in the form of a numerical vector to form a complex representation containing both text information and position information. This complex representation is then input into the decoder; the candidate video segment is embedded into the decoder through a visual encoder pre-trained on the ImageNet dataset, so that the visual encoder processes the candidate video segment and outputs feature vectors, which are also input into the decoder. Since the objects appearing in each frame of the candidate video segment are consistent, only the objects in the first frame of the candidate video segment need to be recognized, thereby saving computing time and reducing the performance requirements of the object recognition deep model.

[0086] Step S1033, object recognition of the candidate video segment according to each of the search text labels is performed by the object recognition deep model to obtain a plurality of image labels.

[0087] In the embodiment, the object recognition deep model decoder uses cross-attention mechanism to simultaneously focus on the features of the retrieved text labels and the candidate video clips, so as to assign labels to each object in the candidate video clips, and obtain image labels corresponding to each image object. Benefiting from the image coding capability of the pre-trained visual base large model and the cross-attention of the text labels in the transformer decoder, the image object recognition can be applied to open set object recognition.

[0088] For example, referring to Figure 5 , the first frame of image includes object 1, object 2, object 3, and object 4, then the object recognition deep model decoder can match the image label “sun” for object 1, the image label “person” for object 2, the image label “column” for object 3, and the image label “cloud” for object 4.

[0089] In step S1034, the to-be-replaced label and the replaced label are obtained by comparing each of the retrieved text labels and each of the image labels.

[0090] In the embodiment, each of the retrieved text labels and each of the image labels are compared according to the order of the labels in the original text, such as the retrieved text label “girl, in, swimming,” and the image label “boy, in, swimming,” so as to ensure the object to be replaced.

[0091] Further, the inconsistent labels are taken as the to-be-replaced label and the replaced label. The label inconsistent with the image label in the retrieved text label is the replaced label, and the label inconsistent with the retrieved text label in the image label is the to-be-replaced label, such as “girl” as the replaced label and “boy” as the to-be-replaced label.

[0092] For example, referring to Figure 1 , in step S104, the mask decoder generates a mask according to the to-be-replaced label.

[0093] In the embodiment, the mask decoder generates a mask of the region to be segmented in the candidate video clip according to the to-be-replaced label, so as to ensure that only the part to be replaced can generate the object expected by the user in the subsequent steps.

[0094] In a specific embodiment, step S104 includes: mapping the to-be-replaced label to a unique identification vector according to a predefined vocabulary table; finding a matrix row vector with the shortest distance from the unique identification vector from an embedding matrix; embedding the candidate video clip into the mask decoder through a visual encoder; and generating the mask by the mask decoder according to the candidate video clip and the matrix row vector.

[0095] In the embodiment, referring to Figure 6Since the data diversity in the field of video retrieval still has a certain range, and the segmentation large model (SAM) is for general image object segmentation. Therefore, the mask decoder is designed to convert the label into a vector without using position encoding but using word embedding. This requires mapping the label to be replaced to a unique identification vector according to a predefined vocabulary, and then finding the matrix row vector with the shortest distance from the unique identification vector in an embedding matrix. Here, the vocabulary and embedding matrix provided by the large language model are used, and it is recommended to use the gpt2 version.

[0096] At the same time, the image of the candidate video segment is also embedded into the mask decoder through the image encoder. Since the image label used to save calculation time and reduce the performance requirements of the object recognition deep model is the image label corresponding to the first frame image of the candidate video segment, here we only need to embed the first frame image of the candidate video segment into the mask decoder. Further, the mask is generated by the mask decoder according to the image of the candidate video segment and the matrix row vector. The mask indicates the position and outline of the region to be replaced in the candidate video segment.

[0097] In a specific embodiment, the generating the mask by the mask decoder according to the candidate video segment and the matrix row vector comprises: extracting image features corresponding to the matrix row vector in the candidate video segment by cross attention in the mask decoder; generating the mask according to the image features by a fully connected convolutional network.

[0098] In this embodiment, the cross attention in the mask decoder includes image-to-label cross attention and label-to-image cross attention. The image-to-label cross attention focuses on the attention degree of the image to the label to be replaced, i.e. the matrix row vector, and the label-to-image cross attention focuses on the attention degree of each element in the label to be replaced to each position in the image, so that the extracted image features are more focused on the label to be replaced.

[0099] Further, the mask is predicted by a fully connected convolutional network with Resnet-152 as the network skeleton according to the image features. Wherein, Resnet-152 as the network skeleton, its deep convolutional layer can extract high-level features in the image, which are crucial for image understanding and classification; in the fully connected convolutional network, through further convolution and pooling operations, the feature representation can be further optimized to be more compact and effective.

[0100] Please refer again to Figure 1 , step S105, adjusting the candidate video segment according to the mask and the replacement label by the text-to-video model to obtain a target video segment.

[0101] In the embodiment, the candidate video segment is adjusted by the text-to-video model according to the mask of the part to be replaced and the replacement label, so as to obtain a target video segment in line with the user's intention, thereby ensuring that the video object in the candidate video segment can be replaced, and other contents of the original video segment are also retained. In addition, without fine-tuning using vertical field audio and video data as the audio and video large model, the cost of fine-tuning training can be saved.

[0102] For example, as shown in Figure 7 , the candidate video segment includes image labels A and B, and the retrieval text label is A and C, labels B and C are replaced to obtain a target video segment including labels A and C.

[0103] For example, as shown in Figure 8 In a specific embodiment, step S105 includes steps S1051-S1052, which will be described in detail below.

[0104] In step S1051, the text-to-video model extracts the motion trajectory of the object to be replaced corresponding to the label to be replaced from the candidate video segment.

[0105] In the embodiment, the text-to-video model extracts the motion trajectory of the object to be replaced corresponding to the label to be replaced from the candidate video segment, which can be obtained by an optical flow tracking method. For example, sparse optical flow is calculated by the optical flow algorithm (Lucas-Kanade, LK), and the corner points of the replacement image are detected (Shi-Tomasi) as the pixels for LK tracking.

[0106] In step S1052, the text-to-video model adjusts the candidate video segment according to the mask, the replacement label and the motion trajectory to obtain the target video segment.

[0107] In the embodiment, the text-to-video model adjusts the candidate video segment according to the mask of the part to be replaced, the motion trajectory of the replacement image and the replacement label, to obtain a target video segment containing a replacement object corresponding to the replacement label, and the motion trajectory of the replacement object is consistent with the motion trajectory of the object to be replaced. This ensures that the video object can be accurately replaced, and other contents of the original video segment are also retained to the greatest extent.

[0108] In a specific embodiment, step S1052 includes: identifying an adjustment region in the candidate video segment according to the mask by the text-to-video model; and generating a replacement object in the adjustment region according to the replacement label and the motion trajectory by the text-to-video model to obtain the target video segment.

[0109] In this embodiment, the commonly used parameter efficient fine-tuning method (LoRA) is used to fine-tune the added motion trajectory of the text-to-video model as the input cross-attention layer, and the weights of other layers of the text-to-video model deep network are frozen to identify the adjustment area in the candidate video segment, so as to realize the generation of the replacement object in the adjustment area according to the replacement label and the motion trajectory, so as to realize that the text-to-video large model controls the motion trajectory of the replacement object in the target video segment as the additional instruction of the text control instruction. The motion trajectory of the replacement object in the candidate video object is consistent. Wherein, the text-to-video model can be a text-to-video network structure based on stablediffusion plus time attention mechanism.

[0110] It should be noted that a replacement image object appearance enhancement method can be designed to ensure that the replacement object is consistent with the style of the candidate video segment. Specifically, style fine-tuning is added to the time attention layer, and the image appearance representation outside the mask area is used as additional input to the time attention layer. Similarly, LoRA fine-tuning is used to learn the relationship between the newly generated replacement image appearance and the additional input, thereby meeting the appearance consistency.

[0111] It should be noted that the present embodiment can also support editing and video generation of multiple video objects, and at the same time, network sub-blocks can be added to the deep network model of the audio-video large model and retrained, thereby becoming a tightly coupled prompt enhancement method.

[0112] The video cross-modal retrieval method proposed in this embodiment uses a pre-trained embedding model to perform prompt enhancement on the user query text to obtain a retrieval text; an audio-video large model is used to perform cross-modal retrieval on the retrieval text to obtain a candidate video segment corresponding to the retrieval text; an object recognition depth model is used to perform object recognition on the candidate video segment according to the retrieval text to obtain a replacement label and a replacement label; a mask decoder is used to generate a mask according to the replacement label; and a text-to-video model is used to adjust the candidate video segment according to the mask and the replacement label to obtain a target video segment. In this way, the prompt enhancement can understand the user query intent, and the retrieval text obtained after the prompt enhancement can effectively find more relevant candidate video segments; at the same time, considering the semantic difference between the prompt in the candidate video segment retrieved from the media asset and the retrieval prompt, the part to be replaced and the corresponding replacement part are extracted from the existing candidate video segment, so as to replace the part to be replaced and the replacement part, and then generate a new video after replacement as the target video segment expected by the user in the case that no completely consistent video segment is retrieved.

[0113] Embodiment 2

[0114] In addition, the embodiment of the disclosure provides a video cross-modal retrieval device 900, please see Figure 9 The device comprises:

[0115] The enhancement module 901 is configured to perform prompt word enhancement on the user query text by using a pre-trained embedding model to obtain retrieval text.

[0116] The retrieval module 902 is configured to perform cross-modal retrieval on the retrieval text by using an audio-video large model to obtain a candidate video segment corresponding to the retrieval text.

[0117] The identification module 903 is configured to perform object identification on the candidate video segment according to the retrieval text by using an object identification deep model to obtain a to-be-replaced label and a replacement label.

[0118] The generation module 904 is configured to generate a mask according to the to-be-replaced label by using a mask decoder.

[0119] The adjustment module 905 is configured to adjust the candidate video segment according to the mask and the replacement label by using a text-to-video model to obtain a target video segment.

[0120] Optionally, the adjustment module 905 is further configured to extract a motion track of a to-be-replaced object corresponding to the to-be-replaced label from the candidate video segment by using the text-to-video model; and adjust the candidate video segment according to the mask, the replacement label and the motion track by using the text-to-video model to obtain the target video segment.

[0121] Optionally, the adjustment module 905 is further configured to identify an adjustment region in the candidate video segment according to the mask by using the text-to-video model; and generate a replacement object in the adjustment region according to the replacement label and the motion track by using the text-to-video model to obtain the target video segment.

[0122] Optionally, the enhancement module 901 is further configured to input an audio-video knowledge base into the pre-trained embedding model for encoding to obtain a plurality of cross-modal representation vectors; input the user query text into the pre-trained embedding model for encoding to obtain a text vector; obtain a target retrieval vector closest to the text vector from the cross-modal representation vectors by cross-modal retrieval; and determine the retrieval text according to the target retrieval vector.

[0123] Optionally, the identification module 903 is further configured to perform word segmentation on the search text to obtain a plurality of search text labels; embed each search text label and the candidate video segment into the object recognition deep model; perform object recognition on the candidate video segment according to each search text label through the object recognition deep model to obtain a plurality of image labels; and compare each search text label with each image label to obtain the to-be-replaced label and the replacement label.

[0124] Optionally, the generation module 904 is further configured to map the to-be-replaced label to a unique identification vector according to a predefined vocabulary table; find a matrix row vector with the shortest distance to the unique identification vector from an embedding matrix; embed the candidate video segment into the mask decoder through a visual encoder; and generate the mask through the mask decoder according to the candidate video segment and the matrix row vector.

[0125] Optionally, the generation module 904 is further configured to extract an image feature corresponding to the matrix row vector in the candidate video segment through cross attention in the mask decoder; and generate the mask according to the image feature through a fully connected convolutional network.

[0126] The apparatus provided by the embodiments of the present disclosure can perform the steps of the video cross-modal retrieval method provided in Embodiment 1, and thus will not be described again.

[0127] The video cross-modal retrieval apparatus provided by the embodiments can perform the steps of the video cross-modal retrieval method provided in Embodiment 1, and thus will not be described again.

[0128] Embodiment 3

[0129] In addition, the disclosure embodiment provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to realize the video cross-modal retrieval method in embodiment 1.

[0130] The device provided by the disclosure embodiment can execute the steps of the video cross-modal retrieval method provided in embodiment 1, and details are not repeated.

[0131] Embodiment 4

[0132] The disclosure embodiment provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the video cross-modal retrieval method in embodiment 1.

[0133] In the embodiment, the computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0134] The computer readable storage medium provided by the embodiment can realize the video cross-modal retrieval method provided in embodiment 1, and details are not repeated.

[0135] In all examples shown and described herein, any specific values should be interpreted as merely exemplary and not as a limitation, and thus, other examples of the example embodiments can have different values.

[0136] It should be noted that: similar labels and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0137] The above-described embodiments only express several embodiments of the present application, which are described in detail and specifically, but should not be understood as a limitation on the scope of the present application. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are within the scope of protection of the present application.

Claims

1. A method for video cross-modal retrieval, characterized in that, The method comprises the following steps: prompt word enhancement is performed on the user query text by a pre-training embedding model to obtain retrieval text; cross-modal retrieval is performed on the retrieval text by an audio-video large model to obtain a candidate video segment corresponding to the retrieval text; object recognition is performed on the candidate video segment according to the retrieval text by an object recognition deep model to obtain a to-be-replaced label and a replacement label; a mask is generated according to the to-be-replaced label by a mask decoder; the candidate video segment is adjusted according to the mask and the replacement label by a text-to-video model to obtain a target video segment; the method of adjusting the candidate video segment according to the mask and the replacement label by the text-to-video model to obtain the target video segment comprises the following steps: motion trajectories of a to-be-replaced object corresponding to the to-be-replaced label are extracted from the candidate video segment by the text-to-video model; the candidate video segment is adjusted according to the mask, the replacement label and the motion trajectories by the text-to-video model to obtain the target video segment; the method of adjusting the candidate video segment according to the mask, the replacement label and the motion trajectories by the text-to-video model to obtain the target video segment comprises the following steps: an adjustment region in the candidate video segment is identified according to the mask by the text-to-video model; a replacement object is generated in the adjustment region according to the replacement label and the motion trajectories by the text-to-video model to obtain the target video segment.

2. The video cross-modal retrieval method of claim 1, wherein, the method of performing prompt word enhancement on the user query text by a pre-training embedding model to obtain retrieval text comprises the following steps: an audio-video knowledge base is input into the pre-training embedding model for encoding to obtain a plurality of cross-modal representation vectors; the user query text is input into the pre-training embedding model for encoding to obtain a text vector; a target retrieval vector closest to the text vector is obtained from each of the cross-modal representation vectors by cross-modal retrieval; the retrieval text is determined according to the target retrieval vector.

3. The video cross-modal retrieval method of claim 1, wherein, the method of performing object recognition on the candidate video segment according to the retrieval text by an object recognition deep model to obtain a to-be-replaced label and a replacement label comprises the following steps: the retrieval text is segmented to obtain a plurality of retrieval text labels; each of the retrieval text labels and the candidate video segment is embedded into the object recognition deep model; object recognition is performed on the candidate video segment according to each of the retrieval text labels by the object recognition deep model to obtain a plurality of image labels; the to-be-replaced label and the replacement label are obtained by comparing each of the retrieval text labels and each of the image labels.

4. The video cross-modal retrieval method of claim 1, characterized in that, the method of generating a mask according to the to-be-replaced label by a mask decoder comprises the following steps: the to-be-replaced label is mapped to a unique identification vector according to a predefined vocabulary table; a matrix row vector with the shortest distance from the unique identification vector is found from an embedding matrix; the candidate video segment is embedded into the mask decoder by a visual encoder; the mask is generated according to the candidate video segment and the matrix row vector by the mask decoder.

5. The video cross-modal retrieval method of claim 4, characterized in that, The generating the mask by the mask decoder according to the candidate video clip and the matrix row vector comprises: extracting image features corresponding to the matrix row vector in the candidate video clip by cross attention in the mask decoder; generating the mask according to the image features by a fully connected convolutional network.

6. A video cross-modal retrieval apparatus, characterized by comprising: The video cross-modal retrieval method of claim 1 comprises: an enhancement module configured to perform prompt enhancement on user query text by a pre-trained embedding model to obtain retrieval text; a retrieval module configured to perform cross-modal retrieval on the retrieval text by an audio-video large model to obtain candidate video clips corresponding to the retrieval text; an identification module configured to perform object identification on the candidate video clips according to the retrieval text by an object identification deep model to obtain to-be-replaced labels and replacement labels; a generation module configured to generate a mask according to the to-be-replaced labels by a mask decoder; an adjustment module configured to adjust the candidate video clips according to the mask and the replacement labels by a text-to-video model to obtain target video clips.

7. A computer device, comprising: A computer program product comprising a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to implement the video cross-modal retrieval method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, A computer program product comprising a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to implement the video cross-modal retrieval method of any one of claims 1 to 5.

Images