Method and device for determining multi-modal retrieval model and electronic equipment

Abstract

The invention discloses a method and a device for determining a multi-modal retrieval model and electronic equipment. The method comprises the following steps: acquiring video data and text data for training a multi-modal retrieval model; feature extraction processing is conducted on the video data through a video encoder in the multi-mode retrieval model, video representation of the video data is obtained, and the video representation comprises an abstract token, a local prompt token and a global prompt token; carrying out encoding processing on the text data through a text encoder in the multi-modal retrieval model to obtain text representation; and optimizing the multi-modal retrieval model according to the similarity between the video representation and the text representation to obtain a target multi-modal retrieval model. The technical problem that the performance is reduced or text and visual information cannot be fully utilized due to the fact that the video field cannot be effectively adapted while the CLIP zero sample generalization ability is maintained in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and more specifically, to a method, device and electronic device for determining a multimodal retrieval model. Background Art

[0002] Text-video retrieval is a technology that queries video databases for relevant video clips based on text descriptions. It aims to use text-video retrieval models to more accurately match relevant text and video pairs. Currently, there are many challenges and problems when applying pre-trained graph-text models to text-video retrieval tasks. These include the difficulty of obtaining large-scale video-text pair data, the existence of multimodal feature alignment problems, and the computational complexity of processing multiple frames of data in videos. In addition, while maintaining the zero-shot generalization capability of CLIP, related technologies often fail to effectively adapt to the video field, resulting in performance degradation or failure to fully utilize text and visual information.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method, device and electronic device for determining a multimodal retrieval model, so as to at least solve the technical problem in the related art that while maintaining the CLIP zero-sample generalization capability, it is often unable to effectively adapt to the video field, resulting in performance degradation or inability to fully utilize text and visual information.

[0005] According to one aspect of an embodiment of the present application, a method for determining a multimodal retrieval model is provided, comprising: acquiring video data and text data for training the multimodal retrieval model; performing feature extraction processing on the video data through a video encoder in the multimodal retrieval model to obtain a video representation of the video data, wherein the video representation includes a summary token, a local prompt token, and a global prompt token, the summary token is used to represent identification information between each frame of video data and other frames of video data, the local prompt token is a learnable vector at the frame level, and the global prompt token is a learnable vector at the video level; encoding the text data through a text encoder in the multimodal retrieval model to obtain a text representation; optimizing the multimodal retrieval model based on the similarity between the video representation and the text representation to obtain a target multimodal retrieval model.

[0006] Optionally, after obtaining the video data and text data for training the multimodal retrieval model, the method also includes: dividing the video data to obtain vector data composed of patches of preset shapes; projecting the vector data from a first dimension to a second dimension through a first linear projection layer to obtain token data; determining an initial token sequence for each frame of video data based on the token data, wherein the initial token sequence also includes classification tokens, spatial position codes, and temporal position codes, and the classification tokens are used to represent key features in each frame of video data.

[0007] Optionally, feature extraction processing is performed on the video data through a video encoder in a multimodal retrieval model to obtain a video representation of the video data, including: obtaining an initial token sequence for each frame of video data; inputting the initial token sequence into the video encoder to obtain a first frame sequence output by each layer of the video encoder; determining a first summary token and a first local prompt token for each frame of video data in a second layer of the video encoder based on the first frame sequence output by the first layer of the video encoder, wherein the first layer of the video encoder is any layer of the video encoder, and the second layer of the video encoder is the next layer of the first layer of the video encoder; processing the first data sequence corresponding to each layer of the video encoder including the first frame sequence, the first summary token, the first local prompt token, and the global prompt token to obtain a processed target data sequence corresponding to the second layer of the encoder; extracting a target classification token from the target data sequence corresponding to the last layer of the video encoder; projecting the target classification token from the second dimension to the third dimension through a second linear projection layer to obtain a frame representation of each frame of video data by the video encoder; and performing average pooling on the frame representations of all video frame data to obtain a video representation of the video data.

[0008] Optionally, the first summary token of each frame of video data in the second layer video encoder is determined in the following manner: the first classification token in the first frame sequence is processed by a third linear projection layer to obtain a second classification token; the normalized second classification token is processed by a multi-head self-attention mechanism to obtain a third classification token; based on the third classification token and the second classification token, the first summary token of each frame of video data in the second layer video encoder is obtained.

[0009] Optionally, the first local prompt token of each frame of video data in the second layer video encoder is determined by: obtaining an initial local prompt mark corresponding to each frame of video data, and obtaining a first classification token in a first frame sequence corresponding to the first layer video encoder; based on the initial local prompt mark and the first classification token, determining the first local prompt token of each frame of video data in the second layer video encoder.

[0010] Optionally, the processed target data sequence corresponding to the second layer encoder is determined in the following manner: the normalized first data sequence is processed through frame-specific adaptation to obtain a second data sequence; a third data sequence is determined based on the first data sequence and the second data sequence; and a frame sequence in the third data sequence is processed through a feedforward network to obtain a target data sequence.

[0011] Optionally, the frame sequence in the third data sequence is processed through a feedforward network, including: processing the normalized frame sequence in the third data sequence through a feedforward network to obtain a second frame sequence; and obtaining a target frame sequence based on the second frame sequence and the frame sequence in the third data sequence.

[0012] According to another aspect of an embodiment of the present application, a device for determining a multimodal retrieval model is also provided, including: an acquisition module, used to acquire video data and text data for training the multimodal retrieval model; a first processing module, used to perform feature extraction processing on the video data through a video encoder in the multimodal retrieval model to obtain a video representation of the video data, wherein the video representation includes a summary token, a local prompt token and a global prompt token, the summary token is used to represent the identification information between each frame of video data and other frames of video data, the local prompt token is a learnable vector at the frame level, and the global prompt token is a learnable vector at the video level; a second processing module, used to perform encoding processing on the text data through a text encoder in the multimodal retrieval model to obtain a text representation; an optimization module, used to optimize the multimodal retrieval model according to the similarity between the video representation and the text representation to obtain a target multimodal retrieval model.

[0013] According to another aspect of an embodiment of the present application, there is also provided an electronic device, comprising: a memory and a processor, the memory being used to store program instructions; the processor being connected to the memory and being used to execute a method for determining the multimodal retrieval model.

[0014] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned method for determining the multimodal retrieval model by running the computer program.

[0015] According to another aspect of the embodiments of the present application, a computer program product is also provided, including computer instructions, which, when executed by a processor, implement the above-mentioned method for determining the multimodal retrieval model.

[0016] In an embodiment of the present application, video data and text data for training a multimodal retrieval model are obtained; a video encoder in the multimodal retrieval model performs feature extraction processing on the video data to obtain a video representation of the video data, wherein the video representation includes a summary token, a local prompt token, and a global prompt token, the summary token is used to represent the identification information between each frame of video data and other frames of video data, the local prompt token is a learnable vector at the frame level, and the global prompt token is a learnable vector at the video level; the text data is encoded by a text encoder in the multimodal retrieval model to obtain a text representation; the multimodal retrieval model is optimized according to the similarity between the video representation and the text representation to obtain a target multimodal retrieval model, thereby achieving the purpose of improving the accuracy of video-text retrieval, thereby achieving the technical effect of enhancing the flexibility and adaptability of the multimodal retrieval model, and further solving the technical problem in the related art that while maintaining the CLIP zero-sample generalization capability, it is often unable to effectively adapt to the video field, resulting in performance degradation or inability to fully utilize text and visual information. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 It is a hardware structure block diagram of a computer terminal for implementing a method for determining a multimodal retrieval model according to an embodiment of the present application;

[0019] Figure 2 is a flowchart of a method for determining a multimodal retrieval model according to an embodiment of the present application;

[0020] Figure 3 It is a structural diagram of a device for determining a multimodal retrieval model according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or reject automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0024] First, some nouns or terms that appear in the process of explaining the embodiments of the present application are subject to the following explanations:

[0025] Multimodal Prompt Methods: Learn specific context vectors between different modalities (e.g., images and text) to improve the performance and generalization ability of the model.

[0026] Text-Video Adaptive CLIP: A video and text adaptive CLIP model that achieves effective adaptation between video and text through a multimodal cueing approach, balancing the performance of supervised learning and zero-shot learning.

[0027] Temporal cues: The temporal information contained in the video, such as the relationship between frames, the order of actions, etc., helps the model understand the video content.

[0028] Frame-level visual representation: The visual feature representation of each frame in the video is used to represent the video content and dynamic information.

[0029] Text-video retrieval technology can be applied to the following aspects: ①Automated deployment monitoring: During the automated deployment process, the system may generate a large number of logs and operation videos. Through text-video retrieval technology, operation and maintenance personnel can quickly find video clips related to specific deployments, thereby speeding up problem diagnosis; ②Intelligent monitoring analysis: In an intelligent monitoring system, when the system detects abnormal behavior or performance degradation, text-video retrieval can help operation and maintenance personnel quickly locate the operation videos before and after the problem occurs based on the alarm information, so as to solve the problem faster.

[0030] However, while maintaining the CLIP zero-sample generalization capability, the related art often cannot effectively adapt to the video field, resulting in performance degradation or failure to fully utilize text and visual information. In order to solve the above problems, the embodiments of the present application provide corresponding solutions, which are described below.

[0031] The method for determining a multimodal retrieval model provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a method for determining a multimodal retrieval model. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0032] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0033] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for determining the multimodal retrieval model in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the method for determining the multimodal retrieval model described above is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0034] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0035] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0036] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0037] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a method for determining a multimodal retrieval model. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] Figure 2is a flow chart of a method for determining a multimodal retrieval model according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0039] Step S202, obtaining video data and text data for training a multimodal retrieval model.

[0040] In the above-mentioned step S202, the multimodal retrieval model can be, for example, a text-video retrieval model, and these video data and text data can be collected from different data sources through web crawlers, API interfaces or other tools. After obtaining the video data and text data, video preprocessing and text preprocessing can be performed, and data alignment is ensured to ensure that each video clip has a corresponding text description. In addition, the text description in the text data should be closely related to the video content and can accurately reflect the scenes, actions or events in the video. In order to allow the model to learn, the video data and text data need to be labeled or classified, such as manually labeling the categories or tags of the video and text data, or automatically performing preliminary labeling through some known classifiers, and the labeling process should be as detailed and accurate as possible to cover different scenes, actions and events.

[0041] Step S204, performing feature extraction processing on the video data through the video encoder in the multimodal retrieval model to obtain a video representation of the video data, wherein the video representation includes a summary token, a local hint token, and a global hint token. The summary token is used to represent the identification information between each frame of video data and other frames of video data. The local hint token is a learnable vector at the frame level, and the global hint token is a learnable vector at the video level.

[0042] In the above step S204, the role of the summary token is to summarize and represent the identification information between each frame and other frames in the video. At each layer of the video encoding, the classification token of each frame is processed by a linear projection operation, and then multi-head self-attention (MHSA) is applied between these frame-level tokens to capture the relationship between frames. In this way, the summary token can capture the dynamic changes and time clues within the video, such as the movement of objects, the transition of scenes, etc., which is crucial for understanding the coherence of the video and the order of events. Local prompt tokens are learnable vectors at the frame level, which are equal to the number of frames in the clip during training (such as the number of frames of video data). They are conditioned on their respective classification tags, which means that the local prompt tokens of each frame of the video are updated and adjusted according to the specificity of the frame. The main goal of the local prompt token is to convey the identification information of each frame to other frames in the video, promote the flow of information between frames, and enhance the model's ability to understand the local features of the video. By conditioning the [CLS] token, the local prompt token of each frame can reflect more specific frame-level visual information, thereby improving the fineness of the video representation. The Global Prompt Token is a video-level learnable vector that is used to provide additional learning capabilities and enable the model to adapt to the overall distribution of the video dataset. The Global Prompt Token helps the model capture the global characteristics of the video, such as the overall theme, atmosphere, or style of the video, which is critical for understanding the high-level semantics of the video. By introducing the Global Prompt Token in each layer, the model can better learn the abstract representation of the video data and improve the generalization performance on different video categories.

[0043] The summary token, local hint token, and global hint token together constitute the video representation output by the video encoder. The summary token captures the dynamic connection between frames, the local hint token focuses on the specificity of the frame, and the global hint token focuses on the macro characteristics of the video. This multi-level representation learning strategy not only improves the model's ability to parse video data, but also enhances its performance in text-video retrieval tasks, especially when dealing with complex and diverse video categories. It can more accurately match text descriptions with video content and achieve a balance between supervised learning and zero-shot learning.

[0044] Step S206, encoding the text data through the text encoder in the multimodal retrieval model to obtain a text representation.

[0045] In the above step S206, the text data needs to be converted into a format that the model can process. For example, the text is divided into words or subwords (tokens) and a numerical ID is assigned to each token. In addition, in order to maintain the structural information of the sentence, each token is also attached with a position code, which is a numerical representation indicating the position of the token in the sentence, which helps the model understand grammatical and sequential information. After obtaining the token IDs, they are sent to the embedding layer. The embedding layer converts each ID into a vector of fixed dimension, which contains the semantic information of the token. For example, a pre-trained text encoder, such as BERT, can generate a 512-dimensional semantic vector for each token through its embedding layer. The Multi-Head Self-Attention layer (MHSA) updates the representation of each token by calculating the relationship between all tokens, thereby capturing the global dependencies in the text. This mechanism enables the model to focus on different parts of the text, understand the context and handle long-distance dependencies. After the attention mechanism updates the token representation, these representations are usually further processed through a feed forward network (FFN) to enhance the expressive power of the model. FFN consists of two fully connected layers, the first layer maps the vector to a higher dimension, and the second layer maps it back to the original dimension. In order to speed up the training process and avoid the problem of gradient disappearance or explosion, the output of each layer is usually processed by layer normalization. This helps to maintain numerical stability and ensures that the vectors output by each layer have similar scale and distribution. After multiple layers of encoding and processing, the text encoder generates a richer and more abstract text representation.

[0046] In order to make the pre-trained text encoder better adapt to specific tasks (such as video description), additional learnable hints can be introduced. These hints can be seen as part of the model to guide the text encoder on how to understand and process the input, especially when dealing with new categories of video descriptions. By training these hints, the model can more effectively combine textual information with visual information and improve the accuracy of cross-modal retrieval.

[0047] Step S208, optimizing the multimodal retrieval model according to the similarity between the video representation and the text representation to obtain a target multimodal retrieval model.

[0048] In the above step S208, for each pair of video and text, the model calculates the cosine similarity between their embedded representations. Cosine similarity is a metric that calculates the cosine of the angle between two vectors, thereby reflecting their similarity in a multidimensional space. A high similarity score indicates that the video and text are semantically closer. A loss function is designed to maximize the similarity of positive sample pairs (i.e., the video and its correct text description) while minimizing the similarity of negative sample pairs (i.e., the video and irrelevant text description). This is usually in the form of contrastive loss, ensuring that similar samples within the same modality are close in the embedding space, while samples of different modalities or irrelevant ones are kept at a greater distance. Through optimization techniques such as backpropagation and gradient descent, the model parameters are adjusted to minimize this loss function, thereby training the model's performance on the multimodal retrieval task. During the training process, the model will learn how to establish a stronger association between the video representation and the text representation, and can effectively retrieve even on new data that has not been seen. After training and optimization in the above steps, the resulting model is the target multimodal retrieval model. This model can find the most relevant video clips in the video library based on text queries, or conversely, find the most relevant text description based on the video content, thereby achieving efficient and accurate text-video retrieval.

[0049] In step S202 of the above-mentioned method for determining the multimodal retrieval model, after obtaining the video data and text data used to train the multimodal retrieval model, the method also includes: dividing the video data to obtain vector data composed of patches of preset shapes; projecting the vector data from a first dimension to a second dimension through a first linear projection layer to obtain token data; determining an initial token sequence for each frame of video data based on the token data, wherein the initial token sequence also includes classification tokens, spatial position codes, and temporal position codes, and the classification tokens are used to represent key features in each frame of video data.

[0050] In the embodiments of the present application, the above description is explained through the following process:

[0051] 1. Video data patch division:

[0052] First, each frame of video image is divided into multiple patches of preset shapes. These patches are usually squares of equal size, which ensures that the amount of visual information contained in each patch is roughly the same, thus maintaining the consistency and comparability of information in subsequent processing.

[0053] Specifically, in the embodiment of the present application, the video data V∈R T×H×W×3 , the spatial size is H×W, H represents the height, W represents the width, and the number of sampling frames is T. Each frame t∈{1...T} is divided into N non-overlapping square patches of size P×P, and the total number of patches is N=H×W / P2 For each frame, all patches of shape P×P×3 are flattened into a set of vectors, represented as That is the above vector data, where t is the number of frames and i is the number of patches.

[0054] 2. Linear projection:

[0055] In order to make the video data be represented in the same semantic space as the text data, a linear projection layer is needed to project the vector data of each patch from its original dimension (the first dimension) to another dimension (the second dimension), which usually matches the embedding dimension of the model. The role of the linear projection layer is to adjust the representation of the data to make it more suitable for subsequent multimodal fusion and contrastive learning.

[0056] Specifically, a linear projection layer is used (i.e. the first linear projection layer mentioned above) projects the vector into a token (or token data) embedding, and the output dimension of each token is D (i.e. the second dimension mentioned above). A classification token x is appended to the embedded token sequence of each frame cls , x cls ∈R D .

[0057] 3. Construct the initial token sequence:

[0058] The initial token sequence for each frame of data fed into the video encoder Depend on:

[0059]

[0060] e=e sp +e tm ,

[0061] Among them, e sp and e tm Represent spatial position coding and temporal position coding, respectively. Represents the above token data.

[0062] In step S204 of the method for determining the multimodal retrieval model, the video data is subjected to feature extraction processing by a video encoder in the multimodal retrieval model to obtain a video representation of the video data, including: obtaining an initial token sequence for each frame of video data; inputting the initial token sequence into the video encoder to obtain a first frame sequence output by each layer of the video encoder; determining a first summary token and a first local prompt token for each frame of video data in a second layer of the video encoder according to the first frame sequence output by the first layer of the video encoder, wherein the first layer of the video encoder is any layer of the video encoder, and the second layer of the video encoder is a layer of the encoder next to the first layer of the video encoder; processing the first data sequence corresponding to each layer of the video encoder including the first frame sequence, the first summary token, the first local prompt token, and the global prompt token to obtain a processed target data sequence corresponding to the second layer of the encoder; extracting a target classification token from the target data sequence corresponding to the last layer of the video encoder; projecting the target classification token from the second dimension into the third dimension by a second linear projection layer to obtain a frame representation of each frame of video data by the video encoder; and performing average pooling on the frame representations of all video frame data to obtain a video representation of the video data.

[0063] In the above steps, the first summary token of each frame of video data in the second layer video encoder is determined by: processing the first classification token in the first frame sequence through the third linear projection layer to obtain the second classification token; processing the normalized second classification token through the multi-head self-attention mechanism to obtain the third classification token; and obtaining the first summary token of each frame of video data in the second layer video encoder based on the third classification token and the second classification token.

[0064] In the above steps, the first local prompt token of each frame of video data in the second layer video encoder is determined by the following method: obtaining the initial local prompt mark corresponding to each frame of video data, and obtaining the first classification token in the first frame sequence corresponding to the first layer video encoder; based on the initial local prompt mark and the first classification token, determining the first local prompt token of each frame of video data in the second layer video encoder.

[0065] In the above steps, the processed target data sequence corresponding to the second layer encoder is determined in the following manner: the normalized first data sequence is processed by frame-specific adaptation to obtain a second data sequence; a third data sequence is determined based on the first data sequence and the second data sequence; and a frame sequence in the third data sequence is processed by a feedforward network to obtain a target data sequence.

[0066] In the above steps, the frame sequence in the third data sequence is processed through a feedforward network, including: processing the normalized frame sequence in the third data sequence through a feedforward network to obtain a second frame sequence; and obtaining a target frame sequence based on the second frame sequence and the frame sequence in the third data sequence.

[0067] In the embodiment of the present application, from L v In the layered video encoder, the first frame sequence of each layer l is obtained as follows:

[0068]

[0069] in, It is the lth layer of the video encoder.

[0070] In each layer of the video encoder, a summary token is introduced to summarize the identification information between all frames, and the local hint token at the T frame level is used to convey the identification information of each frame to the rest of the frames in the video data, and M v The global hint token at the video level provides learning capability to adapt the model to the distribution of the video dataset. The summary token is used to summarize the discriminative information from each frame in the video data and feed it back to each frame, and then the pre-trained self-attention is applied for this layer. More specifically, the summary token of the tth frame at layer l (i.e., the first summary token in the second layer video encoder of each frame of video data) is obtained by (i.e., the first classification token in the first frame sequence above) is firstly applied with a linear projection P sum (i.e. the third linear projection layer mentioned above), and then apply the MHSA operation between these frame-level tokens to obtain:

[0071]

[0072] in, And LN stands for layer normalization. represents the second classification token mentioned above, represents the third classification token mentioned above, and then the corresponding summary token is appended to the frame sequence before applying the frozen pre-trained self-attention of this layer middle.

[0073] Global hint token at the video level are randomly initialized learnable vectors. They are used to provide additional learning power to the model to learn the data distribution.

[0074] Frame-level local hint markers are also randomly initialized learnable vectors equal to the number of frames T in the video data during training, but they are conditioned on their respective classification labels. [CLS] tokens for L(l) layers The conditionalization enables a top-down discriminative information flow in the frame-level learnable tag. The first local hint token of each frame of video data in the l-th layer video encoder (i.e., the second layer video encoder mentioned above) is defined as:

[0075]

[0076] in, represents the initial local hint mark, represents the first classification token, Represents the first local hint token mentioned above.

[0077] Finally, the token and Add to each frame sequence Then the layer is frozen with pre-trained self-attention.

[0078] The first data sequence is The first data sequence is processed by the following formula to obtain the third data sequence:

[0079]

[0080] Among them, FSA stands for frame specific adaptation, represents the second data sequence mentioned above, Represents the above-mentioned third data sequence.

[0081] Finally, remove the extra token and only Apply a feed-forward network as shown below:

[0082]

[0083] in, represents a frame sequence in the third data sequence, represents the target data sequence, which can also be understood as the sequence obtained after processing the first frame sequence. Represents the second frame sequence.

[0084] From the last layer Extract the target classification token x from the output target data sequence cls , and use a linear projection layer (i.e. the second linear projection layer mentioned above) P out ∈R D×D' Projecting to a dimension D' (i.e. the third dimension mentioned above), we get the frame representation of each frame of video data by the video encoder:

[0085]

[0086] where v t is the output representation of frame t, is the target classification token of the target data sequence from the last layer of the video encoder. To obtain the video representation, simply represent each frame v t Perform average pooling to obtain the aggregate representation:

[0087] v=AvgPool([v1,...,v T ]).

[0088] Among them, v represents the above video representation.

[0089] For the input text data, a pre-trained text encoder is used, combined with an additional text hint learning method. The pre-trained text encoder is a 12-layer BERT model (for CLIP B / 16 variant) with an embedding size of 512 and a context length of 77. Each layer of the model contains a multi-head self-attention (MHSA) followed by a feed-forward network. Given a text description C of a video, the text encoder is used to obtain the representation

[0090] In the process of learning text encoding prompts, trainable vectors are used to model context words. The input of the text encoder is a sequence of tokens of the form:

[0091]

[0092] Among them, u i ,i∈{1,...,M c} is a trainable vector of the same size as the input vector of the text encoder, M c is the number of uniform cues that can be trained. This token sequence is then passed to a text encoder, which produces a text embedding c = f θ (C).

[0093] In the embodiment of the present application, given a set of videos and a set of text descriptions, the descriptions are transmitted to the videos respectively. and text Encoder. In video and text representations are given as follows:

[0094]

[0095] Define the cosine similarity loss function L between video and text representations cos as follows:

[0096]

[0097] The above loss function is used to maximize the similarity between positive sample pairs (i.e., videos and their correct text descriptions) and minimize the similarity between negative sample pairs (i.e., videos and irrelevant text descriptions), ensuring that similar samples in the same modality are close in the embedding space, while samples in different modalities or irrelevant samples are kept far away. That is, the ultimate goal is to maximize the L of the true v and c pairs. cos , and minimize the L of other pairs cos , thereby optimizing the multimodal retrieval model based on the similarity between video representation and text representation.

[0098] Through optimization techniques such as back propagation and gradient descent, the model parameters are adjusted to minimize this loss function, thereby training the model's performance on multimodal retrieval tasks. During the training process, the model will learn how to establish stronger associations between video representations and text representations, and can effectively retrieve even new data that has never been seen. After the training and optimization of the above steps, the resulting model is the target multimodal retrieval model. This model can find the most relevant video clips in the video library based on text queries, or conversely, find the most relevant text description based on the video content, thereby achieving efficient and accurate text-video retrieval.

[0099] The method for determining the multimodal retrieval model provided in the embodiment of the present application learns specific context vectors in the video and text fields through the video and text adaptive CLIP multimodal prompting method to effectively adapt the image-text pre-trained CLIP model to the video recognition task. This method helps the model to better understand the correlation between video and text, improves the performance and generalization ability of the model, and brings significant performance improvement in the field of text-video retrieval. This method not only achieves outstanding results in supervision and zero-sample learning, eliminates traditional performance trade-offs, but also provides new possibilities for multimodal learning and understanding, not only captures the detailed information in the video data set, but also improves the model's ability to understand complex category descriptions. In addition, the time clues and data distribution of the video are clearly modeled in terms of vision, and the identification information is aggregated between each frame through the prompt learning method, giving each layer of learning ability to better capture data distribution. This method of time clue modeling helps to improve the model's understanding and classification accuracy of video sequences.

[0100] Figure 3 is a structural diagram of a device for determining a multimodal retrieval model according to an embodiment of the present application, such as Figure 3 As shown, the device comprises:

[0101] An acquisition module 30, used to acquire video data and text data for training a multimodal retrieval model;

[0102] A first processing module 32 is used to perform feature extraction processing on the video data through a video encoder in a multimodal retrieval model to obtain a video representation of the video data, wherein the video representation includes a summary token, a local hint token, and a global hint token, the summary token is used to represent the identification information between each frame of video data and other frames of video data, the local hint token is a frame-level learnable vector, and the global hint token is a video-level learnable vector;

[0103] A second processing module 34 is used to encode the text data through a text encoder in the multimodal retrieval model to obtain a text representation;

[0104] The optimization module 36 is used to optimize the multimodal retrieval model according to the similarity between the video representation and the text representation to obtain a target multimodal retrieval model.

[0105] In the acquisition module in the determination device of the above-mentioned multimodal retrieval model, the acquisition module is also used to divide the video data to obtain vector data composed of patches of preset shapes; project the vector data from the first dimension to the second dimension through the first linear projection layer to obtain token data; determine the initial token sequence of each frame of video data based on the token data, wherein the initial token sequence also includes classification tokens, spatial position codes and temporal position codes, and the classification tokens are used to represent key features in each frame of video data.

[0106] In the first processing module in the determination device of the above-mentioned multimodal retrieval model, the first processing module is used to obtain an initial token sequence for each frame of video data; input the initial token sequence into the video encoder to obtain the first frame sequence output by each layer of the video encoder; determine the first summary token and the first local prompt token of each frame of video data in the second layer of the video encoder according to the first frame sequence output by the first layer of the video encoder, wherein the first layer of the video encoder is any layer of the video encoder, and the second layer of the video encoder is the next layer of the first layer of the video encoder; process the first data sequence corresponding to each layer of the video encoder including the first frame sequence, the first summary token and the first local prompt token, and the global prompt token to obtain a processed target data sequence corresponding to the second layer of the encoder; extract the target classification token from the target data sequence corresponding to the last layer of the video encoder; project the target classification token from the second dimension to the third dimension through the second linear projection layer to obtain the frame representation of each frame of video data by the video encoder; average pool the frame representations of all video frame data to obtain the video representation of the video data.

[0107] In the first processing module in the determination device of the above-mentioned multimodal retrieval model, the first processing module is also used to determine the first summary token of each frame of video data in the second layer video encoder. Specifically, the first summary token of each frame of video data in the second layer video encoder is determined by: processing the first classification token in the first frame sequence through the third linear projection layer to obtain the second classification token; processing the normalized second classification token through the multi-head self-attention mechanism to obtain the third classification token; and obtaining the first summary token of each frame of video data in the second layer video encoder based on the third classification token and the second classification token.

[0108] In the first processing module in the determination device of the above-mentioned multimodal retrieval model, the first processing module is also used to determine the first local prompt token of each frame of video data in the second layer video encoder. Specifically, the first local prompt token of each frame of video data in the second layer video encoder is determined in the following manner: obtaining the initial local prompt mark corresponding to each frame of video data, and obtaining the first classification token in the first frame sequence corresponding to each layer of video encoder; based on the initial local prompt mark and the first classification token, determining the first local prompt token of each frame of video data in the second layer video encoder.

[0109] In the first processing module in the determination device of the above-mentioned multimodal retrieval model, the first processing module is also used to determine the processed target data sequence corresponding to the second layer encoder. Specifically, the processed target data sequence corresponding to the second layer encoder is determined in the following manner: the normalized first data sequence is processed through frame-specific adaptation to obtain a second data sequence; based on the first data sequence and the second data sequence, a third data sequence is determined; and the frame sequence in the third data sequence is processed through a feedforward network to obtain a target data sequence.

[0110] In the first processing module in the determination device of the above-mentioned multimodal retrieval model, the first processing module is also used to process the frame sequence in the normalized third data sequence through a feedforward network to obtain a second frame sequence; and obtain a target frame sequence based on the second frame sequence and the frame sequence in the third data sequence.

[0111] It should be noted that Figure 3 The multimodal retrieval model determination device shown is used to perform Figure 2 The method for determining the multimodal retrieval model shown in the figure, therefore the relevant explanations and instructions in the above-mentioned method for determining the multimodal retrieval model are also applicable to the device for determining the multimodal retrieval model, and will not be repeated here.

[0112] An embodiment of the present application also provides an electronic device, including: a memory and a processor, the memory being used to store program instructions; the processor being connected to the memory and being used to execute a method for determining the multimodal retrieval model.

[0113] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned method for determining the multimodal retrieval model by running the computer program.

[0114] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the method for determining a multimodal retrieval model in each embodiment of the present application.

[0115] An embodiment of the present application also provides a computer program, which, when executed by a processor, implements the steps of the method for determining a multimodal retrieval model in each embodiment of the present application.

[0116] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0117] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0118] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0120] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0121] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0122] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for determining a multimodal retrieval model, characterized in that: include: Obtain video data and text data for training a multimodal retrieval model; Performing feature extraction processing on the video data through a video encoder in the multimodal retrieval model to obtain a video representation of the video data, wherein the video representation includes a summary token, a local prompt token, and a global prompt token, the summary token is used to represent identification information between each frame of video data and other frames of video data, the local prompt token is a frame-level learnable vector, and the global prompt token is a video-level learnable vector; The text data is encoded by a text encoder in the multimodal retrieval model to obtain a text representation; The multimodal retrieval model is optimized according to the similarity between the video representation and the text representation to obtain a target multimodal retrieval model.

2. The method according to claim 1, characterized in that After acquiring the video data and text data for training the multimodal retrieval model, the method further includes: Dividing the video data to obtain vector data composed of patches of a preset shape; Projecting the vector data from the first dimension into the second dimension through a first linear projection layer to obtain token data; An initial token sequence for each frame of video data is determined based on the token data, wherein the initial token sequence also includes a classification token, a spatial position code, and a temporal position code, and the classification token is used to represent a key feature in each frame of video data.

3. The method according to claim 2, characterized in that The method further comprises: performing feature extraction processing on the video data by using a video encoder in the multimodal retrieval model to obtain a video representation of the video data, including: Get the initial token sequence of each frame of video data; Inputting the initial token sequence into the video encoder to obtain a first frame sequence output by each layer of the video encoder; Determine, according to a first frame sequence output by a first-layer video encoder, a first summary token and a first local prompt token of each frame of video data in a second-layer video encoder, wherein the first-layer video encoder is an encoder of any layer among video encoders, and the second-layer video encoder is an encoder of a layer below the first-layer video encoder; Processing a first data sequence corresponding to each layer of the video encoder including the first frame sequence, the first summary token, the first local prompt token, and the global prompt token to obtain a processed target data sequence corresponding to the second layer of the encoder; Extracting target classification tokens from the target data sequence corresponding to the last layer of the video encoder; Projecting the target classification token from the second dimension into a third dimension through a second linear projection layer to obtain a frame representation of each frame of video data by the video encoder; Average pooling is performed on the frame representations of all video frame data to obtain a video representation of the video data.

4. The method according to claim 3, characterized in that The first summary token of each frame of video data in the second layer video encoder is determined by: Processing the first classification tokens in the first frame sequence through a third linear projection layer to obtain second classification tokens; The normalized second classification tokens are processed through a multi-head self-attention mechanism to obtain third classification tokens; A first summary token of each frame of video data in a second-layer video encoder is obtained according to the third classification token and the second classification token.

5. The method according to claim 3, characterized in that: The first local hint token of each frame of video data in the second layer video encoder is determined by: Obtaining an initial local prompt mark corresponding to each frame of video data, and obtaining a first classification token in a first frame sequence corresponding to a first layer of video encoder; A first local hint token for each frame of video data in a second layer video encoder is determined according to the initial local hint mark and the first classification token.

6. The method according to claim 3, characterized in that The processed target data sequence corresponding to the second layer encoder is determined by: Processing the normalized first data sequence through frame-specific adaptation to obtain a second data sequence; Determine a third data sequence according to the first data sequence and the second data sequence; The frame sequence in the third data sequence is processed through a feedforward network to obtain the target data sequence.

7. The method according to claim 6, characterized in that Processing the frame sequence in the third data sequence through a feedforward network includes: Processing the normalized frame sequence in the third data sequence through the feedforward network to obtain a second frame sequence; A target frame sequence is obtained according to the frame sequences in the second frame sequence and the third data sequence.

8. A device for determining a multimodal retrieval model, characterized in that: include: An acquisition module, used for acquiring video data and text data for training a multimodal retrieval model; A first processing module is used to perform feature extraction processing on the video data through a video encoder in the multimodal retrieval model to obtain a video representation of the video data, wherein the video representation includes a summary token, a local prompt token and a global prompt token, the summary token is used to represent the identification information between each frame of video data and other frames of video data, the local prompt token is a frame-level learnable vector, and the global prompt token is a video-level learnable vector; A second processing module, configured to encode the text data through a text encoder in the multimodal retrieval model to obtain a text representation; An optimization module is used to optimize the multimodal retrieval model according to the similarity between the video representation and the text representation to obtain a target multimodal retrieval model.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the method for determining the multimodal retrieval model described in any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the method for determining a multimodal retrieval model as described in any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method for determining a multimodal retrieval model described in any one of claims 1 to 7 is implemented.

Images