Video text combined retrieval method, device, electronic device and storage medium

Abstract

The present application discloses a video text combined retrieval method, device, electronic device and storage medium, and relates to the field of video retrieval technology. By obtaining the original video and the retrieval text, the original video frame is encoded to obtain high-level visual features and mid-level visual features, and the retrieval text is encoded to obtain retrieval text features. A high-level branch is set to extract high-level retention features in high-level vision and high-level difference features in retrieval text features based on temporal information, and fuse them to obtain high-level fusion features. A mid-level branch is set to use the attention mechanism to extract finer-grained spatiotemporal features based on mid-level retention features and mid-level difference features, and fuse them to obtain mid-level fusion features. Finally, the target fusion features are obtained by performing hierarchical multi-fusion on each feature, thereby retrieving the preset video library to obtain the target video. In this way, the user's visual needs are described from different granularities, which effectively improves the accuracy of video retrieval and accurately finds the target video that meets the user's needs.

Description

Technical Field

[0001] The present application relates to the field of video retrieval technology, and in particular to a video-text combined retrieval method, device, electronic device and storage medium. Background Art

[0002] With the explosive growth of video data, accurately finding videos that meet user needs is becoming increasingly difficult. Common video retrieval methods use user-entered query text to search for videos related to the query text within a large number of candidate videos. Alternatively, they combine multiple user-provided images to construct an image query that represents the user's needs. The query is then compared with images in the video to find videos with similar combinations of query images.

[0003] However, retrieval text can only convey a high-level explanation of user needs, but cannot describe the user's more fine-grained visual requirements. Moreover, compared to image data, video data has complex temporal and spatial variations. Image combination retrieval methods alone cannot effectively capture the temporal characteristics of videos, resulting in low video retrieval accuracy. Therefore, how to accurately find videos that meet user needs from a huge video library is an urgent problem to be solved. Summary of the Invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the embodiments of the present application provide a video-text combined retrieval method, device, electronic device, and storage medium, which can effectively improve the accuracy of video retrieval and accurately find target videos that meet user needs.

[0005] In a first aspect, an embodiment of the present application provides a video text combination retrieval method, comprising:

[0006] Acquire original video and retrieve text; wherein the original video includes multiple original video frames;

[0007] Encoding the original video frame using an image encoder to obtain high-level visual features and mid-level visual features, and encoding the search text using a text encoder to obtain search text features;

[0008] extracting high-level retained features from the high-level visual features according to the first weight information and the timing information of the original video frame, extracting high-level difference features from the retrieved text features according to the second weight information, and obtaining high-level fused features from the high-level retained features and the high-level difference features based on the fusion weight information;

[0009] Extracting mid-level retained features from the mid-level visual features using a convolutional neural network, extracting mid-level difference features from the retrieved text features based on third weight information, and obtaining mid-level fusion features based on the mid-level retained features and the mid-level difference features based on an attention mechanism;

[0010] Obtaining a target fusion feature according to the high-level visual feature, the retrieved text feature, the high-level fusion feature and the middle-level fusion feature;

[0011] The target video is obtained by searching a preset video library based on the target fusion feature.

[0012] In some embodiments of the present application, the image encoder includes M coding layers, where M is an integer greater than 1; and encoding the original video frame using the image encoder to obtain high-level visual features and mid-level visual features includes:

[0013] Each of the original video frames is encoded using the image encoder, and the output result of the Mth encoding layer is used as the high-level visual feature of the original video frame, and the output result of the M-1th encoding layer is used as the mid-level visual feature of the original video frame.

[0014] In some embodiments of the present application, extracting high-level retained features from the high-level visual features according to the first weight information and the timing information of the original video frame, extracting high-level difference features from the retrieved text features according to the second weight information, and obtaining high-level fused features from the high-level retained features and the high-level difference features based on the fusion weight information includes:

[0015] Inputting the high-level visual features into a temporal encoder for temporal encoding based on the temporal information to obtain temporal high-level visual features;

[0016] Inputting the temporal high-level visual features into a first multi-layer perceptron, obtaining a first linear feature based on the first weight information and the first bias information, mapping the first linear feature to a nonlinear space using a first activation function to obtain a first nonlinear feature, inputting the first nonlinear feature into a first regularization function for feature screening to obtain the high-level retained feature;

[0017] Inputting the search text feature into a second multilayer perceptron, obtaining a second linear feature based on the second weight information and the second bias information, mapping the second linear feature into a nonlinear space using a second activation function to obtain a second nonlinear feature, and inputting the second nonlinear feature into a second regularization function for feature screening to obtain the high-level difference feature;

[0018] The high-level retained features and the high-level difference features are input into a fused multi-layer perceptron, and fused linear features are obtained according to the fusion weight information and the fusion bias information. The fused linear features are mapped to a nonlinear space using a fusion activation function to obtain a fused nonlinear feature. The fused nonlinear feature is input into a fusion regularization function for feature screening to obtain the high-level fusion feature.

[0019] In some embodiments of the present application, the extracting of mid-level retained features from the mid-level visual features using a convolutional neural network, extracting mid-level difference features from the retrieved text features based on third weight information, and obtaining mid-level fusion features based on the mid-level retained features and the mid-level difference features based on an attention mechanism include:

[0020] Inputting the middle-layer visual features into a convolutional neural network to obtain middle-layer linear features, mapping the middle-layer linear features into a nonlinear space using a middle-layer activation function to obtain middle-layer nonlinear features, and inputting the middle-layer nonlinear features into a middle-layer regularization function for feature screening to obtain the middle-layer retained features;

[0021] Inputting the search text feature into a third multi-layer perceptron, obtaining a third linear feature based on the third weight information and the third bias information, mapping the third linear feature into a nonlinear space using a third activation function to obtain a third nonlinear feature, inputting the third nonlinear feature into a third regularization function for feature screening to obtain the middle-layer difference feature;

[0022] The middle-layer retained features and the middle-layer difference features are input into the attention mechanism, and the middle-layer retained features and the middle-layer difference features are spatiotemporally fused according to the retrieved text features to obtain the middle-layer fused features.

[0023] In some embodiments of the present application, the attention mechanism includes n attention heads, where n is an integer greater than 1; inputting the middle-level retained features and the middle-level difference features into the attention mechanism, and performing spatiotemporal fusion of the middle-level retained features and the middle-level difference features according to the retrieved text features to obtain the middle-level fused features, includes:

[0024] Splicing the middle layer retained features and the middle layer difference features to obtain middle layer splicing features;

[0025] The middle-layer splicing feature is subjected to the first linear transformation of the i-th attention head to obtain feature Ki, the middle-layer splicing feature is subjected to the second linear transformation of the i-th attention head to obtain feature Vi, and the retrieval text feature is subjected to the third linear transformation of the i-th attention head to obtain feature Qi; wherein i is an integer between [1, n];

[0026] Obtaining the attention feature of the i-th attention head according to the feature Ki, the feature Vi, the feature Qi, the number of attention heads n, and the feature dimension d of the middle-layer retained feature;

[0027] The attention features of each attention head are concatenated to obtain the middle-layer fusion features.

[0028] In some embodiments of the present application, obtaining the target fusion feature according to the high-level visual feature, the search text feature, the high-level fusion feature, and the mid-level fusion feature includes:

[0029] Inputting the high-level visual features and the search text features into a fourth multi-layer perceptron, obtaining a fourth linear feature based on fourth weight information and fourth bias information, and mapping the fourth linear feature to a nonlinear space using a fourth activation function to obtain a first intermediate weight of the high-level visual features and a second intermediate weight of the search text features;

[0030] Multiplying the first intermediate weight and the high-level visual feature to obtain a first intermediate result, and multiplying the second intermediate weight and the search text feature to obtain a second intermediate result;

[0031] The first intermediate result, the second intermediate result, the high-level fusion feature and the middle-level fusion feature are added together to obtain the target fusion feature.

[0032] In some embodiments of the present application, the preset video library includes a plurality of reference videos; and the step of retrieving a target video based on the target fusion feature includes:

[0033] Obtaining a reference video from the preset video library; wherein the reference video includes a plurality of reference video frames;

[0034] Encoding the reference video frame using the image encoder to obtain reference visual features;

[0035] Inputting the high-level visual feature into a temporal encoder for temporal encoding based on reference temporal information of the reference visual feature to obtain a temporal reference visual feature;

[0036] The similarity between the temporal reference visual feature and the target fusion feature is calculated, and if the similarity meets a preset similarity threshold, the reference video corresponding to the similarity is used as the target video.

[0037] In a second aspect, an embodiment of the present application further provides a video-text combined retrieval device, which applies the video-text combined retrieval method described in the embodiment of the first aspect of the present application, including:

[0038] An acquisition module, configured to acquire an original video and retrieve text; wherein the original video includes a plurality of original video frames;

[0039] An encoding module, configured to encode the original video frame using an image encoder to obtain high-level visual features and mid-level visual features, and to encode the search text using a text encoder to obtain search text features;

[0040] a high-level extraction module, configured to extract high-level retained features from the high-level visual features according to the first weight information and the timing information of the original video frame, extract high-level difference features from the retrieved text features according to the second weight information, and obtain high-level fused features from the high-level retained features and the high-level difference features based on the fusion weight information;

[0041] a middle-layer extraction module, configured to extract middle-layer retained features from the middle-layer visual features using a convolutional neural network, extract middle-layer difference features from the retrieved text features according to third weight information, and obtain middle-layer fusion features based on the middle-layer retained features and the middle-layer difference features based on an attention mechanism;

[0042] A hierarchical fusion module, which obtains a target fusion feature according to the high-level visual feature, the search text feature, the high-level fusion feature and the middle-level fusion feature;

[0043] The retrieval module searches a preset video library based on the target fusion feature to obtain the target video.

[0044] In a third aspect, an embodiment of the present application further provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the video text combination retrieval method as described in the embodiment of the first aspect of the present application is implemented.

[0045] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the video-text combination retrieval method as described in the embodiment of the first aspect of the present application.

[0046] The embodiments of the present application include at least the following beneficial effects:

[0047] Embodiments of the present application provide a video-text combined retrieval method, apparatus, electronic device, and storage medium. The method obtains an original video comprising multiple original video frames and a retrieval text, then uses an image encoder to encode the original video frames to obtain high-level visual features and mid-level visual features, and uses a text encoder to encode the retrieval text to obtain retrieval text features. High-level retained features are extracted from the high-level visual features based on first weight information and the temporal information of the original video frames, high-level difference features are extracted from the retrieval text features based on second weight information, and high-level fused features are obtained from the high-level retained features and high-level difference features based on fusion weight information. A high-level branch is thus configured to extract visual features that need to be retained in the high-level visual domain based on temporal information and to retrieve semantic features required to be included in the target video in this document. A convolutional neural network is then used to extract mid-level retained features from the mid-level visual features, mid-level difference features are extracted from the retrieval text features based on third weight information, and mid-level fused features are obtained from the mid-level retained features and mid-level difference features based on an attention mechanism. A mid-level branch is thus configured to extract finer-grained spatiotemporal features based on the mid-level retained features and mid-level difference features using an attention mechanism. Finally, the target fusion feature is derived from high-level visual features, retrieved text features, high-level fusion features, and mid-level fusion features. Based on the target fusion feature, the preset video library is retrieved to obtain the target video. By setting high-level and mid-level branches and extracting the retained features from the original video and the differential features from the retrieval paper respectively, the user's visual needs are described at different granularities. The attention mechanism is used to focus on spatiotemporal features to obtain the target fusion feature for retrieval, effectively improving the accuracy of video retrieval and accurately finding the target video that meets the user's needs.

[0048] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0050] Figure 1 This is a flowchart of a video-text combined retrieval method provided by one embodiment of the present application;

[0051] Figure 2 yes Figure 1 Flow diagram of step S103;

[0052] Figure 3 yes Figure 1 Flow chart of step S104;

[0053] Figure 4 yes Figure 3Flow diagram of step S303;

[0054] Figure 5 yes Figure 1 Flow diagram of step S105;

[0055] Figure 6 yes Figure 1 Flow chart of step S106;

[0056] Figure 7 This is a flow chart of a video-text combination retrieval module provided by one embodiment of the present application;

[0057] Figure 8 This is a flow chart of a video-text combined retrieval process provided by one embodiment of the present application;

[0058] Figure 9 This is a schematic diagram of a video-text combination retrieval result provided by an embodiment of the present application;

[0059] Figure 10 This is a schematic diagram of a module of a video-text combined retrieval device provided by an embodiment of the present application;

[0060] Figure 11 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application.

[0061] Reference numerals: acquisition module 100 , encoding module 200 , high-level extraction module 300 , middle-level extraction module 400 , hierarchical fusion module 500 , retrieval module 600 , electronic device 1000 , processor 1001 , memory 1002 . DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0063] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0064] In the description of this application, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on this application.

[0065] In the description of this application, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The terms "first" and "second" are used solely to distinguish technical features and are not to be construed as indicating or implying relative importance, or as implicitly specifying the number or order of the technical features indicated.

[0066] In the description of this application, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in this application based on the specific content of the technical solution.

[0067] With the explosive growth of video data, accurately finding videos that meet user needs is becoming increasingly difficult. Common video retrieval methods use the query text entered by the user to search for videos related to the query text within a large number of candidate videos. This is primarily concept-based retrieval, which summarizes conceptual information in a video (such as objects, scenes, and actions) to find videos that contain these concepts. Alternatively, multiple images provided by the user are combined to construct an image query that represents the user's needs. This query is then compared with images in the video to find videos with similar image combinations.

[0068] As the performance of pre-trained models improves, researchers are increasingly focusing on integrating multimodal features extracted from these models. However, retrieved text can only convey a high-level explanation of user needs, but cannot describe more fine-grained visual requirements or distinguish between highly similar visual concepts. Furthermore, compared to image data, video data exhibits complex temporal and spatial variations. Image-based retrieval methods alone cannot effectively capture the temporal features of videos, resulting in low video retrieval accuracy. Therefore, accurately finding videos that meet user needs from a vast video library is a pressing issue.

[0069] Based on this, an embodiment of the present application provides a video-text combined retrieval method, device, electronic device, and storage medium. The method obtains an original video including multiple original video frames and a retrieval text, then uses an image encoder to encode the original video frames to obtain high-level visual features and mid-level visual features, and uses a text encoder to encode the retrieval text to obtain retrieval text features. High-level retained features are extracted from the high-level visual features based on first weight information and the temporal information of the original video frames, high-level difference features are extracted from the retrieval text features based on second weight information, and high-level fused features are obtained based on the high-level retained features and high-level difference features based on the fusion weight information. In this way, a high-level branch is set to extract visual features that need to be retained in high-level vision based on temporal information and retrieve semantic features required to be included in the target video in this article. Then, a convolutional neural network is used to extract mid-level retained features from the mid-level visual features, and mid-level difference features are extracted from the retrieval text features based on third weight information. In this way, a mid-level fused feature is obtained based on the mid-level retained features and mid-level difference features based on an attention mechanism. In this way, a mid-level branch is set to extract finer-grained spatiotemporal features based on the mid-level retained features and mid-level difference features using an attention mechanism. Finally, the target fusion feature is derived from high-level visual features, retrieved text features, high-level fusion features, and mid-level fusion features. Based on the target fusion feature, the preset video library is retrieved to obtain the target video. By setting high-level and mid-level branches and extracting the retained features from the original video and the differential features from the retrieval paper respectively, the user's visual needs are described at different granularities. The attention mechanism is used to focus on spatiotemporal features to obtain the target fusion feature for retrieval, effectively improving the accuracy of video retrieval and accurately finding the target video that meets the user's needs.

[0070] The embodiments of the present application provide a video-text combined retrieval method, device, electronic device and storage medium, which are specifically illustrated by the following embodiments. First, the video-text combined retrieval method in the embodiments of the present application is described.

[0071] The video-text combined retrieval method provided in the embodiment of the present application relates to the field of multimedia technology, and in particular to the field of video retrieval technology. The video-text combined retrieval method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be a computer program running in a terminal or a server side. For example, a computer program can be a native program or software module in an operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system before it can run, such as a client that supports video-text combined retrieval, that is, a program that only needs to be downloaded to a browser environment to run. In short, the above-mentioned computer program can be an application, module or plug-in in any form. Among them, the terminal communicates with the server through a network. The video-text combined retrieval method can be executed by a terminal or a server, or by a terminal and a server in collaboration.

[0072] In some embodiments, the terminal can be a smart phone, tablet computer, laptop computer, desktop computer or smart watch, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where each service node in the blockchain system forms a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP) protocol. The server can be installed with a server end of a video text combination retrieval system, through which the server end can interact with the terminal, for example, the server end can be installed with corresponding software, and the software can be an application that implements a video text combination retrieval method, etc., but is not limited to the above forms. The terminal and the server can be connected via Bluetooth, USB (Universal Serial Bus) or a network or other communication connection method, which is not limited in this embodiment.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0074] The following describes a video-text combined retrieval method according to an embodiment of the present invention.

[0075] Reference Figure 1 As shown, an embodiment of the present application provides a video-text combined retrieval method, which is applied to an infrastructure entity. The video-text combined retrieval method includes but is not limited to the following steps S101 to S106.

[0076] Step S101, obtaining original video and search text.

[0077] In some embodiments, the original video includes multiple original video frames, each of which is a continuous still image. The continuous original video frames are combined to form the complete original video. The original video frames include visual information of the target video that the user wants to retrieve, such as color, texture, shape, edges, etc. The retrieval text is used to describe the user's needs or query intent, including the difference information between the target video and the original video. It can be a natural language query entered by the user or a summary or annotation automatically generated by the system.

[0078] For example, if we input the original video of "a person playing basketball" and the corresponding search text "playing football", the search result will be the target video of "a person playing football". Therefore, obtaining the original video and the search text and analyzing them is the basis for accurately finding the target video required by the user.

[0079] Step S102: Encode the original video frame using an image encoder to obtain high-level visual features and mid-level visual features, and encode the search text using a text encoder to obtain search text features.

[0080] In some embodiments, both the image encoder and the text encoder can be CLIP encoders. CLIP (Contrastive Language-Image Pretraining) is a model for encoding images and text under a contrastive learning framework, proposed by OpenAI. CLIP uses a shared encoder to convert images and text into high-dimensional feature vectors, thereby achieving cross-modal matching and retrieval. The CLIP image encoder uses a convolutional neural network (such as ViT) to extract features from images and map them to a fixed-length vector representation; the CLIP text encoder uses a Transformer architecture to extract features from text and map them to a fixed-length vector representation.

[0081] In some embodiments, the CLIP image encoder is used to encode raw video frames to obtain high-level and mid-level visual features. Each raw video frame is first taken as input and converted into a high-dimensional feature vector by the image encoder. High-level and mid-level visual features are then extracted from the output of the image encoder. High-level visual features capture more abstract and semantic features, while mid-level visual features capture more local and detailed features.

[0082] In some embodiments of the present application, the image encoder includes M coding layers, where M is an integer greater than 1. Each original video frame is encoded using the image encoder, and the output of the Mth coding layer is used as the high-level visual features of the original video frame, and the output of the M-1th coding layer is used as the mid-level visual features of the original video frame. For example, in a CLIP-ResNet50 image encoder, there are a total of four coding layers, and the output of the third coding layer is used as the mid-level visual features, and the output of all four coding layers is used as the high-level visual features.

[0083] In some embodiments, the search text is subjected to pre-processing such as word segmentation and then input into a CLIP text encoder to be converted into a high-dimensional feature vector. Exemplarily, the CLIP text encoder uses a Transformer architecture to extract features from the search text and maps it to a fixed-length vector representation. Finally, the search text features are obtained from the output of the CLIP text encoder. This can be achieved by extracting the output of the last hidden layer of the text encoder or by using a pooling operation (such as average pooling or maximum pooling) of the text encoder, which is not limited in this embodiment.

[0084] Step S103, extracting high-level retained features from high-level visual features according to the first weight information and the timing information of the original video frame, extracting high-level difference features from the retrieved text features according to the second weight information, and obtaining high-level fusion features according to the high-level retained features and the high-level difference features based on the fusion weight information.

[0085] It should be understood that the continuous playback of raw video frames forms a time sequence, so there is temporal information between different raw video frames. By analyzing the temporal information between raw video frames, we can capture the dynamic changes and action information in the original video, which is important for understanding the visual content of the original video and semantically matching it with the retrieved text.

[0086] In some embodiments, high-level retained features are extracted from high-level visual features based on the first weight information and the timing information of the original video frame. The high-level retained features are visual features that the user desires to retain in the target video. The high-level retained features can be extracted from the high-level visual features using the first weight information learned through pre-training and combined with the timing information of the original video.

[0087] In some embodiments, high-level difference features are extracted from the search text features based on the second weight information. The high-level difference features are semantic features that the user requires to be included in the target video. The high-level difference features can be extracted from the search text features using the second weight information learned through pre-training.

[0088] In some embodiments, high-level difference features and high-level retained features are cross-modally fused based on fusion weight information to extract high-level fused features. Thus, a high-level branch is set up to extract the high-level retained features that need to be retained from the high-level visual features based on temporal information, and to retrieve the high-level difference features required by the target video in this article. Ultimately, high-level fused features are obtained to accurately retrieve the target video that meets the user's semantic and visual needs.

[0089] In step S104, a convolutional neural network is used to extract mid-level retained features from mid-level visual features, mid-level difference features are extracted from retrieved text features according to the third weight information, and mid-level fusion features are obtained based on the mid-level retained features and the mid-level difference features based on the attention mechanism.

[0090] It's understandable that a convolutional neural network (CNN) is a deep learning model that effectively extracts features from input data and performs tasks such as classification, detection, or generation through the stacking of convolutional and pooling layers, as well as the combination of fully connected layers. Due to their local perception and weight sharing properties, convolutional neural networks are widely used in image processing and computer vision tasks such as image classification, object detection, and image generation.

[0091] In some embodiments, a convolutional neural network is used to extract mid-level retained features from mid-level visual features. The mid-level retained features are visual features that the user needs to retain in the target video. Compared with high-level retained features, the mid-level retained features include more local and detailed features and have richer spatiotemporal features, thereby providing more fine-grained visual features.

[0092] In some embodiments, mid-level difference features are extracted from the search text features based on the third weight information. The mid-level difference features are semantic features that the user requires to be included in the target video. The third weight information learned through pre-training can be used to extract the mid-level difference features from the search text features.

[0093] It is understandable that the attention mechanism is used to enhance the degree of attention paid to different parts of the input data. By giving different weights to different parts, it is possible to pay more attention to important information when processing data. In some embodiments, the middle-level fusion features are obtained based on the middle-level retained features and the middle-level difference features based on the spatio-temporal attention mechanism. The spatio-temporal attention mechanism is an extended attention mechanism used to process spatio-temporal data such as videos. It introduces attention mechanisms in both time and space dimensions, and can better focus on important moments and spatial positions when processing video data. Therefore, the middle-level branch is set to use the spatio-temporal attention mechanism to perform cross-modal fusion of the middle-level retained features and the middle-level difference features, and extract the middle-level fusion features with finer granularity and spatio-temporal features than the high-level fusion features, so as to accurately retrieve the target video that meets the user's semantic and visual needs.

[0094] Step S105 , obtaining target fusion features according to high-level visual features, retrieved text features, high-level fusion features and middle-level fusion features.

[0095] In some embodiments, high-level visual features, retrieved text features, high-level fusion features, and mid-level fusion features are fused to obtain target fusion features. Specifically, each feature may be weighted summed, normalized, or otherwise processed. It is understood that those skilled in the art may configure the features based on actual needs, and this embodiment does not impose any limitation thereto.

[0096] Step S106: Search the preset video library based on the target fusion feature to obtain the target video.

[0097] In some embodiments, a preset video library is searched based on the target fusion feature, and a similarity calculation is performed between the target fusion feature and the feature of each video in the preset video library, such as cosine similarity, Euclidean distance, correlation coefficient, etc. Based on the results of the similarity calculation, the videos in the preset video library are sorted, and the video most similar to the target video is found. Several videos with the highest similarity can be selected as candidates. Based on the sorting results, the video with the highest similarity is selected as the target video. If there are multiple candidates, further screening can be performed based on other evaluation indicators (such as video quality, relevance, etc.), which is not limited in this embodiment.

[0098] Therefore, by setting high-level branches and middle-level branches and extracting the retained features in the original video and retrieving the difference features in this article for fusion, the user's visual needs are described from different granularities, and the spatiotemporal attention mechanism is used to focus on the spatiotemporal features to obtain the target fusion features for searching the preset video library, effectively improving the accuracy of video retrieval and accurately finding the target video that meets the user's needs.

[0099] Reference Figure 2 As shown, in some embodiments of the present application, the above-mentioned step S103 may include but is not limited to the following steps S201 to S204.

[0100] Step S201 : inputting high-level visual features into a temporal encoder for temporal encoding based on temporal information to obtain temporal high-level visual features.

[0101] In some embodiments, the temporal encoder can be a temporal encoder based on an LSTM (Long Short-Term Memory) network. Based on the temporal information, high-level visual features are input into the LSTM temporal encoder for temporal encoding to obtain temporal high-level visual features. Specifically, the high-level visual features corresponding to each original video frame are sorted according to the temporal information to form a high-level visual feature sequence, which is then input into the LSTM temporal encoder for temporal encoding.

[0102] The LSTM temporal encoder is composed of multiple LSTM units, each of which calculates the current input and the hidden state of the previous moment to generate the output of the current moment and the new hidden state. The high-level visual feature sequence is gradually input into the LSTM temporal encoder to obtain the LSTM output of each time step. The output of the LSTM is fused to obtain temporal high-level visual features. Specifically, you can choose to use the LSTM output of the last time step as the temporal high-level visual feature, or you can fuse the LSTM outputs of all time steps, for example, using average pooling, maximum pooling, etc. to obtain temporal high-level visual features. This embodiment does not limit this.

[0103] It is understandable that the time series encoder can also be a time series encoder based on Mean-pooling or a time series encoder based on Transformer. Those skilled in the art can set it according to actual needs, and this embodiment does not limit this.

[0104] In step S202, the temporal high-level visual features are input into a first multi-layer perceptron, a first linear feature is obtained according to the first weight information and the first bias information, the first linear feature is mapped to a nonlinear space using a first activation function to obtain a first nonlinear feature, and the first nonlinear feature is input into a first regularization function for feature screening to obtain a high-level retained feature.

[0105] It can be understood that the multilayer perceptron (MLP) is a feedforward neural network model. Specifically, the temporal high-level visual features are input into the first multilayer perceptron, and the first linear features are obtained based on the first weight information and the first bias information. A linear layer (also called a fully connected layer) is used to linearly map the input temporal high-level visual features. The linear layer is composed of multiple neurons, each neuron is connected to each dimension of the input feature, and the first linear features are obtained by weighted sum calculation using the weight matrix in the first weight information and the bias vector in the first bias information.

[0106] The first linear feature is mapped to a nonlinear space through a first activation function to obtain a first nonlinear feature. The first activation function can be a ReLU activation function. By inputting the first linear feature into the ReLU activation function, it sets negative values ​​to 0 and retains non-negative values ​​to obtain a first nonlinear feature, thereby introducing nonlinearity and enhancing the representation ability of the first multi-layer perceptron. Finally, the first nonlinear feature is input into the first regularization function for feature screening. The first regularization function can be a Dropout function, and a Dropout layer is introduced after the ReLU activation function. Specifically, Dropout is a regularization technique that randomly sets the output of a part of neurons to 0 to reduce the dependency between neurons and prevent overfitting. Specifically, the extraction of high-level retained features can be expressed as:

[0107]

[0108] in, Represents temporal high-level visual features, Represents the first weight information, Represents the first bias information.

[0109] In step S203, the retrieved text features are input into a second multi-layer perceptron, a second linear feature is obtained according to the second weight information and the second bias information, the second linear feature is mapped to a nonlinear space using a second activation function to obtain a second nonlinear feature, and the second nonlinear feature is input into a second regularization function for feature screening to obtain a high-level difference feature.

[0110] In some embodiments, the retrieved text features are input into a second multi-layer perceptron, and linear mapping is performed according to the weight matrix in the second weight information and the bias vector in the second bias information to calculate the second linear features. The second linear features are then mapped to a nonlinear space using a second activation function to obtain a second nonlinear feature. Specifically, the second linear features are input into a ReLU activation function to obtain the corresponding second nonlinear features. Finally, the second nonlinear features are input into a second regularization function for feature screening. A Dropout layer can be introduced after the ReLU activation function to randomly set the outputs of some neurons to 0 to reduce the dependency between neurons and prevent overfitting, thereby obtaining high-level difference features. Specifically, the extraction of high-level difference features can be expressed as:

[0111]

[0112] Among them, t represents the retrieval text feature, Represents the second weight information, Represents the second bias information.

[0113] In step S204, the high-level retained features and the high-level difference features are input into a fused multi-layer perceptron, and fused linear features are obtained according to the fusion weight information and the fusion bias information. The fused linear features are mapped to a nonlinear space using a fusion activation function to obtain fused nonlinear features. The fused nonlinear features are input into a fusion regularization function for feature screening to obtain high-level fusion features.

[0114] In some embodiments, the high-level retention features and the high-level difference features are spliced ​​and input into a fusion multi-layer perceptron, and linear mapping is performed according to the weight matrix in the fusion weight information and the bias vector in the fusion bias information to calculate the fusion linear features. The fusion activation function is then used to map the fusion linear features to a nonlinear space to obtain fusion nonlinear features. Specifically, the fusion linear features are input into the ReLU activation function to obtain the corresponding fusion nonlinear features. Finally, the fusion nonlinear features are input into the fusion regularization function for feature screening. A Dropout layer can be introduced after the ReLU activation function to randomly set the output of a part of the neurons to 0, thereby obtaining high-level fusion features. Specifically, the extraction of high-level fusion features can be expressed as:

[0115]

[0116] Among them, w h represents the fusion weight information, b h Represents the fusion bias information.

[0117] Reference Figure 3 As shown, in some embodiments of the present application, the above-mentioned step S104 may include but is not limited to the following steps S301 to S303.

[0118] In step S301, the middle-level visual features are input into the convolutional neural network to obtain the middle-level linear features, the middle-level linear features are mapped to the nonlinear space using the middle-level activation function to obtain the middle-level nonlinear features, and the middle-level nonlinear features are input into the middle-level regularization function for feature screening to obtain the middle-level retained features.

[0119] In some embodiments, the mid-level visual features are input into a convolutional neural network to obtain mid-level linear features, and then the mid-level linear features are mapped to a nonlinear space using a mid-level activation function to obtain mid-level nonlinear features. Specifically, the mid-level activation function can be a ReLU activation function, or an activation function such as a sigmoid or tanh. Finally, the mid-level nonlinear features are input into a mid-level regularization function Dropout layer for feature screening to obtain mid-level retained features. Specifically, the extraction of the mid-level retained features can be expressed as:

[0120]

[0121] Among them, p m Represents mid-level visual features.

[0122] In step S302, the retrieved text features are input into a third multi-layer perceptron, a third linear feature is obtained according to the third weight information and the third bias information, the third linear feature is mapped to a nonlinear space using a third activation function to obtain a third nonlinear feature, and the third nonlinear feature is input into a third regularization function for feature screening to obtain a mid-level difference feature.

[0123] In some embodiments, the retrieved text features are input into a third multi-layer perceptron, and linear mapping is performed according to the weight matrix in the third weight information and the bias vector in the third bias information to calculate the third linear features. The third linear features are then mapped to a nonlinear space using a third activation function to obtain a third nonlinear feature. Specifically, the third linear features are input into a ReLU activation function to obtain the corresponding third nonlinear features. Finally, the third nonlinear features are input into a third regularization function for feature screening. A Dropout layer can be introduced after the ReLU activation function to randomly set the outputs of some neurons to 0 to reduce the dependency between neurons and prevent overfitting, thereby obtaining mid-layer difference features. Specifically, the extraction of mid-layer difference features can be expressed as:

[0124]

[0125] in, Represents the third weight information, Represents the third bias information.

[0126] In step S303, the middle-level retained features and the middle-level difference features are input into the attention mechanism, and the middle-level retained features and the middle-level difference features are spatially and temporally fused according to the retrieved text features to obtain the middle-level fused features.

[0127] In some embodiments, the mid-level retained features and mid-level difference features are input into a multi-head spatiotemporal attention mechanism, and the retrieval text features are used to guide the spatiotemporal fusion process of the mid-level retained features and mid-level difference features, thereby obtaining the mid-level fused features. It is understood that the multi-head spatiotemporal attention mechanism is an attention mechanism for processing spatiotemporal sequence data. By combining multi-head attention with spatiotemporal characteristics, it can effectively capture information related to spatiotemporal features and improve the accuracy of video retrieval.

[0128] Reference Figure 4 As shown, in some embodiments of the present application, the above-mentioned step S303 may include but is not limited to the following steps S401 to S404.

[0129] Step S401: concatenate the middle-layer retained features and the middle-layer difference features to obtain the middle-layer concatenated features.

[0130] In some embodiments, the middle-level retained features and the middle-level difference features are concatenated to obtain middle-level concatenated features. The middle-level retained features and the middle-level difference features are concatenated in a predetermined order. This can be done on a feature dimension, i.e., two features are concatenated on the feature dimension to form a larger feature vector. This is not a limitation in this embodiment.

[0131] Step S402: Perform the first linear transformation of the i-th attention head on the middle-level concatenated features to obtain feature K i , perform the second linear transformation of the middle-level splicing feature on the i-th attention head to obtain the feature V i , the retrieved text features are subjected to the third linear transformation of the i-th attention head to obtain the feature Q i .

[0132] In some embodiments, the attention mechanism is a multi-head spatiotemporal attention mechanism. For example, the multi-head spatiotemporal attention mechanism has 8 attention heads. For the i-th attention head, the middle layer splicing feature is subjected to a first linear change to obtain the feature K i And the middle layer splicing feature is subjected to the second linear change to obtain the feature V i , and perform the third linear transformation on the retrieved text features to obtain feature Q iwhere i is an integer between [1 and 8]. Specifically, each attention head corresponds to a linear layer Ψ, which is used to implement linear transformations. It is understood that the linear layer of each attention head is independent, that is, each attention head has its own set of linear transformation parameters for generating corresponding features. This allows each attention head to focus on different feature subspaces and learn different attention weights, thereby enhancing the expressive power of the model.

[0133] Step S403: Based on the feature K i , Feature V i , Feature Q i , the number of attention heads n and the feature dimension d of the middle-layer retained features, and obtain the attention features of the i-th attention head.

[0134] In some embodiments, according to the feature K i , Feature V i , Feature Q i , the number of attention heads n and the feature dimension d of the middle layer retained features, the attention feature x of the i-th attention head can be calculated i :

[0135]

[0136] in, T stands for transpose operation.

[0137] In step S404, the attention features of each attention head are concatenated to obtain the middle-level fusion features.

[0138] In some embodiments, the attention features of each attention head are concatenated to obtain the middle-level fusion feature x, that is:

[0139]

[0140] Reference Figure 5 As shown, in some embodiments of the present application, the above-mentioned step S105 may include but is not limited to the following steps S501 to S503.

[0141] Step S501: Input the high-level visual features and the retrieval text features into a fourth multi-layer perceptron, obtain a fourth linear feature based on the fourth weight information and the fourth bias information, and use a fourth activation function to map the fourth linear feature to a nonlinear space to obtain a first intermediate weight of the high-level visual feature and a second intermediate weight of the retrieval text feature.

[0142] In some embodiments, the high-level visual features and the search text features are concatenated and input into a fourth multi-layer perceptron. Linear mapping is performed based on the weight matrix in the fourth weight information and the bias vector in the fourth bias information to calculate a fourth linear feature. The fourth linear feature is then mapped into a nonlinear space using a fourth activation function to obtain a first intermediate weight of the high-level visual feature and a second intermediate weight of the search text feature. Specifically, the fourth linear feature is input into a sigmoid activation function to obtain the corresponding first intermediate weight a and second intermediate weight 1-a. The weight values ​​of the high-level visual features and the search text are thus learned by the fourth multi-layer perceptron.

[0143] Step S502: multiplying the first intermediate weight and the high-level visual feature to obtain a first intermediate result, and multiplying the second intermediate weight and the search text feature to obtain a second intermediate result.

[0144] In some embodiments, the first intermediate weight a and the high-level visual feature Multiply to get the first intermediate result The second intermediate weight 1-a is multiplied by the retrieved text feature t to obtain a second intermediate result (1-a)t.

[0145] Step S503: Add the first intermediate result, the second intermediate result, the high-level fusion feature, and the middle-level fusion feature to obtain the target fusion feature.

[0146] In some embodiments, the first intermediate result The second intermediate result (1-a)t, high-level fusion feature x h and the middle-level fusion feature x m Add the target fusion features, that is,

[0147] Reference Figure 6 As shown, in some embodiments of the present application, the above-mentioned step S106 may include but is not limited to the following steps S601 to S604.

[0148] Step S601: Obtain a reference video from a preset video library.

[0149] In some embodiments, the preset video library includes multiple reference videos, each of which corresponds to multiple reference video frames. Each reference video in the preset video library and its corresponding reference video frame are obtained.

[0150] Step S602: Encode the reference video frame using an image encoder to obtain reference visual features.

[0151] In some embodiments, an image encoder is used to encode the reference video frame to obtain the reference visual features. Specifically, each reference video frame of the reference video is used as input and encoded into a high-dimensional feature vector by a CLIP image encoder to obtain the corresponding reference visual features.

[0152] Step S603 : inputting the high-level visual feature into a temporal encoder for temporal encoding based on the reference temporal information of the reference visual feature to obtain a temporal reference visual feature.

[0153] In some embodiments, the continuous playback of the reference video frames forms a time sequence, so there is reference timing information between different reference video frames. Based on the reference timing information, the reference visual features are input into the LSTM timing encoder for timing encoding to obtain a timing reference visual feature. It is understandable that the timing encoder can also be a timing encoder based on mean-pooling (average pooling) or a timing encoder based on Transformer. Those skilled in the art can set it according to actual needs, and this embodiment does not limit this.

[0154] Step S604 : Calculate the similarity between the temporal reference visual features and the target fusion features, such as cosine similarity, Euclidean distance, correlation coefficient, etc. If the similarity meets a preset similarity threshold, the reference video corresponding to the similarity is used as the target video.

[0155] In some embodiments, the similarity between the temporal reference visual features and the target fusion features is calculated. If the similarity meets a preset similarity threshold, the reference video corresponding to the similarity is used as the target video. Alternatively, based on the results of the similarity calculation, the videos in the preset video library are sorted to find the videos most similar to the target video, and several videos with the highest similarity can be selected as candidates. Based on the sorting results, the video with the highest similarity is selected as the target video. If there are multiple candidates, further screening can be performed based on other evaluation indicators (such as video quality, relevance, etc.), which is not limited in this embodiment.

[0156] Therefore, by setting high-level branches and middle-level branches and extracting the retained features in the original video and retrieving the difference features in this article for fusion, the user's visual needs are described from different granularities, and the spatiotemporal attention mechanism is used to focus on the spatiotemporal features to obtain the target fusion features for searching the preset video library, effectively improving the accuracy of video retrieval and accurately finding the target video that meets the user's needs.

[0157] The present application is described below through a complete embodiment:

[0158] Reference Figure 7 The video text combination retrieval module flow chart and Figure 8The video text combination retrieval process flow chart, after obtaining the original video and the retrieval text, feature extraction is performed. Specifically, the retrieval text is input into the CLIP text encoder to obtain the retrieval text feature t, and the original video is input into the CLIP image encoder to obtain the high-level visual feature p h and mid-level visual features p m . Set up a high-level branch, specifically, the retrieval text feature t is passed through the high-level difference module containing the second multi-layer perceptron to obtain the high-level difference feature And the high-level visual features p h After passing through the temporal encoder, the temporal high-level visual features are obtained Then the high-level retention feature is obtained through the high-level retention module containing the first multi-layer perceptron Finally, the high-level difference features and high-level feature preservation After fusion through the high-level preservation-difference fusion module including the fusion multi-layer perceptron, the high-level fusion feature x is obtained. h .

[0159] The corresponding middle-level branch is to pass the retrieval text feature t through the middle-level difference module containing the third multi-layer perceptron to obtain the middle-level difference feature And the middle-level visual feature p m The input includes the middle layer retention module of the convolutional neural network to obtain the middle layer retention features Finally, the middle layer retains the features and middle-level differences The splicing is performed and then fused through the middle-layer preservation-difference fusion module composed of a multi-head spatiotemporal attention mechanism. The retrieval text feature t is used to guide the spatiotemporal fusion to obtain the attention feature corresponding to each attention head, and the various attention features are spliced ​​to obtain the middle-layer fusion feature x m .

[0160] Using the fourth multi-layer perceptron to learn temporal high-level visual features and retrieve the weight of the text feature t, and combine it with the high-level fusion feature x h and the middle-level fusion feature x m A hierarchical multi-feature fusion module is used to obtain the target fusion feature o. Finally, each reference video in the preset video library is retrieved based on the target fusion feature o. Specifically, image encoding and time encoding are performed on the reference video to obtain the temporal reference video feature. The similarity between the temporal reference feature and the target fusion feature is calculated to determine whether it is the target video to be retrieved.

[0161] Reference Figure 9The schematic diagram of the video text combination retrieval result is shown. By inputting the original video and the search text, the corresponding target video can be retrieved. Table 1 below compares the performance of the current image combination retrieval method in video on the DiCa-CVR dataset. First, the non-combination retrieval method is compared. (1) Retrieval using only video and (2) Retrieval using only text have poor results on the combination retrieval dataset, indicating that single-modality retrieval cannot meet user needs and proves the necessity of video combination retrieval tasks. Then, (3) the method of directly adding video features and text features for retrieval has poor results, indicating that the video combination retrieval method proposed in this application can effectively meet user needs. Secondly, the four best image combination retrieval methods are compared. Since the comparison method is only for image combination retrieval tasks, this application uses Mean-pooling as the temporal encoder of the image combination retrieval method and compares it with the Mean-pooling method of this application. The experimental results show that this application has good results when using the same temporal encoder, proving the effectiveness of the hierarchical multi-feature fusion method proposed in this application. Finally, the influence of the temporal encoder on the method is explored. This application uses Mean-pooling, Transformer, and LSTM as time series encoders respectively. Experimental results show that using LSTM as the time series encoder is the best.

[0162] Table 1 Experimental results

[0163]

[0164]

[0165] It can be seen that compared with the single-modal retrieval query of text-video, the video-text combined retrieval method of this application is more able to meet user needs and effectively improve the retrieval recall rate. On the one hand, this application performs feature extraction on the retained features and difference features respectively, effectively integrating the important information in the original video and the retrieval text, and improving the retrieval recall rate; on the other hand, through multi-level feature extraction and fusion methods, it effectively extracts the spatiotemporal characteristics of the original video, improves the video comprehension ability, thereby improving the retrieval accuracy and accurately finding the target video that meets the user's needs.

[0166] The embodiment of the present application also provides a video text combination retrieval device, which can implement the above-mentioned video text combination retrieval method, referring to Figure 10 As shown, in some embodiments of the present application, the video-text combined retrieval device includes:

[0167] The acquisition module 100 is used to acquire the original video and retrieve the text; wherein the original video includes multiple original video frames;

[0168] The encoding module 200 is used to encode the original video frame using an image encoder to obtain high-level visual features and mid-level visual features, and to encode the search text using a text encoder to obtain search text features;

[0169] a high-level extraction module 300 for extracting high-level retained features from high-level visual features based on the first weight information and the timing information of the original video frame, extracting high-level difference features from the retrieved text features based on the second weight information, and obtaining high-level fused features from the high-level retained features and the high-level difference features based on the fusion weight information;

[0170] a middle-layer extraction module 400 for extracting middle-layer retained features from middle-layer visual features using a convolutional neural network, extracting middle-layer difference features from retrieved text features based on third weight information, and obtaining middle-layer fusion features based on the middle-layer retained features and the middle-layer difference features based on an attention mechanism;

[0171] Hierarchical fusion module 500 obtains target fusion features based on high-level visual features, retrieved text features, high-level fusion features and mid-level fusion features;

[0172] The retrieval module 600 searches the preset video library based on the target fusion feature to obtain the target video.

[0173] The specific implementation of the video-text combined retrieval device of this embodiment is basically the same as the specific implementation of the above-mentioned video-text combined retrieval method, and will not be described in detail here.

[0174] Figure 11 The electronic device 1000 provided in an embodiment of the present application is shown. The electronic device 1000 includes: a processor 1001, a memory 1002, and a computer program stored in the memory 1002 and executable on the processor 1001. When the computer program is executed, it is used to execute the above-mentioned video-text combination retrieval method.

[0175] The processor 1001 and the memory 1002 may be connected via a bus or other means.

[0176] Memory 1002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs, such as the video-text combined retrieval method described in the embodiments of this application. Processor 1001 implements the above-described video-text combined retrieval method by executing the non-transitory software program and instructions stored in memory 1002.

[0177] The memory 1002 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data for executing the above-mentioned video text combination retrieval method. In addition, the memory 1002 may include a high-speed random access memory 1002, and may also include a non-volatile memory 1002, such as at least one storage device memory device, a flash memory device or other non-volatile solid-state memory device. In some embodiments, the memory 1002 may optionally include a memory 1002 remotely located relative to the processor 1001, and these remote memories 1002 may be connected to the electronic device 1000 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0178] The non-transient software program and instructions required to implement the above-mentioned video text combination retrieval method are stored in the memory 1002. When executed by one or more processors 1001, the above-mentioned video text combination retrieval method is executed, for example, Figure 1 Steps S101 to S106 of the method, Figure 2 Steps S201 to S204 of the method, Figure 3 Steps S301 to S303 of the method, Figure 4 Steps S401 to S404 of the method, Figure 5 Steps S501 to S503 of the method, Figure 6 Method steps S601 to S604 in .

[0179] The embodiment of the present application also provides a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program, which implements the above-mentioned video text combination retrieval method when executed by the processor. The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory optionally includes a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0180] The embodiments of the present application provide a video-text combined retrieval method, device, electronic device, and storage medium, wherein the method obtains an original video including multiple original video frames and a retrieval text, then uses an image encoder to encode the original video frames to obtain high-level visual features and mid-level visual features, and uses a text encoder to encode the retrieval text to obtain retrieval text features. High-level retained features are extracted from the high-level visual features based on first weight information and the temporal information of the original video frames, high-level difference features are extracted from the retrieval text features based on second weight information, and high-level fused features are obtained based on the high-level retained features and high-level difference features based on the fusion weight information. Thus, a high-level branch is set to extract visual features that need to be retained in high-level vision based on temporal information and retrieve semantic features required to be included in the target video in this article. Then, a convolutional neural network is used to extract mid-level retained features from the mid-level visual features, and mid-level difference features are extracted from the retrieval text features based on third weight information. Based on the mid-level retained features and mid-level difference features, a mid-level fused feature is obtained based on the attention mechanism. Thus, a mid-level branch is set to extract finer-grained spatiotemporal features based on the mid-level retained features and mid-level difference features using the attention mechanism. Finally, the target fusion feature is derived from high-level visual features, retrieved text features, high-level fusion features, and mid-level fusion features. Based on the target fusion feature, the preset video library is retrieved to obtain the target video. By setting high-level and mid-level branches and extracting the retained features from the original video and the differential features from the retrieval paper respectively, the user's visual needs are described at different granularities. The attention mechanism is used to focus on spatiotemporal features to obtain the target fusion feature for retrieval, effectively improving the accuracy of video retrieval and accurately finding the target video that meets the user's needs.

[0181] The embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0182] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, storage device storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0183] It should also be understood that the various implementations provided in the embodiments of this application can be arbitrarily combined to achieve different technical effects. The above is a specific description of the preferred implementation of this application, but this application is not limited to the above implementation. Those skilled in the art can also make various equivalent modifications or substitutions under the conditions that do not violate the spirit of this application.

Claims

1. A video text combined retrieval method, characterized in that: include: Acquire original video and retrieve text; wherein the original video includes multiple original video frames; Encoding the original video frame using an image encoder to obtain high-level visual features and mid-level visual features, and encoding the search text using a text encoder to obtain search text features; extracting high-level retained features from the high-level visual features according to the first weight information and the timing information of the original video frame, extracting high-level difference features from the retrieved text features according to the second weight information, and obtaining high-level fused features from the high-level retained features and the high-level difference features based on the fusion weight information; Extracting mid-level retained features from the mid-level visual features using a convolutional neural network, extracting mid-level difference features from the retrieved text features based on third weight information, and obtaining mid-level fusion features based on the mid-level retained features and the mid-level difference features based on an attention mechanism; Inputting the high-level visual features and the search text features into a fourth multi-layer perceptron, obtaining a fourth linear feature based on fourth weight information and fourth bias information, mapping the fourth linear feature to a nonlinear space using a fourth activation function, obtaining a first intermediate weight of the high-level visual feature and a second intermediate weight of the search text feature, multiplying the first intermediate weight by the high-level visual feature to obtain a first intermediate result, multiplying the second intermediate weight by the search text feature to obtain a second intermediate result, and adding the first intermediate result, the second intermediate result, the high-level fusion feature, and the middle-level fusion feature to obtain a target fusion feature; The target video is obtained by searching a preset video library based on the target fusion feature.

2. The video text combination retrieval method according to claim 1, characterized in that: The image encoder includes M coding layers, where M is an integer greater than 1; encoding the original video frame using the image encoder to obtain high-level visual features and mid-level visual features includes: Each of the original video frames is encoded using the image encoder, and the output result of the Mth encoding layer is used as the high-level visual feature of the original video frame, and the output result of the M-1th encoding layer is used as the mid-level visual feature of the original video frame.

3. The video text combination retrieval method according to claim 1, characterized in that: The extracting high-level retained features from the high-level visual features according to the first weight information and the timing information of the original video frame, extracting high-level difference features from the retrieved text features according to the second weight information, and obtaining high-level fused features from the high-level retained features and the high-level difference features based on the fusion weight information includes: Inputting the high-level visual features into a temporal encoder for temporal encoding based on the temporal information to obtain temporal high-level visual features; Inputting the temporal high-level visual features into a first multi-layer perceptron, obtaining a first linear feature based on the first weight information and the first bias information, mapping the first linear feature to a nonlinear space using a first activation function to obtain a first nonlinear feature, inputting the first nonlinear feature into a first regularization function for feature screening to obtain the high-level retained feature; Inputting the search text feature into a second multilayer perceptron, obtaining a second linear feature based on the second weight information and the second bias information, mapping the second linear feature into a nonlinear space using a second activation function to obtain a second nonlinear feature, and inputting the second nonlinear feature into a second regularization function for feature screening to obtain the high-level difference feature; The high-level retained features and the high-level difference features are input into a fused multi-layer perceptron, and fused linear features are obtained according to the fusion weight information and the fusion bias information. The fused linear features are mapped to a nonlinear space using a fusion activation function to obtain a fused nonlinear feature. The fused nonlinear feature is input into a fusion regularization function for feature screening to obtain the high-level fusion feature.

4. The video text combined retrieval method according to claim 1, characterized in that: The method comprises extracting middle-level retained features from the middle-level visual features using a convolutional neural network, extracting middle-level difference features from the retrieved text features according to third weight information, and obtaining middle-level fusion features based on the middle-level retained features and the middle-level difference features based on an attention mechanism, including: Inputting the middle-layer visual features into a convolutional neural network to obtain middle-layer linear features, mapping the middle-layer linear features into a nonlinear space using a middle-layer activation function to obtain middle-layer nonlinear features, and inputting the middle-layer nonlinear features into a middle-layer regularization function for feature screening to obtain the middle-layer retained features; Inputting the search text feature into a third multi-layer perceptron, obtaining a third linear feature based on the third weight information and the third bias information, mapping the third linear feature into a nonlinear space using a third activation function to obtain a third nonlinear feature, inputting the third nonlinear feature into a third regularization function for feature screening to obtain the middle-layer difference feature; The middle-layer retained features and the middle-layer difference features are input into the attention mechanism, and the middle-layer retained features and the middle-layer difference features are spatiotemporally fused according to the retrieved text features to obtain the middle-layer fused features.

5. The video text combination retrieval method according to claim 4, characterized in that: The attention mechanism includes n attention heads, where n is an integer greater than 1; inputting the middle-level retained features and the middle-level difference features into the attention mechanism, and performing spatiotemporal fusion on the middle-level retained features and the middle-level difference features according to the retrieved text features to obtain the middle-level fused features, including: Splicing the middle layer retained features and the middle layer difference features to obtain middle layer splicing features; The middle-level splicing feature is subjected to the first linear transformation of the i-th attention head to obtain the feature K i , the middle-level splicing feature is subjected to the second linear transformation of the i-th attention head to obtain the feature V i , the retrieval text feature is subjected to the third linear transformation of the i-th attention head to obtain the feature Q i ; Where i is an integer between [1,n]; According to the feature K i 、The feature V i , the feature Q i , the number of attention heads n and the feature dimension d of the middle layer retained features, to obtain the attention feature of the i-th attention head; The attention features of each attention head are concatenated to obtain the middle-layer fusion features.

6. The video text combined retrieval method according to claim 1 or 3, characterized in that: The preset video library includes a plurality of reference videos; the retrieval based on the target fusion feature to obtain the target video includes: Obtaining a reference video from the preset video library; wherein the reference video includes a plurality of reference video frames; Encoding the reference video frame using the image encoder to obtain reference visual features; Inputting the high-level visual feature into a temporal encoder for temporal encoding based on reference temporal information of the reference visual feature to obtain a temporal reference visual feature; The similarity between the temporal reference visual feature and the target fusion feature is calculated, and if the similarity meets a preset similarity threshold, the reference video corresponding to the similarity is used as the target video.

7. A video text combined retrieval device, characterized in that: Applying the video-text combined retrieval method according to any one of claims 1 to 6, comprising: An acquisition module, configured to acquire an original video and retrieve text; wherein the original video includes a plurality of original video frames; An encoding module, configured to encode the original video frame using an image encoder to obtain high-level visual features and mid-level visual features, and to encode the search text using a text encoder to obtain search text features; a high-level extraction module, configured to extract high-level retained features from the high-level visual features according to the first weight information and the timing information of the original video frame, extract high-level difference features from the retrieved text features according to the second weight information, and obtain high-level fused features from the high-level retained features and the high-level difference features based on the fusion weight information; a middle-layer extraction module, configured to extract middle-layer retained features from the middle-layer visual features using a convolutional neural network, extract middle-layer difference features from the retrieved text features according to third weight information, and obtain middle-layer fusion features based on the middle-layer retained features and the middle-layer difference features based on an attention mechanism; a hierarchical fusion module, configured to input the high-level visual features and the search text features into a fourth multi-layer perceptron, obtain a fourth linear feature based on fourth weight information and fourth bias information, map the fourth linear feature to a nonlinear space using a fourth activation function, obtain a first intermediate weight of the high-level visual feature and a second intermediate weight of the search text feature, multiply the first intermediate weight by the high-level visual feature to obtain a first intermediate result, multiply the second intermediate weight by the search text feature to obtain a second intermediate result, and add the first intermediate result, the second intermediate result, the high-level fused feature, and the mid-level fused feature to obtain a target fused feature; The retrieval module is used to retrieve the target video from a preset video library based on the target fusion feature.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the video-text combination retrieval method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the video-text combination retrieval method according to any one of claims 1 to 6.

Images