Video retrieval method

Abstract

Embodiments of the present description provide a video retrieval method. The video retrieval method comprises: obtaining a retrieval text and at least one candidate video; inputting the retrieval text and a target candidate video into a video matching model to obtain a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, the matching weight is determined based on a first matching result and a second matching result between the retrieval text and the target candidate video, the first matching result is used for representing the matching degree between the retrieval text and each target object in the target candidate video, and the second matching result is used for representing the matching degree between the retrieval text and video semantics of the target candidate video; and determining at least one target video among the at least one candidate video on the basis of the matching weight corresponding to each candidate video. Local content and overall content of the candidate video are analyzed to improve the accuracy of text-based video retrieval.

Description

Video retrieval methods

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 1, 2023, with application number 202310961671.3 and application name “Video Retrieval Method”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of this specification relate to the field of data processing technology, and in particular to a video retrieval method. Background Art

[0003] With the development of various video platforms, people often spend their free time watching videos. Due to different viewing needs, users may find themselves searching for videos based on text if they are not interested in the videos recommended by the video platform. In current practical applications, searching for semantically relevant videos based on natural language input by users often requires appropriate matching modeling between video and text data. Alternatively, multiple single-modal pre-trained models can be trained to extract features and perform feature fusion.

[0004] However, because videos contain rich visual elements, text descriptions may only correspond to part of the video content, lacking a comprehensive understanding of the overall and local content of the video. This makes video retrieval inaccurate and reduces the user experience. Therefore, a method to address this issue is urgently needed.

[0005] Summary of the Invention

[0006] In view of this, embodiments of this specification provide a video retrieval method. One or more embodiments of this specification also relate to a video retrieval apparatus, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art.

[0007] According to a first aspect of the embodiments of this specification, a video retrieval method is provided, including:

[0008] Obtaining a search text and at least one candidate video;

[0009] Inputting the search text and the target candidate video into a video matching model, obtaining a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on a first matching result and a second matching result between the search text and the target candidate video, wherein the first matching result is used to characterize the matching degree between the search text and each target object in the target candidate video, and the second matching result is used to characterize the matching degree between the search text and the video semantics of the target candidate video;

[0010] At least one target video is determined from the at least one candidate video based on the matching weight corresponding to each candidate video.

[0011] According to a second aspect of the embodiments of this specification, a video retrieval device is provided, including:

[0012] An acquisition module, configured to acquire a search text and at least one candidate video;

[0013] An input module is configured to input the search text and the target candidate video into a video matching model, and obtain a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on a first matching result and a second matching result between the search text and the target candidate video, wherein the first matching result is used to characterize the matching degree between the search text and each target object in the target candidate video, and the second matching result is used to characterize the matching degree between the search text and the video semantics of the target candidate video;

[0014] The determination module is configured to determine at least one target video from the at least one candidate video based on the matching weight corresponding to each candidate video.

[0015] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0016] memory and processor;

[0017] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned video retrieval method are implemented.

[0018] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned video retrieval method are implemented.

[0019] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned video retrieval method.

[0020] The video retrieval method provided in this specification includes: obtaining a retrieval text and at least one candidate video; inputting the retrieval text and the target candidate video into a video matching model, and obtaining a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on a first matching result and a second matching result between the retrieval text and the target candidate video, the first matching result being used to characterize the matching degree between the retrieval text and each target object in the target candidate video, and the second matching result being used to characterize the matching degree between the retrieval text and the video semantics of the target candidate video; based on the matching weight corresponding to each candidate video, at least one target video is determined in the at least one candidate video.

[0021] One embodiment of this specification calculates a first matching result between a search text and a target candidate video to determine the matching degree between the search text and each target object in the target candidate video. A second matching result is then calculated between the search text and the target candidate video to determine the matching degree between the search text and the video semantics of the target candidate video. Furthermore, the matching weight of the target candidate video is determined by combining the first and second matching results. The accuracy of determining the matching weight of the target candidate video is improved by analyzing the local and overall content of the target candidate video. Finally, based on the matching weight corresponding to each candidate video, the target video is determined from the candidate videos, thereby improving the accuracy of text-based video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is an architecture diagram of a video retrieval system provided by one embodiment of this specification;

[0023] FIG2 is a flow chart of a video retrieval method provided by one embodiment of this specification;

[0024] FIG3 is a schematic diagram of a model architecture of a video matching model provided by one embodiment of this specification;

[0025] FIG4a is a schematic diagram of an interactive interface of a video retrieval method provided by one embodiment of this specification;

[0026] FIG4 b is a schematic diagram of an interactive interface of another video retrieval method provided by an embodiment of this specification;

[0027] FIG5 is a flowchart of a processing process of a video retrieval method provided by one embodiment of this specification;

[0028] FIG6 is a schematic structural diagram of a video retrieval device provided by one embodiment of this specification;

[0029] FIG7 is a structural block diagram of a computing device provided in one embodiment of this specification. DETAILED DESCRIPTION

[0030] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0031] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0032] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0033] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and corresponding operation entrances must be provided for users to choose to authorize or refuse.

[0034] The large model in one or more embodiments of this specification specifically refers to a deep learning model with large-scale model parameters, which typically contains hundreds of millions, tens of billions, or even hundreds of billions of model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained by large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as large-scale language models (LLM) and multi-modal pre-training models.

[0035] When large models are used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. Large models can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0036] First, the terms involved in one or more embodiments of this specification are explained.

[0037] Video-Text Retrieval: Traditional image retrieval tasks are only applicable to searches between images, limiting their application scenarios. Video-Text Retrieval enables cross-modal search between natural language and video. Its models are typically trained on large amounts of video-text data pairs.

[0038] CLIP: A multimodal pre-training model. It processes both images and text simultaneously, combining them for tasks such as classification, retrieval, and generation. During training, CLIP utilizes a large amount of image and text data. Its pre-training goal is to learn to align text and image representations so that similar text and images are closer in the embedding space. To achieve this goal, CLIP employs a contrastive learning approach, maximizing the cosine similarity between similar text and images while minimizing the cosine similarity between dissimilar text and images. This results in a trained embedding space that aligns data from different modalities.

[0039] Transformer: A model that primarily uses self-attention to capture long-term dependencies in sequences. The Transformer contains an encoder and a decoder. The encoder is used to generate a representation of the input sequence, while the decoder is used to generate a representation of the target sequence. Both the encoder and decoder consist of multiple layers of self-attention and feedforward neural networks. Furthermore, the self-attention mechanism in the Transformer is a mechanism that calculates the representation of each element in the sequence, helping the model focus on the part of the input sequence that is relevant to the current position and generating an importance weight vector. For example, when a sentence is input, the meaning of each word in the sentence is related to the other words in the sentence.

[0040] Prototype: A representative representation vector. For example, an event can be represented by an event-level vector (prototype).

[0041] Understanding multimodal information is an important way for humans to perceive the world. As a fundamental task in multimodal understanding, video text retrieval has attracted great interest from researchers with the rapid development of short video platforms. Video text retrieval aims to search for semantically related videos based on natural language input by users, which requires appropriate matching modeling between video and text data. However, the inherent modal differences increase the difficulty of associating multimodal data. To address this problem, in practical applications, multiple unimodal pre-trained models are usually used to extract features, and then metric learning strategies are used to strengthen modal alignment in the joint space. However, the initial distributions of multiple unimodal offline features vary greatly, which inevitably brings challenges to feature fusion and affects retrieval results.

[0042] In addition, since videos contain rich visual elements, text descriptions may only correspond to part of the video content, lacking perception of the overall and local content of the video, making the accuracy of video retrieval insufficient to meet user needs and reducing the user experience.

[0043] In this specification, a video retrieval method is provided. This specification also relates to a video retrieval system, a video retrieval apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0044] Referring to FIG1 , FIG1 shows an architecture diagram of a video retrieval system provided by an embodiment of the present specification. The video retrieval system may include a client 100 and a server 200;

[0045] The client 100 is used to send a video retrieval instruction to the server 200;

[0046] The server 200 is configured to receive a video retrieval instruction sent by the client 100; obtain a retrieval text and at least one candidate video in response to the video retrieval instruction; input the retrieval text and the target candidate video into a video matching model to obtain a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on a first matching result and a second matching result between the retrieval text and the target candidate video, the first matching result being used to characterize the matching degree between the retrieval text and each target object in the target candidate video, and the second matching result being used to characterize the matching degree between the retrieval text and the video semantics of the target candidate video; determine at least one target video from the at least one candidate video based on the matching weight corresponding to each candidate video; and send the at least one target video to the client 100;

[0047] The client 100 is further configured to receive at least one target video sent by the server 200 .

[0048] By applying the solution of the embodiments of this specification, a user can trigger a video retrieval instruction through the client and send the video retrieval instruction to the server. After receiving the video retrieval instruction, the server obtains the retrieval text carried in the video retrieval instruction and obtains multiple candidate videos from a preset database. The retrieval text and candidate videos are input into the video matching model to determine the target video associated with the retrieval text among the multiple candidate videos, and the target video is fed back to the client.

[0049] The video retrieval system may include multiple clients 100 and a server 200. The clients 100 may be referred to as client-side devices, and the server 200 may be referred to as cloud-side devices. The multiple clients 100 may establish a communication connection through the server 200. In a video retrieval scenario, the server 200 provides video retrieval services between the multiple clients 100. The multiple clients 100 may act as either senders or receivers, communicating through the server 200.

[0050] Users can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the video retrieval scenario, the user can publish a data stream to the server 200 through the client 100. The server 200 determines at least one target video based on the data stream and pushes the at least one target video to other clients with which communication has been established.

[0051] The client 100 and the server 200 are connected via a network. The network provides a medium for the communication link between the client 100 and the server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, or other processing before being released to the server 200.

[0052] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5, Hypertext Markup Language 5) application, a light application (also known as a mini-program, a lightweight application), or a cloud application. The client 100 can be based on the software development kit (SDK) of the corresponding service provided by the server 200, such as developed based on the real-time communication (RTC) SDK. The client 100 can be deployed in an electronic device and needs to rely on the device to run or certain APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0053] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers that provide background training to support models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server that is integrated with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0054] It is worth noting that the video retrieval method provided in the embodiments of this specification is generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server to execute the video retrieval method provided in the embodiments of this specification. In other embodiments, the video retrieval method provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0055] Referring to FIG. 2 , FIG. 2 shows a flow chart of a video retrieval method provided by an embodiment of this specification, which specifically includes the following steps:

[0056] Step 202: Obtain a search text and at least one candidate video.

[0057] In actual applications, various short video platforms are gradually favored by the public. People often spend time by refreshing and watching videos. However, in the process of watching videos, there are cases where users are not interested in the video content recommended by the short video platform. In this case, users often search for videos in the search bar.

[0058] In this scenario, although various short video platforms will make corresponding video recommendations based on the text entered by the user, there may also be problems such as the videos recommended by the short video platform not matching the text entered by the user's search or the matching degree is low, thereby reducing the user's viewing experience.

[0059] The video retrieval method provided in an embodiment of the present specification can recommend videos that match the search text input by the user to the user.

[0060] The search text refers to the text used to search for videos, and the search text can be a keyword, a phrase, a sentence, or a text paragraph, etc. The candidate video refers to a video stored in a preset database, which is video data for searching videos based on the search text.

[0061] For example, if it is necessary to retrieve videos related to "cooking noodles" from database A, then "cooking noodles" is the search text, and all videos stored in database A are candidate videos.

[0062] Specifically, when a user needs to search for a video, the user can manually enter the search text to be searched in the search bar, or input the search text by voice, thereby generating a video search instruction. The client sends the video search instruction to the server. After receiving the video search instruction, the server obtains the search text carried in the video search instruction and obtains the candidate videos in the preset database.

[0063] For example, if a user wants to obtain videos related to "cooking noodles", he or she can enter the word "cooking noodles" or text related to "cooking noodles" in the search bar of the client. Then, the client sends the video retrieval instruction generated for "cooking noodles" to the server. After receiving the video retrieval instruction, the server obtains the retrieval text "cooking noodles" carried in the video retrieval instruction and obtains multiple candidate videos from the preset database.

[0064] After obtaining the search text and multiple candidate videos, the search text and the multiple candidate videos can be matched to determine the matching degree between the search text and the multiple candidate videos, and based on the matching degree between the search text and the multiple candidate videos, the video that the user needs to retrieve is determined.

[0065] In order to improve the retrieval efficiency of video text retrieval, a video matching model can be used to process the retrieval text and each candidate video.

[0066] Step 204: Input the search text and the target candidate video into a video matching model to obtain a matching weight corresponding to the target candidate video output by the video matching model.

[0067] The target candidate video refers to any one of the candidate videos. The video matching model is used to determine the matching weight between the search text and each candidate video. The matching weight represents the degree of matching between the search text and each candidate video and can be expressed as a numerical value, such as 0.5, 0.8, 50%, 80%, etc.

[0068] Specifically, the retrieved text and target candidate video are input into the video matching model to obtain the matching weight corresponding to the target candidate video output by the video matching model.

[0069] In a specific embodiment provided in this specification, the video matching model includes a coding layer, a first matching layer, and a second matching layer;

[0070] Inputting the search text and the target candidate video into a video matching model, and obtaining a matching weight corresponding to the target candidate video output by the video matching model, including:

[0071] Inputting the search text and the target candidate video into the encoding layer to obtain a text feature sequence of the search text and a target video frame block feature sequence of the target candidate video;

[0072] Inputting the text feature sequence and the target video frame block feature sequence into the first matching layer, obtaining a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence, and determining a first matching result based on the phrase feature sequence and the target object feature sequence;

[0073] The target object feature sequence, the text feature sequence and the first matching result are input into the second matching layer, a second matching result is determined according to the target object feature sequence and the text feature sequence, and a matching weight corresponding to the target candidate video is determined based on the first matching result and the second matching result.

[0074] Among them, the encoding layer is used to encode the search text and each candidate video to obtain a text feature sequence corresponding to the search text; the first matching layer is used to determine the first matching result between the search text and each candidate video, and the first matching result is used to characterize the matching degree between the search text and each target object in each candidate video. The target object may refer to a person, object, etc. in the video frame; the second matching layer is used to determine the second matching result between the search text and each candidate video, and the second matching result is used to characterize the matching degree between the search text and the video semantics of each candidate video.

[0075] Specifically, the retrieval text and the target candidate video are input into the video matching model. First, the retrieval text and the target candidate video need to be input into the encoding layer of the video matching model, and the retrieval text and the target candidate video are encoded respectively to obtain the text feature sequence corresponding to the retrieval text and the target video frame block feature sequence corresponding to the target candidate video.

[0076] A text feature sequence refers to a feature sequence composed of text feature vectors corresponding to the retrieved text, and the text feature vectors include word feature vectors and identifier feature vectors; a target video frame block feature sequence refers to a feature sequence composed of target video frame block feature vectors corresponding to the target candidate video, and the target video frame block feature vectors include key frame block feature vectors and global frame feature vectors.

[0077] Since the retrieved text and the target candidate video belong to data of different modalities, the retrieved text and the target candidate video can be encoded using a text encoder and a video encoder respectively.

[0078] In a specific embodiment provided in this specification, the encoding layer includes a text encoder and a video encoder;

[0079] Inputting the search text and the target candidate video into a video matching model to obtain a text feature sequence of the search text and a target video frame block feature sequence of the target candidate video, including:

[0080] Inputting the search text into the text encoder to obtain a text feature sequence of the search text;

[0081] The target candidate video is input into the video encoder to obtain a target video frame block feature sequence of the target candidate video.

[0082] The text encoder refers to an encoder used to encode the retrieved text, and the video encoder refers to an encoder used to encode the candidate video.

[0083] The search text and the target candidate video are input into the encoding layer of the video matching model, that is, the search text is input into the text encoder, and the target candidate video is input into the video encoder, thereby obtaining the text feature sequence corresponding to the search text and the target video frame block feature sequence corresponding to the target candidate video.

[0084] Specifically, in a specific embodiment provided in this specification, obtaining the text feature sequence of the search text includes:

[0085] Identify the search text and obtain words in the search text;

[0086] Encode each word to obtain the word feature vector corresponding to each word;

[0087] Obtaining preset text identifiers, encoding each preset text identifier, and obtaining an identifier feature vector corresponding to each preset text identifier;

[0088] Based on each word feature vector and each identifier feature vector, a text feature sequence of the search text is obtained.

[0089] Preset text identifiers are pre-set identifiers used to determine the starting and ending positions of the search text. They include the start identifier and the end identifier, which can be represented by [start] and [end]. The identifier feature vector is a vector representation of the start and end identifiers. The identifier feature vector corresponding to the end identifier is also the global feature vector of the search text.

[0090] The search text is segmented and recognized to obtain each word in the search text, and each word is encoded to obtain a word feature vector corresponding to each word, a preset text identifier is obtained, the preset identifier is encoded, and an identifier feature vector corresponding to each preset identifier is obtained, and a text feature sequence is constructed based on each word feature vector and each identifier feature vector.

[0091] For example, for the search text Y i , the text feature sequence obtained based on the above method can be expressed as Among them, y S is the starting identifier feature vector, is the word feature vector, y E is the end identifier feature vector, also the retrieval text Y iThe global feature vector of M is the number of words, and D is the feature dimension, which is usually set to 512 dimensions. This manual does not limit the feature dimension of the feature vector.

[0092] In a specific embodiment provided in this specification, obtaining a target video frame block feature sequence of the target candidate video includes:

[0093] Sampling a preset number of key frames from the target candidate video;

[0094] Segmenting a target key frame to obtain at least one video frame block corresponding to the target key frame, wherein the target key frame is any one of the preset number of key frames;

[0095] Encoding each video frame block to obtain a key frame block feature sequence of the target key frame;

[0096] Obtaining a preset classification embedding symbol, encoding the preset classification embedding symbol, and obtaining a global frame feature vector corresponding to the preset classification embedding symbol;

[0097] Based on the key frame block feature sequence of each key frame and each global frame feature vector, a target video frame block feature sequence of the target candidate video is obtained.

[0098] The preset number refers to the pre-set number of video frames that need to be sampled; the key frame refers to the video frame sampled from the target candidate video; the key frame block feature sequence refers to the feature sequence composed of the feature vectors of each key frame block corresponding to the key frame.

[0099] The preset classification embedding character is a special classification embedding character that is pre-set and inserted before the keyframe. It is used in the classification task and can be specifically represented by [CLS]. The global frame feature vector is the vector representation of the preset classification embedding character.

[0100] A preset number of key frames are obtained and sampled from the target candidate video based on the preset number of key frames. Each sampled key frame is segmented to obtain multiple video frame blocks corresponding to each key frame. Each video frame block is then encoded to obtain a key frame block feature vector corresponding to each video frame block, and a key frame block feature sequence corresponding to each key frame is generated. A preset classification embedding symbol is obtained and encoded to obtain a global frame feature vector corresponding to the preset classification embedding symbol. A target video frame block feature sequence is constructed based on each key frame block feature sequence and each global frame feature vector.

[0101] For example, for the target candidate video X i , the target video frame block feature sequence obtained based on the above method can be expressed as in, is the global frame feature vector, is the key frame block feature vector of the lth frame, is the key frame block feature sequence of the lth frame, K is the number of video frame blocks, L is the preset number, D is the feature dimension, usually set to 512 dimensions, and this manual does not limit the feature dimension of the feature vector.

[0102] Furthermore, after encoding the retrieval text and the target candidate video respectively and obtaining the text feature sequence corresponding to the retrieval text and the target video frame block feature sequence corresponding to the target candidate video, the text feature sequence and the target video frame block feature sequence are input into the first matching layer of the video matching model to obtain the phrase feature sequence corresponding to the text feature sequence and the target object feature sequence corresponding to the target video frame block feature sequence, so that the first matching result can be determined based on the phrase feature sequence and the target object feature sequence.

[0103] In a specific embodiment provided in this specification, the first matching layer includes a spatial prototype generator and a target phrase matcher;

[0104] Inputting the text feature sequence and the target video frame block feature sequence into the first matching layer, obtaining a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence, and determining a first matching result based on the phrase feature sequence and the target object feature sequence, including:

[0105] Inputting the text feature sequence and the target video frame block feature sequence into the spatial prototype generator to obtain a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence;

[0106] The phrase feature sequence and the target object feature sequence are input into the target phrase matcher to obtain a first matching result.

[0107] The spatial prototype generator is used to aggregate each text feature vector in the text feature sequence into a phrase feature vector, and aggregate each target video frame block feature vector in the target video frame block feature sequence into a target object feature vector.

[0108] The target phrase matcher is used to perform feature matching between each target object feature vector and each phrase feature vector.

[0109] The phrase feature sequence refers to a feature sequence composed of each phrase feature vector aggregated from the text feature sequence; the target object feature sequence refers to a feature sequence composed of each target object feature vector aggregated from the target video frame block feature sequence.

[0110] Specifically, the text feature sequence and the target video frame block feature sequence are input into the spatial prototype generator to obtain a phrase feature sequence corresponding to the text feature sequence, and a target object feature sequence corresponding to the target video frame block feature sequence. Furthermore, the obtained phrase feature sequence and target object feature sequence are input into the target phrase matcher to obtain a first matching result that characterizes the matching degree between the search text and each target object in the target candidate video. Specifically, the phrase feature sequence is obtained as follows:

[0111] In a specific embodiment provided in this specification, obtaining a phrase feature sequence corresponding to the text feature sequence includes:

[0112] Determining a predicted text weight of a target text feature vector, wherein the target text feature vector is any one of the text feature vectors in the text feature sequence;

[0113] generating a target phrase feature vector based on the target text feature vector and the predicted text weight;

[0114] Based on the feature vectors of each target phrase, a phrase feature sequence is obtained.

[0115] Since there may be redundant feature vectors in each text feature vector, not all text feature vectors will be aggregated during the process of aggregating each text feature vector. In order to screen out the valid text feature vectors in each text feature vector, the weight corresponding to each text feature vector can be predicted, and the text feature vectors can be aggregated according to the weight corresponding to each text feature vector to generate a phrase feature sequence.

[0116] The target text feature vector refers to any one of the text feature vectors in the text feature sequence. The predicted text weight is used to represent the importance of the text feature vector in obtaining effective information. To express, N p is the number of phrase feature vectors.

[0117] Specifically, any text feature vector is selected as the target text feature vector in the text feature sequence, the predicted text weight of the target text feature vector is determined, and the target phrase feature vector is generated based on the target text feature vector and the predicted text weight of the target text feature vector. Then, a phrase feature sequence is formed based on each generated target phrase feature vector. The phrase feature sequence can be expressed as The generation process of phrase feature sequence can be expressed as

[0118] Furthermore, the target object feature sequence is obtained as follows:

[0119] In a specific embodiment provided in this specification, obtaining a target object feature sequence corresponding to the target video frame block feature sequence includes:

[0120] Determining a prediction frame weight of a first frame block feature vector, wherein the first frame block feature vector is any one of the target video frame block feature vectors in the target video frame block feature sequence;

[0121] generating a first object feature vector based on the first frame block feature vector and the predicted frame weight;

[0122] A target object feature sequence is obtained based on each first object feature vector.

[0123] Since redundant feature vectors may also exist in the feature vectors of each target video frame block, which may even interfere with cross-modal alignment, in the process of aggregating the feature vectors of each target video frame block, not all target video frame block feature vectors will be aggregated. In order to screen out the valid target video frame block feature vectors in each target video frame block feature vector, the weight corresponding to each target video frame block feature vector can be predicted. According to the weight corresponding to each target video frame block feature vector, the feature vectors of each target video frame block are filtered and aggregated to generate a target object feature sequence.

[0124] The first frame block feature vector refers to any one of the target video frame block feature vectors in the target video frame block feature sequence. The predicted frame weight is used to characterize the importance of the target video frame block feature vector in obtaining effective information. To express, N o is the number of target object feature vectors.

[0125] Specifically, any target video frame block feature vector is selected as the first frame block feature vector in the target video frame block feature sequence, the predicted frame weight of the first frame block feature vector is determined, and the first object feature vector is generated based on the first frame block feature vector and the predicted frame weight of the first frame block feature vector. Then, the target object feature sequence is formed based on the generated first object feature vectors. The target object feature sequence can be expressed as The generation process of the target object feature sequence can be expressed as

[0126] Through the above method, redundant text feature vectors and redundant target video frame block feature vectors can be filtered, and the phrase feature sequence corresponding to the text feature sequence and the target object feature sequence corresponding to the target video frame block feature sequence can be aggregated to improve the accuracy of determining the target video.

[0127] After the phrase feature sequence and the target object feature sequence are generated, the phrase feature sequence and the target object feature sequence may be input into a target phrase matcher to obtain a first matching result.

[0128] In a specific implementation provided in this specification, obtaining a first matching result includes:

[0129] Selecting a feature vector of an object to be processed from the target object feature sequence;

[0130] Calculating a first similarity between the feature vector of the object to be processed and each phrase feature vector in the phrase feature sequence, and calculating a second similarity between the feature vector of the object to be processed and each target video frame block feature vector in the target video frame block feature sequence;

[0131] Determine a target first similarity among each first similarity, and determine a target second similarity among each second similarity;

[0132] Determining an initial matching result of the feature vector of the object to be processed based on the first target similarity and the second target similarity;

[0133] Based on the initial matching results of the feature vectors of the objects to be processed, a first matching result is generated.

[0134] The "target object feature vector" refers to any one of the target object feature vectors in the target object feature sequence. The "target first similarity" specifically refers to the maximum similarity among the first similarities, and the "target second similarity" specifically refers to the maximum similarity among the second similarities. The "initial matching result" refers to the matching result between the target object feature vector and the phrase feature vectors, and is used to determine the phrase feature vector with the greatest similarity to the target object feature vector among the phrase feature vectors.

[0135] Specifically, after generating a phrase feature sequence and a target object feature sequence, any target object feature vector in the target object feature sequence is selected for processing, i.e., as the object feature vector to be processed, and a first similarity between the object feature vector to be processed and each phrase feature vector in the phrase feature sequence, and a second similarity between the object feature vector to be processed and each target video frame block feature vector in the target video frame block feature sequence is calculated. The maximum first similarity is determined as the target first similarity among the first similarities between the object feature vector to be processed and each phrase feature vector in the phrase feature sequence, and the maximum second similarity is determined as the target second similarity among the second similarities between the object feature vector to be processed and each target video frame block feature vector in the target video frame block feature sequence. Based on the target first similarity and the target second similarity, a final similarity of the object feature vector to be processed is determined, and the similarity is determined as the initial matching result of the object feature vector to be processed. Furthermore, based on the same method as above, the initial matching results of each target object feature vector in the target object feature sequence are calculated, and the initial matching results are directly added or weighted and added as the first matching result.

[0136] By calculating the first matching result between the search text and the target candidate video, the matching degree between the search text and each target object in the target candidate video can be obtained.

[0137] Furthermore, after obtaining the first matching result between the retrieved text and the target candidate video, the target object feature sequence, the text feature sequence and the first matching result are input into the second matching layer to obtain the second matching result between the retrieved text and the target candidate video, so that the matching weight corresponding to the target candidate video can be determined based on the first matching result and the second matching result.

[0138] In a specific embodiment provided in this specification, the second matching layer includes a temporal prototype generator and a semantic matcher;

[0139] Inputting the target object feature sequence, the text feature sequence, and the first matching result into the second matching layer, and determining a second matching result according to the target object feature sequence and the text feature sequence, including:

[0140] Inputting the target object feature sequence into the temporal prototype generator to obtain a semantic feature sequence corresponding to the target object feature sequence;

[0141] A global feature vector is determined in the text feature sequence, and the semantic feature sequence and the global feature vector are input into the semantic matcher to obtain a second matching result.

[0142] The temporal prototype generator is used to aggregate the target object feature vectors in the target object feature sequence into a semantic feature vector. The semantic feature sequence refers to a feature sequence composed of the semantic feature vectors aggregated from the target object feature sequence. The semantic matcher is used to perform feature matching between the global feature vector and the semantic feature vectors.

[0143] Specifically, the target object feature sequence is input into the temporal prototype generator to obtain a semantic feature sequence corresponding to the target object feature sequence. Furthermore, a global feature vector is determined from the text feature vector, and the global feature vector and the semantic feature sequence are input into the semantic matcher to obtain a second matching result between the search text and the target candidate video. Furthermore, obtaining the semantic feature sequence corresponding to the target object feature sequence is achieved as follows:

[0144] In a specific embodiment provided in this specification, obtaining a semantic feature sequence corresponding to the target object feature sequence includes:

[0145] Decoding the target object feature sequence to obtain a key frame feature sequence;

[0146] Determining the correlation relationship between the key frame feature vectors in the key frame feature sequence;

[0147] Based on the association relationship, generating at least one semantic feature vector;

[0148] Based on each semantic feature vector, a semantic feature sequence is obtained.

[0149] A frame decoder is provided in the temporal prototype generator, which can decode the feature vectors of each target object and generate key frame feature vectors at the frame level. Furthermore, a semantic feature vector can be generated based on each key frame feature vector.

[0150] Among them, the key frame feature sequence refers to the feature sequence composed of key frame feature vectors obtained by decoding the target object feature vector. The key frame feature sequence can be expressed as Semantic feature sequence refers to the feature sequence composed of semantic feature vectors generated by the interaction of each key frame feature vector in the key frame feature sequence. The semantic feature sequence can be expressed as Among them, N e is the number of semantic feature vectors.

[0151] Specifically, a frame decoder is used to decode each target object feature vector in the target object feature sequence, and the spatial relationship between the target object feature vectors is analyzed to obtain each key frame feature vector to form a corresponding key frame feature sequence. Correlations between the key frame feature vectors in the key frame feature sequence are determined, and at least one semantic feature vector is generated based on each correlation. A semantic feature sequence is generated based on the at least one semantic feature vector.

[0152] Furthermore, the attention mechanism can be used to analyze the spatial relationship between the feature vectors of each target object. Specifically, the frame query feature vector Q is randomly initialized f , each target object feature vector P o Perform linear transformation and use the linearly transformed feature vectors as frame key feature vectors K o , and the frame value feature vector V o ,Through mask attention calculation, the spatial relationship between the feature vectors of each target object is obtained, and a key frame feature sequence is generated based on the ,spatial relationship between the feature vectors of each target object.

[0153] Furthermore, the mask attention calculation can be implemented by the following formula (1):

[0154] Among them, P f is the key frame feature vector, Q f is the frame query feature vector, K o is the frame key feature vector, V o is the frame value feature vector, M f is the attention mask, specifically expressed as softmax(.) is a normalized exponential function.

[0155] Similarly, the attention mechanism can be used to dynamically analyze the semantic relationship between the feature vectors of each key frame. Specifically, the semantic query feature vector Q is randomly initialized. e , the feature vector P of each key frame f Perform linear transformation and use the transformed feature vectors as semantic key feature vectors K f , and the semantic value feature vector V f Through dynamic attention calculation, the correlation between the feature vectors of each key frame is obtained, and based on the correlation between the feature vectors of each key frame, a semantic feature sequence is generated. The number of semantic feature vectors can be customized based on the actual application, for example, the number of semantic feature vectors can be set to 2, 3, etc.

[0156] Dynamic attention calculation can be achieved by the following formula (2):

[0157] Among them, P e is the semantic feature vector, Q e is the semantic query feature vector, K f is the semantic key feature vector, V f is the semantic value feature vector, and softmax(.) is the normalized exponential function.

[0158] This method analyzes the spatial relationships between the feature vectors of each target object to generate a keyframe feature sequence. It also analyzes the semantic relationships between the feature vectors of each keyframe to generate a semantic feature sequence. This allows us to determine the semantic diversity of the target candidate videos. In the subsequent retrieval process, the semantics of the candidate videos can be combined with the target video to improve the accuracy of the target video retrieval.

[0159] After generating the semantic feature sequence, it is necessary to match the search text with the video semantics of the target candidate video to obtain a second matching result between the search text and the target candidate video.

[0160] In a specific embodiment provided in this specification, obtaining a second matching result includes:

[0161] Determining the semantic similarity between the global feature vector and each semantic feature vector in the semantic feature sequence;

[0162] Based on the semantic similarity of each semantic feature vector, a second matching result between the search text and the target candidate video is determined.

[0163] Specifically, the semantic similarity between the global feature vector and each semantic feature vector in the semantic feature sequence is calculated, and the maximum semantic similarity is determined among the semantic similarities between the global feature vector and each semantic feature vector, and the maximum semantic similarity is determined as the second matching result.

[0164] By calculating the second matching result between the search text and the target candidate video, the matching degree between the search text and the video semantics of the target candidate video can be obtained.

[0165] After obtaining a first matching result and a second matching result between the search text and the target candidate video, a matching weight of the target candidate video is determined based on the first matching result and the second matching result.

[0166] Specifically, the first matching result and the second matching result can be weighted and calculated, and the calculation result can be determined as the matching weight of the target candidate video. The specific calculation method can be found in the following formula (3): s=s es +βs op(3)

[0167] Among them, s is the matching weight, s es is the second matching result, s op is the first matching result, and β is the spatial matching factor. β can be manually adjusted based on actual application conditions, and the specific numerical setting is not limited in this specification.

[0168] The video retrieval method provided in this specification calculates a first matching result between a search text and a target candidate video to determine the matching degree between the search text and each target object in the target candidate video. It then calculates a second matching result between the search text and the target candidate video to determine the matching degree between the search text and the video semantics of the target candidate video. Furthermore, the matching weight of the target candidate video is determined by combining the first and second matching results. The accuracy of determining the matching weight of the target candidate video is improved by analyzing both the local and overall content of the target candidate video.

[0169] See Figure 3, which shows a schematic diagram of the model architecture of a video matching model provided according to one embodiment of this specification. As shown in Figure 3, the video matching model includes an encoding layer, a first matching layer, and a second matching layer. The encoding layer includes a video encoder and a text encoder, the first matching layer includes a spatial prototype generator and a target phrase matcher, and the second matching layer includes a temporal prototype generator and a semantic matcher.

[0170] Furthermore, in actual application, the search text is input into the text encoder of the encoding layer to obtain the text feature sequence corresponding to the search text, and the target candidate video is input into the video encoder of the encoding layer to obtain the target video frame block feature sequence corresponding to the target candidate video. The text feature sequence and the target video frame block feature sequence are input into the spatial prototype generator of the first matching layer to obtain the phrase feature sequence corresponding to the text feature sequence and the target object feature sequence corresponding to the target video frame block feature sequence. The phrase feature sequence and the target object feature sequence are input into the target phrase matcher of the first matching layer to obtain the first matching result. Furthermore, the target object feature sequence is input into the temporal prototype generator of the second matching layer to obtain the semantic feature sequence corresponding to the target object feature sequence. The global feature vector and semantic feature sequence in the text feature sequence are input into the semantic matcher of the second matching layer to obtain the second matching result. Finally, the second matching layer calculates the matching weight of the target candidate video based on the first matching result and the second matching result. By combining the local content and the overall content of the target candidate video, the accuracy of determining the matching weight of the target candidate video is improved.

[0171] Furthermore, in order to improve the accuracy of the video matching model, the video matching model is trained by the following method:

[0172] Obtaining training data sample pairs and matching weight labels corresponding to the training data sample pairs;

[0173] Inputting the training data sample pairs into the video matching model to obtain predicted matching weights;

[0174] Calculating a model loss value of the video matching model according to the matching weight label and the predicted matching weight;

[0175] The model parameters of the video matching model are adjusted according to the model loss value, and the layout generation model is continuously trained until a training stop condition is reached.

[0176] Among them, the training data sample pair refers to the text-video data pair obtained in the training data sample pair set, which is the training sample of the video matching model, including positive examples of the training data sample pair and negative examples of the training data sample pair; the training data sample pair set refers to the text content in the collected input voice or input text, and the video retrieved based on the text content, composed of the text-video data pair set; the matching weight label refers to the actual matching weight corresponding to the training data sample pair; the predicted matching weight refers to the matching weight output by inputting the training data sample pair into the video matching model; the model loss value refers to the difference between the matching weight label and the predicted matching weight, which is used to measure the difference between the matching weight label and the predicted matching weight.

[0177] Specifically, the text data in the training data sample pair is obtained by the above-mentioned acquisition method of obtaining the retrieved text, and the video obtained based on the text data retrieval and the corresponding text data constitute the positive training data sample pair, and the other text-video data pairs in the same batch in the training data sample pair set constitute the negative training data sample pair. The training data sample pair is input into the video matching model, and the video matching model is used to predict the matching weight of the training data sample pair. At this time, the video matching model is an untrained model. There will be a deviation between the predicted matching weight and the actual matching weight label, and the model parameters of the video matching model need to be adjusted accordingly. Specifically, the model loss value of the video matching model is calculated based on the output predicted matching weight and the matching weight label. The loss function for calculating the model loss value can be a 0-1 loss function, a square loss function, a cross entropy loss function, etc. in actual application. In this specification, preferably, the cross entropy function is selected as the loss function for calculating the model loss value, and the model parameters of the video matching model are adjusted according to the model loss value. The video matching model is continued to be trained based on the adjusted model parameters for the next batch of training data sample pairs until the stop condition of the model training is reached.

[0178] Specifically, the model training stopping conditions include the model loss value being less than a preset threshold and / or the number of training rounds reaching a preset number.

[0179] In a specific implementation provided in this specification, taking the training stop condition of the model loss value being less than a preset threshold as an example, the preset threshold is 0.3. When the model loss value is less than 0.3, the video matching model training is considered to be completed.

[0180] In another specific implementation provided in this specification, taking the preset number of training rounds as the training stop condition as an example, the preset number of training rounds is 30 rounds. When the training rounds of the training data sample pair reach 30 rounds, the video matching model training is considered completed.

[0181] In another specific implementation provided in this specification, two training stop conditions, a preset threshold and a preset training round, are set, and the model loss value and training rounds are monitored at the same time. When either the model loss value or the training rounds meets the training stop condition, the video matching model training is considered completed.

[0182] The video retrieval method provided in this specification analyzes and processes the retrieval text and the target candidate video through a trained video matching model to obtain the matching weight corresponding to the target candidate video, thereby improving the accuracy of determining the matching weight of the target candidate video.

[0183] Step 206: Determine at least one target video from the at least one candidate video based on the matching weight corresponding to each candidate video.

[0184] The target video refers to a video that has a high degree of matching with the search text and can be used to be displayed on the client based on the search text.

[0185] Specifically, after determining the matching weight corresponding to each candidate video, the target video corresponding to the search text can be determined in each candidate video based on the matching weight corresponding to each candidate video.

[0186] In a specific embodiment provided in this specification, determining at least one target video from the at least one candidate video based on the matching weight corresponding to each candidate video includes:

[0187] Determine the candidate video whose matching weight is greater than or equal to a preset matching weight threshold as the target video; or

[0188] The candidate videos are sorted according to their corresponding matching weights to obtain a candidate video list, and a target video is determined in the candidate video list based on a preset number of videos.

[0189] The preset matching weight threshold refers to the preset minimum matching weight value, which is used to measure the matching degree between each candidate video and the search text. The preset number of videos refers to the preset number of target videos that can be selected.

[0190] Specifically, after determining the matching weight corresponding to each candidate video, a preset matching weight threshold can be obtained, and candidate videos whose matching weight is greater than or equal to the preset matching weight threshold can be determined among each candidate video, and the candidate videos whose matching weight is greater than or equal to the preset matching weight threshold can be determined as target videos.

[0191] Furthermore, after determining the matching weight corresponding to each candidate video, a preset number of videos can also be obtained, and then, according to the matching weight corresponding to each candidate video, each candidate video is sorted to obtain a candidate video list, and a preset number of candidate videos are selected from the candidate video list as target videos.

[0192] The implementation method for determining the target video can be set according to actual application conditions and is not limited in this specification.

[0193] The video retrieval method provided in this specification, after obtaining the matching weights corresponding to each candidate video, determines the target video among the candidate videos based on the matching weights corresponding to each candidate video, thereby improving the accuracy of determining the target video.

[0194] Referring to Figure 4a, Figure 4a illustrates a schematic diagram of an interactive interface for a video retrieval method according to one embodiment of this specification. As shown in Figure 4a, a user can enter search text in the search bar of a client, thereby generating a video retrieval instruction. Upon receiving the video retrieval instruction, the server utilizes the aforementioned video retrieval method to identify at least one target video in a preset database, and then feeds the target video back to the client for display. The display interface may be as shown in Figure 4a.

[0195] Referring to Figure 4b, Figure 4b illustrates a schematic diagram of an interactive interface for another video retrieval method provided according to one embodiment of the present disclosure. As shown in Figure 4b, a user can enter search text in the search bar of a client, thereby generating a video retrieval instruction. Upon receiving the video retrieval instruction, the server utilizes the aforementioned video retrieval method to identify at least one target video in a preset database, and feeds the target video back to the client for display. The display interface may be as shown in Figure 4b.

[0196] It should be noted that the way in which the user inputs the search text is not limited to text input, and voice input may also be used. FIG. 4 a and FIG. 4 b are only exemplary illustrations.

[0197] The video retrieval method provided in this specification includes: obtaining a retrieval text and at least one candidate video; inputting the retrieval text and the target candidate video into a video matching model, and obtaining a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on a first matching result and a second matching result between the retrieval text and the target candidate video, the first matching result being used to characterize the matching degree between the retrieval text and each target object in the target candidate video, and the second matching result being used to characterize the matching degree between the retrieval text and the video semantics of the target candidate video; based on the matching weight corresponding to each candidate video, at least one target video is determined in the at least one candidate video.

[0198] One embodiment of this specification calculates a first matching result between a search text and a target candidate video to determine the matching degree between the search text and each target object in the target candidate video. A second matching result is then calculated between the search text and the target candidate video to determine the matching degree between the search text and the video semantics of the target candidate video. Furthermore, the matching weight of the target candidate video is determined by combining the first and second matching results. The accuracy of determining the matching weight of the target candidate video is improved by analyzing the local and overall content of the target candidate video. Finally, based on the matching weight corresponding to each candidate video, the target video is determined from the candidate videos, thereby improving the accuracy of text-based video retrieval.

[0199] The following further describes the video retrieval method provided in this specification using the application of the video retrieval method provided in this specification in text-based video retrieval as an example, in conjunction with FIG5 . FIG5 shows a flowchart of the processing process of a video retrieval method provided in one embodiment of this specification, which specifically includes the following steps:

[0200] Step 502: Obtain the search text "cook noodles" and at least one candidate video.

[0201] Step 504: Input the search text "cook noodles" into the text encoder of the video matching model to obtain the text feature sequence of the search text "cook noodles", input the target candidate video into the video encoder of the video matching model to obtain the target video frame block feature sequence of the target candidate video.

[0202] Step 506: Input the text feature sequence and the target video frame block feature sequence into the spatial prototype generator of the video matching model to obtain a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence.

[0203] Step 508: Input the phrase feature sequence and the target object feature sequence into the target phrase matcher of the video matching model to obtain a first matching result.

[0204] Step 510: Input the target object feature sequence into the temporal prototype generator of the video matching model to obtain a semantic feature sequence corresponding to the target object feature sequence.

[0205] Step 512: Determine a global feature vector in the text feature sequence, input the semantic feature sequence and the global feature vector into the semantic matcher of the video matching model, and obtain a second matching result.

[0206] Step 514: Determine a matching weight corresponding to the target candidate video based on the first matching result and the second matching result.

[0207] Step 516: Determine at least one target video from the at least one candidate video based on the matching weight corresponding to each candidate video.

[0208] In one embodiment of this specification, a first matching result between the search text "cook noodles" and a target candidate video is calculated to determine the matching degree between the search text "cook noodles" and each target object in the target candidate video. A second matching result between the search text "cook noodles" and the target candidate video is then calculated to determine the matching degree between the search text "cook noodles" and the video semantics of the target candidate video. Furthermore, the matching weight of the target candidate video is determined by combining the first and second matching results. The accuracy of determining the matching weight of the target candidate video is improved by analyzing the local and overall content of the target candidate video. Finally, based on the matching weights corresponding to each candidate video, the target video is determined from the candidate videos, thereby improving the accuracy of text-based video retrieval.

[0209] Corresponding to the above method embodiment, this specification also provides a video retrieval device embodiment. FIG6 shows a schematic diagram of the structure of a video retrieval device provided by one embodiment of this specification. As shown in FIG6, the device includes:

[0210] An acquisition module 602 is configured to acquire a search text and at least one candidate video;

[0211] An input module 604 is configured to input the search text and the target candidate video into a video matching model, and obtain a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on a first matching result and a second matching result between the search text and the target candidate video, wherein the first matching result is used to represent the matching degree between the search text and each target object in the target candidate video, and the second matching result is used to represent the matching degree between the search text and the video semantics of the target candidate video;

[0212] The determination module 606 is configured to determine at least one target video from the at least one candidate video based on the matching weight corresponding to each candidate video.

[0213] Optionally, the video matching model includes a coding layer, a first matching layer, and a second matching layer;

[0214] The input module is further configured to:

[0215] Inputting the search text and the target candidate video into the encoding layer to obtain a text feature sequence of the search text and a target video frame block feature sequence of the target candidate video;

[0216] Inputting the text feature sequence and the target video frame block feature sequence into the first matching layer, obtaining a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence, and determining a first matching result based on the phrase feature sequence and the target object feature sequence;

[0217] The target object feature sequence, the text feature sequence and the first matching result are input into the second matching layer, a second matching result is determined according to the target object feature sequence and the text feature sequence, and a matching weight corresponding to the target candidate video is determined based on the first matching result and the second matching result.

[0218] Optionally, the encoding layer includes a text encoder and a video encoder;

[0219] The input module is further configured to:

[0220] Inputting the search text into the text encoder to obtain a text feature sequence of the search text;

[0221] The target candidate video is input into the video encoder to obtain a target video frame block feature sequence of the target candidate video.

[0222] Optionally, the first matching layer includes a spatial prototype generator and a target phrase matcher;

[0223] The input module is further configured to:

[0224] Inputting the text feature sequence and the target video frame block feature sequence into the spatial prototype generator to obtain a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence;

[0225] The phrase feature sequence and the target object feature sequence are input into the target phrase matcher to obtain a first matching result.

[0226] Optionally, the input module is further configured to:

[0227] Determining a predicted text weight of a target text feature vector, wherein the target text feature vector is any one of the text feature vectors in the text feature sequence;

[0228] generating a target phrase feature vector based on the target text feature vector and the predicted text weight;

[0229] Based on the feature vectors of each target phrase, a phrase feature sequence is obtained.

[0230] Optionally, the input module is further configured to:

[0231] Determining a prediction frame weight of a first frame block feature vector, wherein the first frame block feature vector is any one of the target video frame block feature vectors in the target video frame block feature sequence;

[0232] generating a first object feature vector based on the first frame block feature vector and the predicted frame weight;

[0233] A target object feature sequence is obtained based on each first object feature vector.

[0234] Optionally, the input module is further configured to:

[0235] Selecting a feature vector of an object to be processed from the target object feature sequence;

[0236] Calculating a first similarity between the feature vector of the object to be processed and each phrase feature vector in the phrase feature sequence, and calculating a second similarity between the feature vector of the object to be processed and each target video frame block feature vector in the target video frame block feature sequence;

[0237] Determine a target first similarity among each first similarity, and determine a target second similarity among each second similarity;

[0238] Determining an initial matching result of the feature vector of the object to be processed based on the first target similarity and the second target similarity;

[0239] Based on the initial matching results of the feature vectors of the objects to be processed, a first matching result is generated.

[0240] Optionally, the second matching layer includes a temporal prototype generator and a semantic matcher;

[0241] The input module is further configured to:

[0242] Inputting the target object feature sequence into the temporal prototype generator to obtain a semantic feature sequence corresponding to the target object feature sequence;

[0243] A global feature vector is determined in the text feature sequence, and the semantic feature sequence and the global feature vector are input into the semantic matcher to obtain a second matching result.

[0244] Optionally, the input module is further configured to:

[0245] Decoding the target object feature sequence to obtain a key frame feature sequence;

[0246] Determining the correlation relationship between the key frame feature vectors in the key frame feature sequence;

[0247] Based on the association relationship, generating at least one semantic feature vector;

[0248] Based on each semantic feature vector, a semantic feature sequence is obtained.

[0249] Optionally, the input module is further configured to:

[0250] Determining the semantic similarity between the global feature vector and each semantic feature vector in the semantic feature sequence;

[0251] Based on the semantic similarity of each semantic feature vector, a second matching result between the search text and the target candidate video is determined.

[0252] Optionally, the determining module is further configured to:

[0253] Determine the candidate video whose matching weight is greater than or equal to a preset matching weight threshold as the target video; or

[0254] The candidate videos are sorted according to their corresponding matching weights to obtain a candidate video list, and a target video is determined in the candidate video list based on a preset number of videos.

[0255] Optionally, the device further includes a training module configured to:

[0256] Obtaining training data sample pairs and matching weight labels corresponding to the training data sample pairs;

[0257] Inputting the training data sample pairs into the video matching model to obtain predicted matching weights;

[0258] Calculating a model loss value of the video matching model according to the matching weight label and the predicted matching weight;

[0259] The model parameters of the video matching model are adjusted according to the model loss value, and the layout generation model is continuously trained until a training stop condition is reached.

[0260] The video retrieval device provided in this specification includes: an acquisition module, configured to acquire a retrieval text and at least one candidate video; an input module, configured to input the retrieval text and the target candidate video into a video matching model, and obtain a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on a first matching result and a second matching result between the retrieval text and the target candidate video, the first matching result being used to characterize the matching degree between the retrieval text and each target object in the target candidate video, and the second matching result being used to characterize the matching degree between the retrieval text and the video semantics of the target candidate video; a determination module, configured to determine at least one target video in the at least one candidate video based on the matching weight corresponding to each candidate video.

[0261] One embodiment of this specification calculates a first matching result between a search text and a target candidate video to determine the matching degree between the search text and each target object in the target candidate video. A second matching result is then calculated between the search text and the target candidate video to determine the matching degree between the search text and the video semantics of the target candidate video. Furthermore, the matching weight of the target candidate video is determined by combining the first and second matching results. The accuracy of determining the matching weight of the target candidate video is improved by analyzing the local and overall content of the target candidate video. Finally, based on the matching weight corresponding to each candidate video, the target video is determined from the candidate videos, thereby improving the accuracy of text-based video retrieval.

[0262] The above is a schematic scheme of a video retrieval device of this embodiment. It should be noted that the technical scheme of the video retrieval device and the technical scheme of the above-mentioned video retrieval method are of the same concept. For details not described in detail in the technical scheme of the video retrieval device, please refer to the description of the technical scheme of the above-mentioned video retrieval method.

[0263] Figure 7 shows a block diagram of a computing device 700 according to one embodiment of this specification. Components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0264] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, or a near field communication (NFC) interface.

[0265] In one embodiment of the present specification, the aforementioned components of computing device 700 and other components not shown in FIG. 7 may also be connected to one another, for example, via a bus. It should be understood that the computing device block diagram shown in FIG. 7 is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0266] Computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 700 may also be a mobile or stationary server.

[0267] The processor 720 is configured to execute the following computer-executable instructions, which implement the steps of the above-mentioned video retrieval method when executed by the processor.

[0268] The above is a schematic solution of a computing device of this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above-mentioned video retrieval method are of the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-mentioned video retrieval method.

[0269] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which implement the steps of the above-mentioned video retrieval method when executed by a processor.

[0270] The above is a schematic solution of a computer-readable storage medium of this embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the above-mentioned video retrieval method are based on the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above-mentioned video retrieval method.

[0271] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned video retrieval method.

[0272] The above is an illustrative solution of a computer program of this embodiment. It should be noted that the technical solution of the computer program and the technical solution of the above-mentioned video retrieval method are of the same concept. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the above-mentioned video retrieval method.

[0273] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0274] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium.

[0275] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0276] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0277] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A video retrieval method, comprising: Obtaining a search text and at least one candidate video; Input the search text and the target candidate video into a video matching model, and obtain a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on a first matching result and a second matching result between the search text and the target candidate video, wherein the first matching result is used to characterize the matching degree between the search text and each target object in the target candidate video, and the second matching result is used to characterize the matching degree between the search text and the video semantics of the target candidate video; At least one target video is determined from the at least one candidate video based on the matching weights corresponding to the candidate videos.

2. The method of claim 1, wherein the video matching model comprises a coding layer, a first matching layer, and a second matching layer; Inputting the search text and the target candidate video into a video matching model, and obtaining a matching weight corresponding to the target candidate video output by the video matching model, comprises: Inputting the search text and the target candidate video into the coding layer to obtain a text feature sequence of the search text and a target video frame block feature sequence of the target candidate video; Inputting the text feature sequence and the target video frame block feature sequence into the first matching layer, obtaining a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence, and determining a first matching result according to the phrase feature sequence and the target object feature sequence; The target object feature sequence, the text feature sequence and the first matching result are input into the second matching layer, a second matching result is determined according to the target object feature sequence and the text feature sequence, and a matching weight corresponding to the target candidate video is determined based on the first matching result and the second matching result.

3. The method of claim 2, wherein the encoding layer comprises a text encoder and a video encoder; Inputting the search text and the target candidate video into a video matching model to obtain a text feature sequence of the search text and a target video frame block feature sequence of the target candidate video, including: Inputting the search text into the text encoder to obtain a text feature sequence of the search text; The target candidate video is input into the video encoder to obtain a target video frame block feature sequence of the target candidate video.

4. The method of claim 2, wherein the first matching layer comprises a spatial prototype generator and a target phrase matcher; The text feature sequence and the target video frame block feature sequence are input into the first matching layer to obtain the phrase feature sequence corresponding to the text feature sequence and the target video frame block feature sequence. The target object feature sequence corresponding to the phrase feature sequence is determined, and a first matching result is determined according to the phrase feature sequence and the target object feature sequence, including: Inputting the text feature sequence and the target video frame block feature sequence into the spatial prototype generator to obtain a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence; The phrase feature sequence and the target object feature sequence are input into the target phrase matcher to obtain a first matching result.

5. The method according to claim 4, obtaining a phrase feature sequence corresponding to the text feature sequence, comprising: Determine the predicted text weight of the target text feature vector, where The target text feature vector is any one of the text feature vectors in the text feature sequence; Generate a target phrase feature vector based on the target text feature vector and the predicted text weight; Based on each target phrase feature vector, a phrase feature sequence is obtained.

6. The method according to claim 4, obtaining the target object feature sequence corresponding to the target video frame block feature sequence, comprising: Determine a prediction frame weight of a first frame block feature vector, wherein the first frame block feature vector is any one of the target video frame block feature vectors in the target video frame block feature sequence; generating a first object feature vector based on the first frame block feature vector and the predicted frame weight; Based on each first object feature vector, a target object feature sequence is obtained.

7. The method of claim 4, obtaining the first matching result, comprising: Selecting a feature vector of an object to be processed from the target object feature sequence; Calculating a first similarity between the feature vector of the object to be processed and each phrase feature vector in the phrase feature sequence, and calculating a second similarity between the feature vector of the object to be processed and each target video frame block feature vector in the target video frame block feature sequence; Determine a target first similarity among each first similarity, and determine a target second similarity among each second similarity; Determining an initial matching result of the feature vector of the object to be processed based on the first target similarity and the second target similarity; Based on the initial matching results of the feature vectors of the objects to be processed, a first matching result is generated.

8. The method according to claim 2, wherein the second matching layer comprises a temporal prototype generator and a semantic matcher; Inputting the target object feature sequence, the text feature sequence, and the first matching result into the second matching layer, and determining a second matching result according to the target object feature sequence and the text feature sequence, comprising: Inputting the target object feature sequence into the temporal prototype generator to obtain a semantic feature sequence corresponding to the target object feature sequence; Determine a global feature vector in the text feature sequence, and combine the semantic feature sequence and the The global feature vector is input into the semantic matcher to obtain a second matching result.

9. The method according to claim 8, wherein obtaining the semantic feature sequence corresponding to the target object feature sequence comprises: Decoding the target object feature sequence to obtain a key frame feature sequence; Determining the correlation relationship between the key frame feature vectors in the key frame feature sequence; Based on the association relationship, generating at least one semantic feature vector; Based on each semantic feature vector, a semantic feature sequence is obtained.

10. The method of claim 8, obtaining the second matching result, comprising: Determining the semantic similarity between the global feature vector and each semantic feature vector in the semantic feature sequence; Based on the semantic similarity of each semantic feature vector, a second matching result between the search text and the target candidate video is determined.

11. The method according to claim 1, determining at least one target video from the at least one candidate video based on the matching weights corresponding to the candidate videos, comprises: Determine the candidate video whose matching weight is greater than or equal to the preset matching weight threshold as the target video; or The candidate videos are sorted according to the matching weights corresponding to the candidate videos to obtain a candidate video list, and a target video is determined in the candidate video list based on a preset number of videos.

12. The method according to claim 1, wherein the video matching model is trained by the following method: Obtaining training data sample pairs and matching weight labels corresponding to the training data sample pairs; Inputting the training data sample pairs into the video matching model to obtain predicted matching weights; Calculating a model loss value of the video matching model according to the matching weight label and the predicted matching weight; The model parameters of the video matching model are adjusted according to the model loss value, and the layout generation model is continuously trained until a training stop condition is reached.

13. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the video retrieval method according to any one of claims 1 to 12 are implemented.

14. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the video retrieval method according to any one of claims 1 to 12.

Images