A video retrieval method, apparatus, device, storage medium, and program product.

Abstract

This application provides a video retrieval method, apparatus, device, computer-readable storage medium, and computer program product. The method includes: performing multi-task extraction processing on a video to be retrieved to obtain global features, content category, and channel category of the video to be retrieved; retrieving multiple reference videos from a first video set based on the global features and content category of the video to be retrieved to form a second video set; determining the modal features of the video to be retrieved based on the channel category of the video to be retrieved, and retrieving a target video from the multiple reference videos in the second video set based on the modal features of the video to be retrieved, so as to use the target video as the retrieval result of the video to be retrieved. This application can improve the efficiency and accuracy of video retrieval.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a video retrieval method, apparatus, device, computer-readable storage medium, and computer program product based on artificial intelligence. Background Technology

[0002] Artificial Intelligence (AI) is a comprehensive technology within computer science that studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. AI technology is a multidisciplinary field encompassing a wide range of areas, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in more fields and play an increasingly important role.

[0003] Artificial intelligence technology is widely used in video retrieval systems. For example, a common method for authorized video retrieval or recommended video retrieval is to use the image feature descriptions of multiple image frames of the video to be retrieved and the reference video to determine the similarity. When two videos have enough similar image frames, it indicates that the two videos are duplicates or similar. However, this method only considers the image feature descriptions, which leads to insufficient consideration at the feature level, resulting in low retrieval accuracy based on direct image feature retrieval. Summary of the Invention

[0004] This application provides a video retrieval method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the efficiency and accuracy of video retrieval.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a video retrieval method, including:

[0007] Multi-task extraction processing is performed on the video to be retrieved to obtain the global features, content category, and channel category of the video to be retrieved;

[0008] Based on the global features and content categories of the video to be retrieved, multiple reference videos are retrieved from the first video set to form a second video set;

[0009] Based on the channel category of the video to be retrieved, the modal features of the video to be retrieved are determined, and based on the modal features of the video to be retrieved, the target video is retrieved from multiple reference videos in the second video set, so as to use the target video as the retrieval result of the video to be retrieved.

[0010] This application provides a video retrieval device, including:

[0011] The task extraction module is used to perform multi-task extraction processing on the video to be retrieved, and obtain the global features, content category and channel category of the video to be retrieved;

[0012] The first retrieval module is used to retrieve multiple reference videos from the first video set based on the global features and content categories of the video to be retrieved, so as to form a second video set;

[0013] The second retrieval module is used to determine the modal features of the video to be retrieved based on the channel category of the video to be retrieved, and to retrieve the target video from multiple reference videos in the second video set based on the modal features of the video to be retrieved, so as to use the target video as the retrieval result of the video to be retrieved.

[0014] In the above scheme, the multi-task extraction processing of the video to be retrieved is implemented through a multi-task model, which includes a basic feature layer, a feature embedding layer, a content classification layer, and a channel classification layer. The task extraction module is further used to extract basic features of the video to be retrieved through the basic feature layer to obtain the basic features of the video to be retrieved; to perform embedding vector transformation processing on the basic features of the video to be retrieved through the feature embedding layer to obtain the global features of the video to be retrieved; to perform content classification processing on the basic features of the video to be retrieved through the content classification layer to obtain the content category of the video to be retrieved; and to perform channel classification processing on the basic features of the video to be retrieved through the channel classification layer to obtain the channel category of the video to be retrieved.

[0015] In the above scheme, the basic feature layer includes a pooling layer and multiple cascaded convolutional layers, and the video to be retrieved includes multiple image frames. The task extraction module is further configured to perform cascaded convolution processing on each of the image frames through the multiple cascaded convolutional layers to obtain the convolutional features of each of the image frames; perform pooling processing on the convolutional features of each of the image frames through the pooling layer to obtain the image features of each of the image frames; and perform concatenation processing on the image features of each of the image frames to obtain the basic features of the video to be retrieved.

[0016] In the above scheme, the device further includes: a model training module, used to extract basic features from video samples through the basic feature layer to obtain the basic features of the video samples; to perform embedding vector transformation processing on the basic features of the video samples through the feature embedding layer to obtain the global features of the video samples; to perform content classification processing on the basic features of the video samples through the content classification layer to obtain the content category of the video samples; to perform channel classification processing on the basic features of the video samples through the channel classification layer to obtain the channel category of the video samples; to construct the target loss function of the multi-task model based on the global features of the video samples, the content category of the video samples, and the channel category of the video samples, and to update the model parameters of the multi-task model based on the target loss function.

[0017] In the above scheme, the model training module is further configured to: construct an embedding loss function for the multi-task model based on the global features of the video samples, the global features of the positive video samples in the video samples, and the global features of the negative video samples in the video samples; construct a content classification loss function for the multi-task model based on the content category and the content category label of the video samples; construct a channel classification loss function for the multi-task model based on the channel category and the channel category label of the video samples; and perform a weighted summation of the embedding loss function, the content classification loss function, and the channel classification loss function to obtain the target loss function of the multi-task model.

[0018] In the above scheme, the model training module is further configured to perform similarity matching between the global features of the video sample and the global features of the positive video samples in the video sample to obtain a first similarity between the video sample and the positive video sample; perform similarity matching between the global features of the video sample and the global features of the negative video samples in the video sample to obtain a second similarity between the video sample and the negative video sample; and construct the embedding loss function of the multi-task model based on the first similarity and the second similarity.

[0019] In the above scheme, before constructing the embedding loss function of the multi-task model based on the global features of the video samples, the global features of the positive video samples in the video samples, and the global features of the negative video samples in the video samples, the apparatus further includes: a sample construction module, used to extract at least one candidate video sample from multiple candidate positive video sample pairs, wherein the multiple candidate positive video sample pairs do not include the video sample; obtain the similarity between each candidate video sample and the video sample, and sort the corresponding candidate video samples in descending order of the similarity; and take the candidate video samples ranked later in the sorting result as the negative video samples in the video samples.

[0020] In the above scheme, the first video set includes multiple videos, and each video includes multiple reference image frames; the video to be retrieved includes multiple image frames, the global features of the video to be retrieved include the global features of each image frame, and the content category of the video to be retrieved includes the content category of each image frame; the first retrieval module is further configured to perform the following processing for each image frame: searching for the global features of a target number of reference image frames from the global feature index table of the video index corresponding to the first video set; obtaining the Euclidean distance between the global features of each reference image frame and the global features of the image frame in the target number of reference image frames; determining the distance threshold corresponding to the content category of the image frame, and selecting multiple reference image frames from the target number of reference image frames whose Euclidean distance is less than the distance threshold; and retrieving multiple reference videos from the multiple videos in the first video set based on the selected multiple reference image frames.

[0021] In the above scheme, the first retrieval module is further configured to: search for the video identifier of the video to which each reference image frame belongs from the identifier index table of the video index corresponding to the first video set; select the video corresponding to each video identifier as a candidate video from the multiple videos of the first video set; obtain the selection duration corresponding to the selected reference image frame in each candidate video, and the total duration of each candidate video; obtain a first ratio of each selection duration to the total duration, and retrieve the candidate video whose first ratio is greater than a ratio threshold from the multiple candidate videos of the first video set as the reference video.

[0022] In the above scheme, the device further includes: a video set creation module, used to obtain the matching duration of the image frames that match the multiple image frames of the video to be retrieved and the duration of the video to be retrieved; obtain a second ratio of each matching duration to the duration of the video to be retrieved; and sort each of the reference videos in descending order based on the second ratio to obtain a reference video sequence to form the second video set.

[0023] In the above scheme, the device further includes: a video index construction module, used to initialize a global feature index table, a time index table, and an identifier index table for the video index corresponding to the first video set; and to perform the following processing for each reference image frame: obtaining the global features of the reference image frame, the time point of the reference image frame in the corresponding video, and the video identifier of the video to which the reference image frame belongs; adding the global features of the reference image frame to the global feature index table, adding the time point of the reference image frame in the corresponding video to the time index table, and adding the video identifier of the video to which the reference image frame belongs to the identifier index table, so as to construct the video index corresponding to the first video set.

[0024] In the above scheme, the second retrieval module is further configured to: when the channel category indicates that the video to be retrieved is a text category, perform text feature extraction processing on the text data of the video to be retrieved to obtain the text features of the video to be retrieved; when the channel category indicates that the video to be retrieved is an audio category, perform audio feature extraction processing on the audio data of the video to be retrieved to obtain the audio features of the video to be retrieved; when the channel category indicates that the video to be retrieved is a multi-target category, perform local feature extraction processing on the multi-targets in the video to be retrieved to obtain the local features of the video to be retrieved; and determine at least one of the text features, the audio features, and the local features as the modal features of the video to be retrieved.

[0025] In the above scheme, the second retrieval module is further configured to perform similarity matching between the modal features of the video to be retrieved and the modal features of each of the reference videos in the second video set, and obtain the corresponding similarity values; and retrieve the reference videos with similarity values ​​greater than the similarity threshold from the multiple reference videos in the second video set as the target videos.

[0026] In the above scheme, after retrieving multiple reference videos from the first video set, the device further includes: a third retrieval module, used to extract local features from the video to be retrieved when the channel category indicates that the video to be retrieved is a multi-target category, to obtain local features of the video to be retrieved; based on the local features, to retrieve multiple candidate target videos from the multiple reference videos in the second video set to form a third video set; the second retrieval module is also used to retrieve target videos from the multiple candidate target videos in the third video set.

[0027] In the above scheme, the device further includes: a feedback output module, used to combine the search results and the channel category to output feedback information for the video to be searched; wherein the feedback information is used to provide recommendation information or authorization prompt information for the video to be searched.

[0028] This application provides an electronic device, including:

[0029] Memory, used to store executable instructions;

[0030] The processor, when executing executable instructions stored in the memory, implements the video retrieval method provided in the embodiments of this application.

[0031] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the video retrieval method provided in this application.

[0032] This application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the video retrieval method provided in this application.

[0033] The embodiments of this application have the following beneficial effects:

[0034] By applying the embodiments of this application, when searching for videos, the single multi-task extraction process for the videos enables the rapid extraction of multiple features required for retrieval, such as global features, content categories, and channel categories, reducing the workload caused by multiple feature engineering processes required at different stages of retrieval. When forming a second video set from the first video set, the global features and content categories of the videos to be retrieved are considered, enabling coarse retrieval or coarse filtering of videos, which not only improves retrieval and filtering efficiency but also enhances the applicability of retrieval for videos with different content categories. When retrieving target videos from the second video set, considering the different modal features corresponding to different channel categories, the modal features corresponding to the videos to be retrieved are used to achieve fine retrieval of the videos to be retrieved, thereby improving the accuracy of the final retrieval results. Attached Figure Description

[0035] Figure 1 This is a schematic diagram illustrating an application scenario of the video retrieval system 10 provided in this application embodiment;

[0036] Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application;

[0037] Figure 3 A flowchart illustrating the video retrieval method provided in this application embodiment;

[0038] Figure 4 A flowchart illustrating the video retrieval method provided in this application embodiment;

[0039] Figure 5 A schematic diagram of the basic feature layer provided in the embodiments of this application;

[0040] Figure 6 A schematic diagram illustrating the training method of the multi-task model provided in an embodiment of this application;

[0041] Figure 7 A schematic diagram illustrating the training method of the multi-task model provided in an embodiment of this application;

[0042] Figure 8 A schematic diagram illustrating the construction of the loss function provided in an embodiment of this application;

[0043] Figure 9 A schematic diagram of the first video set provided in an embodiment of this application;

[0044] Figure 10 This is a schematic diagram of the architecture of the video retrieval system provided in the embodiments of this application;

[0045] Figure 11 This is a schematic diagram of the architecture of the video retrieval system provided in the embodiments of this application;

[0046] Figure 12 This is a schematic diagram of the architecture of the video retrieval system provided in an embodiment of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0049] In the following description, the terms “first, second…” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second…” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0051] In the implementation of this application, the collection and processing of relevant data should be strictly in accordance with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0052] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0053] 1) Image recognition: This refers to the use of computers to process, analyze, and understand images in order to identify targets and objects of various patterns. It is a practical application of deep learning algorithms. Image recognition is a category-level recognition method that does not consider specific instances of objects, but only identifies the object based on its category (such as human, dog, cat, bird, etc.) and determines the category to which the object belongs. For example, the recognition task in the large-scale general object recognition open-source dataset (ImageNet) identifies which of 1000 categories a given object belongs to.

[0054] 2) Image multi-label recognition: This involves using a computer to identify whether an image has a combination of specified attribute labels. An image may have multiple attributes, and the multi-label recognition task is to determine which preset attribute labels an image possesses.

[0055] 3) ImageNet: A large-scale, general-purpose open-source dataset for object recognition.

[0056] 4) ImageNet pre-trained model: A deep learning network model is trained based on ImageNet, and the parameter weights of this model are the ImageNet pre-trained model.

[0057] 5) Video recognition: Identify the channel (or category) of the video. For example, video channels include: concert, variety show, game, live streaming e-commerce, short videos of daily life, other categories, etc. More specifically, when a video mainly describes the scene of a concert, the video channel of the video can be determined as the concert channel (or concert category).

[0058] 6) Image deduplication retrieval: The task of confirming whether two images are the same or that one image can be transformed into another image is called image deduplication recognition. The task of searching the image database to see if a query image (or the image to be retrieved) is a duplicate of an image in the database is called image deduplication retrieval.

[0059] 7) Video Deduplication Search: To save storage space under massive video storage conditions, or to determine whether a user-uploaded video poses a copyright threat to the existing video library (only videos that do not pose a copyright threat can be authorized), it is necessary to exclude or identify duplicate videos or video segments. Therefore, video deduplication identification requires confirming whether two videos are the same or similar, or whether two videos contain the same or similar sub-video segments. Video deduplication search is the process of finding, from the video library, whether a video that is a duplicate of the query video (the video to be searched) or contains duplicate sub-video segments.

[0060] 8) Video Deduplication Search: An application of video deduplication search. For videos that require copyright protection, these videos are stored in the video deduplication search inventory. For the query video (the video to be searched), the video deduplication search process searches the inventory to see if there is a video that is duplicated with it. If there is a video that is duplicated with it in the inventory, it means that the query video (the video to be searched) poses a version threat to the duplicate video in the inventory, and therefore cannot obtain authorization.

[0061] The electronic device for video retrieval provided in this application can be various types of terminals or servers. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0062] Taking servers as an example, such as server clusters deployed in the cloud, AI as a Service (AIaaS) is offered to users. The AIaaS platform breaks down several common AI services and provides them as independent or packaged services in the cloud. This service model is similar to an AI-themed marketplace, where all users can access and use one or more AI services provided by the AIaaS platform through application programming interfaces.

[0063] For example, one type of AI cloud service could be a video retrieval service, where a cloud server encapsulates the video retrieval program provided in this application embodiment. Users invoke the video retrieval service in the cloud service via a terminal (running a client, such as a video retrieval client or video recognition client), causing the cloud-deployed server to invoke the encapsulated video retrieval program to perform multi-task extraction processing on the video to be retrieved, obtaining the global features, content category, and channel category of the video to be retrieved. Based on the global features and content category of the video to be retrieved, a first-level retrieval system retrieves multiple reference videos from a first video set to form a second video set. A second-level retrieval system determines the modal features of the video to be retrieved based on its channel category, and retrieves the target video from the multiple reference videos in the second video set based on these modal features. The target video is then used as the retrieval result for the video to be retrieved. While improving retrieval and filtering efficiency through the first-level retrieval system, the second-level retrieval system also improves the accuracy of the final retrieval results, enhancing the accuracy of subsequent video recommendations or video authorization decisions based on the retrieval results.

[0064] See Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the video retrieval system 10 provided in this application embodiment. The terminal 200 connects to the server 100 via a network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both. The terminal (running a client, such as a video retrieval client or a video recognition client) can be used to obtain retrieval requests for the image to be retrieved. For example, when a user opens the client running on the terminal and selects the video to be retrieved, the terminal automatically obtains the retrieval request for the video.

[0065] In some embodiments, a video retrieval plugin can be embedded in the client running on the terminal to implement a video retrieval method locally on the client. For example, the terminal 200 calls the video retrieval plugin to implement a video retrieval method, which involves performing multi-task extraction processing on the video to be retrieved to obtain its global features, content category, and channel category; based on the global features and content category of the video to be retrieved, a first-level retrieval system retrieves multiple reference videos from a first video set to form a second video set; a second-level retrieval system determines the modal features of the video to be retrieved based on its channel category, and based on the modal features of the video to be retrieved, retrieves target videos similar to the video to be retrieved from the multiple reference videos in the second video set as the retrieval results. While improving the retrieval and filtering efficiency through the first-level retrieval system, the second-level retrieval system can also improve the accuracy of the final retrieval results, thereby improving the accuracy of subsequent video recommendation or video authorization determination based on the retrieval results.

[0066] As an application example, for a video recommendation application, the terminal calls a video retrieval plugin to implement a video retrieval method. This involves multi-task extraction processing of the video to be retrieved to obtain its global features, content category, and channel category. Based on the global features and content category, a first-level retrieval system retrieves multiple reference videos from a first video set to form a second video set. A second-level retrieval system then determines the modal features of the video based on its channel category. Based on these modal features, target videos similar to the video to be retrieved are retrieved from the multiple reference videos in the second video set. These target videos are used as the retrieval results for the video to be retrieved. Based on these retrieval results, recommendation information is output for the video to be retrieved, thereby improving the efficiency and accuracy of video recommendations.

[0067] In some embodiments, after the terminal obtains a search request for a video to be searched, it calls the video search interface of the server 100 (which can be provided as a cloud service, i.e., a video search service). The server 100 responds to the search request by implementing a video search method, which involves performing multi-task extraction processing on the video to be searched to obtain its global features, content category, and channel category. Based on the global features and content category of the video to be searched, a first-level search system retrieves multiple reference videos from a first video set to form a second video set. A second-level search system determines the modal features of the video to be searched based on its channel category. Based on the modal features of the video to be searched, a target video similar to the video to be searched is retrieved from the multiple reference videos in the second video set, and the target video is used as the search result for the video to be searched. While improving the search and filtering efficiency through the first-level search system, the second-level search system can also improve the accuracy of the final search results, thereby improving the accuracy of subsequent video recommendation or video authorization determination based on the search results.

[0068] As an application example, for a video authorization determination application, after receiving a search request for the video to be searched, the server's video search interface is called. The server responds to the search request by implementing a video search method, which involves multi-task extraction processing of the video to be searched to obtain its global features, content category, and channel category. Based on the global features and content category of the video to be searched, a first-level search system retrieves multiple reference videos from a first video set to form a second video set. A second-level search system determines the modal features of the video to be searched based on its channel category. Based on the modal features of the video to be searched, target videos similar to the video to be searched are retrieved from the multiple reference videos in the second video set to determine if the video to be searched poses a copyright threat to the target video. An authorization prompt message indicating that authorization cannot be granted for the video to be searched is then output, thereby improving the efficiency and accuracy of video authorization determination.

[0069] In some embodiments, the terminal or server can implement the video retrieval method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run; it can also be a mini-program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0070] In some embodiments, multiple servers can form a blockchain, and server 100 is a node on the blockchain. Information connections can exist between each node in the blockchain, and information can be transmitted between nodes through these connections. Data related to the video retrieval method provided in this application embodiment (e.g., the logic of video retrieval and the results of video retrieval) can be stored on the blockchain.

[0071] The structure of the electronic device provided in the embodiments of this application is described below. See also... Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application. Taking the electronic device 500 as a server as an example, the following description is provided. Figure 2 The illustrated electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 540.

[0072] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0073] Memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory. Memory 550 may optionally include one or more storage devices physically located away from processor 510.

[0074] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0075] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0076] The network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0077] In some embodiments, the video retrieval device provided in this application can be implemented in software. Figure 2 A video retrieval device 555 stored in memory 550 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a task extraction module 5551, a first retrieval module 5552, and a second retrieval module 5553. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0078] As mentioned above, the video retrieval method provided in this application can be implemented by various types of electronic devices. See also Figure 3 , Figure 3 This is a flowchart illustrating the video retrieval method provided in the embodiments of this application, combined with... Figure 3 The steps shown are explained.

[0079] In step 101, the server performs multi-task extraction processing on the video to be retrieved to obtain the global features, content category, and channel category of the video to be retrieved.

[0080] As an example of obtaining the video to be searched, when a user opens the video client running on the terminal and enters the video to be searched, the terminal automatically obtains the search request for the video to be searched (carrying the video to be searched) and sends the search request for the video to be searched to the server. After receiving the search request for the video to be searched, the server parses the search request for the video to be searched and obtains the video to be searched for the video search.

[0081] Since videos can be quite long, when performing multi-task extraction processing on the video to be retrieved, it is necessary to perform frame extraction processing on the video to be retrieved, such as uniformly extracting one frame per second from the video to be retrieved. Therefore, multi-task extraction processing on the video to be retrieved is essentially performing multi-task extraction processing on each of the multiple image frames extracted from the video to obtain the global features, content category, and channel category corresponding to each image frame. Based on the global features, content category, and channel category corresponding to each image frame in the multiple image frames, the global features, content category, and channel category of the image to be retrieved are obtained.

[0082] In some embodiments, see Figure 4 , Figure 4 This is a flowchart illustrating the video retrieval method provided in an embodiment of this application. Figure 3 The multi-task extraction processing of the retrieved video is achieved through a multi-task model, which includes a basic feature layer, a feature embedding layer, a content classification layer, and a channel classification layer. Figure 3 The process involves multi-task extraction of the video to be retrieved, yielding its global features, content category, and channel category. This can be achieved through... Figure 4 Steps 1011 to 1014 are implemented as follows: In step 1011, basic features of the video to be retrieved are extracted through the basic feature layer to obtain the basic features of the video to be retrieved; in step 1012, the basic features of the video to be retrieved are processed by embedding vector transformation through the feature embedding layer to obtain the global features of the video to be retrieved; in step 1013, the basic features of the video to be retrieved are processed by content classification through the content classification layer to obtain the content category of the video to be retrieved; in step 1014, the basic features of the video to be retrieved are processed by channel classification through the channel classification layer to obtain the channel category of the video to be retrieved.

[0083] In some embodiments, the base feature layer includes a pooling layer and multiple cascaded convolutional layers, and the video to be retrieved includes multiple image frames. Figure 4 In step 1011, the basic features of the video to be retrieved are extracted through the basic feature layer. This can be achieved as follows: multiple cascaded convolutional layers are used to perform cascaded convolution processing on each image frame to obtain the convolutional features of each image frame; pooling layers are used to perform pooling processing on the convolutional features of each image frame to obtain the image features of each image frame; and the image features of each image frame are concatenated to obtain the basic features of the video to be retrieved.

[0084] like Figure 5 As shown, Figure 5This is a schematic diagram of the basic feature layer provided in this application embodiment. Each image frame among multiple image frames including the features to be retrieved is input to the first convolutional layer in the basic feature layer. The first convolutional layer performs convolution processing on the image frame to obtain the first feature map of the image frame. The first feature map of the image frame is output to the second convolutional layer. The second convolutional layer performs convolution processing on the first feature map of the image frame to obtain the second feature map of the image frame. The above convolution processing is performed sequentially until the Mth convolutional layer performs convolution processing on the (M-1)th feature map of the image frame to obtain the convolutional features of the image frame, where M is a positive integer greater than 1. The convolutional features of the image frame are output to the pooling layer in the basic feature layer. The pooling layer performs average pooling or max pooling processing on the convolutional features of the image frame to obtain the image features of the image frame. The image features of all image frames of the video to be retrieved are concatenated to obtain the basic features of the video to be retrieved.

[0085] The basic feature layer can be a convolutional neural network (such as ResNet101, ResNet50, or Inceptionv4) or a deep neural network. Basic features, also known as deep features, are low-order features extracted through preliminary feature extraction, representing the overall features of an image frame or the video to be retrieved (such as the location, attribute information, and pixel value of each pixel). By performing the basic feature extraction process described above, the basic features of the video to be retrieved can be initially extracted, saving computational resources in subsequent processing.

[0086] After obtaining the basic features of the video to be retrieved, the feature embedding layer performs embedding vector transformation on the basic features of the video to obtain the global features (i.e., embedded features) of the video to be retrieved; the content classification layer performs content classification on the basic features of the video to obtain the content category of the video to be retrieved; the channel classification layer performs channel classification on the basic features of the video to obtain the channel category of the video to be retrieved, so as to perform subsequent video retrieval based on the global features, content category and channel category of the video to be retrieved.

[0087] In practical applications, since the video to be retrieved consists of multiple image frames, after obtaining the image features of each image frame, these features can be directly input into the feature embedding layer, content classification layer, and channel classification layer. The feature embedding layer performs embedding vector transformation on the image features of each image frame to obtain the global features (embedded features) of each image frame. The content classification layer performs content classification on the image features of each image frame to obtain the content category corresponding to each image frame. The channel classification layer performs channel classification on the image features of each image frame to obtain the channel category of each image frame. Since one video corresponds to one channel category (such as variety shows, singing, sports, games, live-streaming e-commerce, lifestyle, etc.), after performing channel classification on multiple image frames of the video to be retrieved, the channel category corresponding to each image frame is obtained. The channel category that appears most frequently is taken as the channel category of the video to be retrieved. For example, for a 5-second video to be retrieved, the channel categories corresponding to the 5 images extracted from it are [lifestyle, lifestyle, lifestyle, concert, concert], so the video channel of the video to be retrieved can be determined as lifestyle.

[0088] Using the above method, the video to be retrieved, comprising N (an integer greater than or equal to 1) image frames, undergoes multi-task extraction processing using a multi-task model. This yields global features of the N image frames, content categories of the N image frames, and one channel category. The global feature sequence composed of the global features of the N image frames is then used as the global features of the video to be retrieved; the content category sequence composed of the content categories of the N image frames is used as the content category of the video to be retrieved; and the channel category is used as the channel category of the video to be retrieved. For example, if the video to be retrieved is 10 seconds long and 10 images are extracted from it, the multi-task extraction process yields global features of the 10 image frames, content categories of the 10 image frames, and one channel category. Therefore, the global features of the video to be retrieved include the global features of the 10 image frames, the content categories of the video to be retrieved include the content categories of the 10 image frames, and the channel category of the video to be retrieved is the obtained channel category.

[0089] In some embodiments, see Figure 6 , Figure 6This is a schematic diagram of the training method for the multi-task model provided in this application embodiment. The server can train the multi-task model in the following manner: In step 201, basic features of the video samples are extracted through the basic feature layer to obtain the basic features of the video samples; in step 202, the basic features of the video samples are processed by embedding vector transformation through the feature embedding layer to obtain the global features of the video samples; in step 203, the basic features of the video samples are processed by content classification layer to obtain the content category of the video samples; in step 204, the basic features of the video samples are processed by channel classification layer to obtain the channel category of the video samples; in step 205, the target loss function of the multi-task model is constructed based on the global features of the video samples, the content category of the video samples, and the channel category of the video samples, and the model parameters of the multi-task model are updated based on the target loss function.

[0090] Here, after constructing the target loss function of the multi-task model based on the global features of the video samples, the content category of the video samples, and the channel category of the video samples, it can be determined whether the value of the target loss function exceeds the preset threshold. When the value of the target loss function exceeds the preset threshold, the error signal of the video processing model is determined based on the target loss function, the error information is backpropagated in the video processing model, and the model parameters of each layer are updated during the propagation process. The updated model parameters are used as the model parameters of the trained multi-task model.

[0091] The multi-task model is a neural network model. Here, we explain backpropagation: Training sample data is input into the input layer of the neural network model, passes through the hidden layer, and finally reaches the output layer to output the result. This is the forward propagation process of the neural network model. Since there is an error between the output result of the neural network model and the actual result, the error between the output result and the actual value is calculated and backpropagated from the output layer to the hidden layer until it reaches the input layer. During the backpropagation process, the values ​​of the model parameters are adjusted according to the error. That is, a loss function is constructed based on the error between the output result and the actual value, and the partial derivatives of the loss function with respect to the model parameters are calculated layer by layer to generate the gradient of the loss function with respect to the model parameters of each layer. Since the direction of the gradient indicates the direction of error amplification, the gradient of the model parameters is inverted and summed with the original parameters of each layer of the model. The summation result is used as the updated model parameters of each layer, thereby reducing the error caused by the model parameters. The above process is iterated until convergence.

[0092] In some embodiments, see Figure 7 , Figure 7 This is a schematic diagram illustrating the training method of the multi-task model provided in an embodiment of this application. Figure 6In step 205, the objective loss function of the multi-task model is constructed based on the global features of the video samples, the content category of the video samples, and the channel category of the video samples. This can also be achieved through... Figure 7 Steps 2051-2054 are implemented as follows: In step 2051, an embedding loss function for the multi-task model is constructed based on the global features of the video samples, the global features of the positive video samples, and the global features of the negative video samples; In step 2052, a content classification loss function for the multi-task model is constructed based on the content category and content category label of the video samples; In step 2053, a channel classification loss function for the multi-task model is constructed based on the channel category and channel category label of the video samples; In step 2054, the embedding loss function, content classification loss function, and channel classification loss function are weighted and summed to obtain the target loss function of the multi-task model.

[0093] See Figure 8 , Figure 8 This is a schematic diagram illustrating the construction of the loss function provided in this application embodiment. An embedding loss function is constructed based on the global features of the video samples, a content classification loss function for the multi-task model is constructed based on the content category of the video samples, and a channel classification loss function for the multi-task model is constructed based on the channel category of the video samples. After constructing the three loss functions, the total loss of the multi-task model is obtained based on the three loss functions.

[0094] In some embodiments, the process of constructing the embedding loss function of the multi-task model is as follows: the global features of the video sample are matched with the global features of the positive video samples in the video sample to obtain a first similarity between the video sample and the positive video sample; the global features of the video sample are matched with the global features of the negative video samples in the video sample to obtain a second similarity between the video sample and the negative video sample; based on the first similarity and the second similarity, the embedding loss function of the multi-task model is constructed.

[0095] As an example, the formula for calculating the embedding loss function is: ,in, Indicates the boundary (margin). This represents the L2 distance (i.e., the first similarity) between the global features of sample a and the global features of sample p. This represents the L2 distance (i.e., the second similarity) between the global features of sample a and the global features of sample n. The embedding loss function increases the similarity (or decreases the distance) between originally similar samples, while decreasing the similarity (or increasing the distance) between originally dissimilar samples.

[0096] In some embodiments, when constructing a content classification loss function At that time, the cross-entropy loss function between the content category of the video sample and the content category label of the video sample is calculated. As an example, the formula for calculating the cross-entropy loss function is: Wherein, the above formula represents the average cross-entropy loss of N samples. This represents the predicted probability that sample i belongs to category c. This indicates whether the label of sample i is c. If it is c, then... =1, otherwise =0. In constructing the channel classification loss function. In the same way, the calculation method of the cross-entropy loss function described above can be used, which will not be repeated here.

[0097] After constructing the loss functions for each layer, the target loss function for the multi-task model can be constructed, which is... ,in, , and The corresponding weights can be set according to the actual situation, such as... , and Therefore, when training a multi-task model, by comprehensively considering various features of the video samples and jointly training it using embedding loss functions, content classification loss functions, and channel classification loss functions, the training effect of the multi-task model can be improved, thereby enhancing the subsequent video retrieval performance. For example, in video retrieval applications, combining embedding loss functions, content classification loss functions, and channel classification loss functions to improve the accurate representation of videos can enhance the accuracy of similarity comparisons between videos.

[0098] In some embodiments, before constructing the embedding loss function of the multi-task model based on the global features of video samples, the global features of positive video samples in the video samples, and the global features of negative video samples in the video samples, the server may also construct training samples in the following manner: extract at least one candidate video sample from multiple candidate positive video sample pairs, wherein the multiple candidate positive video sample pairs do not include video samples; obtain the similarity between each candidate video sample and the video sample, and sort the corresponding candidate video samples in descending order of similarity; and take the candidate video samples ranked lower in the sorting result as negative video samples in the video samples.

[0099] For example, for a video sample x in a certain video sample pair, randomly select a candidate video sample i from each candidate positive video sample pair, calculate the distance between candidate video sample i and video sample x, sort them in ascending order of distance, take the top 10 candidate video samples as negative samples (i.e., negative video samples in video sample x), and form triplet pairs with the positive samples of video sample x (i.e., the other video sample in the positive video sample pair).

[0100] In step 102, based on the global features and content categories of the video to be retrieved, multiple reference videos are retrieved from the first video set to form a second video set.

[0101] In some embodiments, the first video set includes multiple videos, each video including multiple reference image frames; the video to be retrieved includes multiple image frames, the global features of the video to be retrieved include the global features corresponding to each image frame, and the content category of the video to be retrieved includes the content category of each image frame; the server can retrieve multiple reference videos from the first video set based on the global features and content category of the video to be retrieved by performing the following processing for each image frame: searching for the global features of a target number of reference image frames from the global feature index table of the video index corresponding to the first video set; obtaining the Euclidean distance between the global features of each reference image frame and the global features of the image frame in the target number of reference image frames; determining the distance threshold corresponding to the content category of the image frame, and selecting multiple reference image frames from the target number of reference image frames whose Euclidean distance is less than the distance threshold; and retrieving multiple reference videos from the multiple videos in the first video set based on the selected multiple reference image frames.

[0102] Here, for the video to be retrieved, after obtaining the global features of each image frame in the multiple image frames of the video to be retrieved, the content category of each image frame in the multiple image frames of the video to be retrieved, and the channel category of the video to be retrieved, the total duration of the video to be retrieved is recorded (since 1 frame is extracted per second, the total duration is equal to the number of global features of the image frames of the video to be retrieved). Based on the channel category of the video to be retrieved, different parameter configurations are selected, such as different parameters for lifestyle, film and television, and concert categories. According to the parameter configuration, for each image frame in the video to be retrieved, the top K (preset, e.g., K = 1000) nearest reference image frames are found from the first video set. Then, based on the content category of each image frame in the video to be retrieved, the distance threshold thr corresponding to that content category is obtained (e.g., if the content category of the i-th image frame is text in a lifestyle video, the distance threshold can be set to 0.1). The top K reference image frames are selected from the first video set (reference image frames whose global features in the first video set have an Euclidean distance less than the distance threshold between them and the global features of the image frames in the video to be retrieved). Based on these selected reference image frames, multiple reference videos are retrieved from multiple videos in the first video set. Through this method, customized filtering and retrieval based on different video channels and content categories is achieved.

[0103] In some embodiments, the server may retrieve multiple reference videos from multiple videos in a first video set based on selected reference image frames in the following manner: searching for the video identifier of the video to which each reference image frame belongs in the identifier index table corresponding to the video index of the first video set; selecting the video corresponding to each video identifier as a candidate video from multiple videos (inventory videos) in the first video set; obtaining the selection duration corresponding to the selected reference image frame in each candidate video, and the total duration of each candidate video; obtaining a first ratio of each selection duration to the total duration, and retrieving candidate videos from multiple candidate videos in the first video set whose first ratio is greater than a ratio threshold as reference videos.

[0104] Here, for each candidate video selected from the first video set, the following processing is performed: Count the number of reference image frames selected in each candidate video, denoted as Ne_db[j]. Taking the extraction of one image frame per second from the video as an example, the number of reference image frames selected in each candidate video is the selection duration corresponding to the selected reference image frames, denoted as Dur_recall_db[j]. Based on this, the first ratio between the selection duration corresponding to the reference image frames in each candidate video and the total duration of the candidate video (the total number of reference image frames corresponding to the candidate video, denoted as db_vid_time_list[j]) can be calculated. The first ratio is: R_db_number = Ne_db[j] / db_vid_time_list[j]. When R_db_number is greater than the ratio threshold (corresponding to the content category of the image frame), the candidate video is retained; otherwise, it is discarded.

[0105] In some embodiments, the server may obtain the matching duration of the image frames that match the multiple image frames of the video to be retrieved, and the duration of the video to be retrieved; obtain a second ratio of each matching duration to the duration of the video to be retrieved; and sort each reference video in descending order based on the second ratio to obtain a reference video sequence to form a second video set.

[0106] Here, for each retained candidate video (i.e., reference video), the number (or duration) of matching image frames between the video to be retrieved and each reference video is counted, Ne_query, which is the duration of matching between the video to be retrieved and each reference video, Dur_recall_query. Based on the second ratio obtained by dividing the duration of matching between the video to be retrieved and each reference video by the total duration of the video to be retrieved (R_query_number = Ne_query / query_time), the similarity between each reference video and the video to be retrieved can be determined. Generally, the larger the second ratio, the more similar the corresponding reference video is to the video to be retrieved; the smaller the second ratio, the less similar the corresponding reference video is to the video to be retrieved. Therefore, the corresponding reference videos are sorted in descending order of the second ratio to obtain the reference video sequence as the second video set.

[0107] In some embodiments, the server may construct a video index corresponding to the first video set by: initializing a global feature index table, a time index table, and an identifier index table for the video index corresponding to the first video set; performing the following processing for each reference image frame: obtaining the global features of the reference image frame, the time point of the reference image frame in the corresponding video, and the video identifier of the video to which the reference image frame belongs; adding the global features of the reference image frame to the global feature index table, adding the time point of the reference image frame in the corresponding video to the time index table, and adding the video identifier of the video to which the reference image frame belongs to the identifier index table, thereby constructing a video index corresponding to the first video set.

[0108] Here, for each stock video in the first video set, a multi-task extraction process is performed on each stock video in the retrieval library (i.e., the first video set) using a multi-task model to obtain the global features of multiple image frames of each stock video, the content category of multiple image frames of each stock video, and the channel category of each stock video. Then, initialize the global feature index table (db_embedding_list), time index table (db_time_list), and identifier index table (video_id_list) corresponding to the video index of the first video set to be empty. Finally, add the global features of each image frame of each stock video to db_embedding_list, add the time point of each image frame in the corresponding stock video (since one frame is extracted every second, the i-th frame represents the i-th second) to db_time_list, and add the video identifier of the video to which each image frame belongs to video_id_list. At this time, the i-th value in the three db_embedding_list, db_time_list, and video_id_list corresponds to the global feature of the same frame in the same video, the time point of the feature, and the video ID to which the feature belongs. Therefore, based on the ID of a certain global feature in db_embedding_list, you can query its time point (which second) in the video and its video ID (video_id_list[id]). Additionally, the duration (in seconds) of each video ID can be recorded as db_vid_time_list, such as db_vid_time_list[i] indicating that the duration of the i-th video is db_vid_time_list[i] seconds, where i represents the video number.

[0109] In step 103, based on the channel category of the video to be retrieved, the modal features of the video to be retrieved are determined, and based on the modal features of the video to be retrieved, the target video is retrieved from multiple reference videos in the second video set, so as to use the target video as the retrieval result of the video to be retrieved.

[0110] In some embodiments, the server may determine the modal features of the video to be retrieved based on the channel category of the video to be retrieved in the following manner: when the channel category indicates that the video to be retrieved is a text category, text feature extraction processing is performed on the text data of the video to be retrieved to obtain the text features of the video to be retrieved; when the channel category indicates that the video to be retrieved is an audio category, audio feature extraction processing is performed on the audio data of the video to be retrieved to obtain the audio features of the video to be retrieved; when the channel category indicates that the video to be retrieved is a multi-object category, local feature extraction processing is performed on the multi-objects in the video to be retrieved to obtain the local features of the video to be retrieved; at least one of the text features, audio features, and local features is determined as the modal features of the video to be retrieved.

[0111] In practical applications, depending on the video channel to which the video to be retrieved belongs, appropriate modal features are selected for secondary retrieval processing (fine matching). For example, for lecture or lecture-type videos, audio features are extracted from multiple audio frames to obtain the audio features of each frame, and fine matching is then performed based on these audio features. For videos with a relatively large number of targets (such as large scenes like squares or indoor global maps), local features are extracted from multiple image frames, such as SIFT feature extraction, to obtain the SIFT features of each image frame as local features, and fine matching is then performed based on these local features. For fast-moving sports videos, since both image and audio are prone to failure, text features are extracted from the text data (such as titles, bullet comments, and reviews), and fine matching is then performed based on these extracted text features. Furthermore, in practical applications, fine matching can also be performed based on multiple modal features (i.e., a combination of two or more features from text features, audio features, and local features), such as using local features combined with audio features for filtering.

[0112] In some embodiments, the server may retrieve a target video from multiple reference videos in a second video set based on the modal features of the video to be retrieved in the following manner: performing similarity matching between the modal features of the video to be retrieved and the modal features of each reference video in the second video set to obtain the corresponding similarity value; and retrieving reference videos from the multiple reference videos in the second video set whose similarity value is greater than a similarity threshold as the target video.

[0113] Here, multiple reference videos selected from the first video set are finely matched with the video to be retrieved. For example, when the modal features are represented by local features, the local features of each reference video and the video to be retrieved are finely matched, that is, the similarity between the local features of the retrieved video and the local features of the reference videos is calculated, such as calculating the Euclidean distance between them, and selecting the reference videos whose Euclidean distance is lower than the distance threshold as the target videos. As another example, when the modal features are represented by text features, the text features of each reference video and the video to be retrieved are finely matched, that is, the similarity between the text features of the video to be retrieved and the text features of each reference video is calculated, such as calculating the Euclidean distance between them, and selecting the reference videos whose Euclidean distance is lower than the distance threshold as the target videos.

[0114] In some embodiments, after the server retrieves multiple reference videos from the first video set, when the channel category indicates that the video to be retrieved belongs to a multi-target category, local feature extraction is performed on the video to be retrieved to obtain the local features of the video to be retrieved; based on the local features, multiple candidate target videos are retrieved from the multiple reference videos in the second video set to form a third video set; correspondingly, the target video is retrieved from the multiple reference videos in the second video set in the following manner: the target video is retrieved from the multiple candidate target videos in the third video set.

[0115] In practical applications, considering that the global features of the video to be retrieved are often insufficient, this application embodiment also provides another video retrieval system, such as... Figure 12 As shown, the architecture includes a multi-task model, a first-level retrieval system (or a first-level recall system), a second-level retrieval system (or a second-level local feature fine matching module), and a third-level retrieval system (a third-level multimodal fine matching module). In actual implementation, after the server retrieves multiple reference videos from the first video set, when the channel category represents the video to be retrieved as a multi-target category, local features are extracted from the video to be retrieved to obtain the local features of the video to be retrieved. Based on the local features, multiple candidate target videos are retrieved from the multiple reference videos in the second video set to form a third video set. Based on the channel category of the video to be retrieved, the multimodal features of the video to be retrieved are determined, and based on the multimodal features of the video to be retrieved, the target video is retrieved from the multiple candidate target videos in the third video set.

[0116] In some embodiments, after obtaining the search results, the server may combine the search results and channel category to output feedback information for the video to be searched; wherein, the feedback information is used to provide recommendation information or authorization prompt information for the video to be searched.

[0117] In practical applications, after obtaining the search results, the server can combine the search results and channel categories to output different feedback information according to business needs. For example, in video recommendation applications, it can output recommendation information for the video to be searched; in video authorization determination, it can output authorization prompt information for the video to be searched. For lifestyle videos, it can display videos with high click-through rates and their titles for the user's reference; for course / lecture videos, it can return videos that influence the authorization of the video to be searched and indicate the degree of influence; for game videos, it can push videos already in the search library to allow users to learn how to play related maps / heroes.

[0118] It is understood that, in the embodiments of this application, the data related to the global features, content categories or channel categories of the video to be retrieved is essentially user data. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0119] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario. The video retrieval method provided by the embodiments of this application can be applied to application scenarios such as video recommendation retrieval and video authorization retrieval.

[0120] In related technologies, a common method for video retrieval is to use the global image features (vector (embedding) description) of multiple image frames of a video for similarity judgment. For example, the video to be retrieved is processed by extracting frames to obtain multiple image frames, acquiring the image embedding of each image frame, and comparing the Euclidean distance between the image embeddings of two image frames in two videos. If the Euclidean distance is less than a distance threshold, it indicates similarity. The number of matching frames in the two videos is counted to obtain the matching duration. If the matching duration exceeds a certain threshold, it indicates that the two videos are duplicates. However, this retrieval method has the following problems: 1) Insufficient features: Direct retrieval based on image embedding is prone to false positives or false negatives. This is because image embedding has different representation capabilities on different images. For example, it performs poorly on images of text, fine-grained faces, and dense crowds, leading to retrieval errors in these categories. 2) Insufficient single-process handling: It cannot meet the different needs of different channel categories and does not support customized channel business requirements. This is because... Videos can be divided into multiple channel categories, such as variety shows, singing, sports, games, live-streaming e-commerce, and lifestyle videos. In business applications, the deduplication (or retrieval) requirements for videos on different channels are different. For example, lifestyle videos require full-screen deduplication, while live-streaming game videos only require deduplication of the game portion of the screen (not the host portion, which often appears in a small box in the upper left or lower right corner of the screen) since the same host can talk about multiple games. Live-streaming e-commerce videos require deduplication of details such as the goods and people in the screen. In addition, the deduplication requirements for specific screens in a video within a channel are also different. For example, text screens in game videos do not require deduplication, only screens such as battles and maps in the game need to be deduplicated. However, different videos under the same map segment may not be very different, and relying solely on screen features is inaccurate. Audio capabilities are also needed. Singing videos often have lyrics displayed in the text, so deduplication of such videos requires text deduplication capabilities. 3) Insufficient customization support: It does not support adding specific types of recall strategies or customized post-processing for different types of videos.

[0121] It is evident that the conventional architecture for video deduplication based on pure image embedding cannot meet the various needs of the above-mentioned businesses, and it cannot comprehensively address the deduplication requirements of multiple video channels in the application. To address this, this application provides a video retrieval method. It proposes adding the ability to classify image content and video channels within videos to existing image embedding feature extraction, establishing a customized video filtering system supporting different content categories as a primary retrieval system. Simultaneously, it generates multiple deduplication and fine-matching capabilities based on the modal feature representation capabilities of different channel categories, supporting the combined use of different fine-matching capabilities, as a secondary retrieval system. In application, after customized filtering in the primary retrieval system, the secondary retrieval system, with different fine-matching capabilities, is invoked according to the different channel categories of the video to be retrieved to retrieve the final target video as the retrieval result. Since the primary retrieval system considers the global features and content categories of the video to be retrieved, it achieves coarse retrieval or coarse filtering of videos, improving not only retrieval and filtering efficiency but also the applicability of retrieval for videos with different content categories. When retrieving the target video in the secondary retrieval system, considering the different modal features corresponding to different channel categories, it uses modal features corresponding to the video to be retrieved to achieve fine retrieval, improving the accuracy of the final retrieval results.

[0122] The implementation principle of the video retrieval method provided in the embodiments of this application will be explained next. See [link to relevant documentation]. Figure 9 , Figure 9 This is a schematic diagram of the first video set provided in the embodiments of this application. First, a retrieval library (i.e., the first video set) of the primary retrieval system is established. Then, the inventory videos in the first video set are processed by a multi-task model to obtain information such as global features, content categories and channel categories of the inventory videos. Finally, the information is saved to the retrieval library.

[0123] See Figure 10 , Figure 10 This is a schematic diagram of the architecture of a video retrieval system provided in an embodiment of this application. The system includes: a multi-task model, a first-level retrieval system (or a first-level recall system), and a second-level retrieval system (or a second-level fine-matching module). When retrieving a video, the multi-task model performs multi-task extraction processing on the video to be retrieved, obtaining the global features, content category, and channel category of the video. Then, the first-level retrieval system filters or recalls a target number of inventory videos from the inventory of a first video set as reference videos based on a customized threshold for the content category, forming a second video set. Then, based on the channel category of the video to be retrieved, it determines which modal features to use for fine-matching in the second-level fine-matching module, so that the target video is retrieved from the target number of reference videos in the second video set based on the determined modal features, serving as the retrieval result for the video to be retrieved.

[0124] The following sections will explain the retrieval processes of the multi-task model, the primary retrieval system, and the secondary retrieval system.

[0125] 1) Multi-task model

[0126] 1.1) Multi-task model structure

[0127] The multi-task model includes: a basic feature layer, a feature embedding layer, a content classification layer, and a channel classification layer. The basic feature layer uses ResNet101 pre-trained on ImageNet. The network structure of ResNet101 is shown in Table 1.

[0128] Table 1 Structure of ResNet101

[0129]

[0130] The feature embedding layer, also known as the image similarity measurement module, uses an embedding extraction network. The network structure of the embedding extraction network is shown in Table 2.

[0131] Table 2 Structure of Feature Embedding Layer

[0132]

[0133] The content classification layer uses the classification layer Fc1 to learn the video content classification task and outputs the video content categories. Nclass1 is the number of video content categories. The results of the content classification layer are shown in Table 3.

[0134] Table 3. Structure of Content Classification Layers

[0135]

[0136] The channel classification layer uses the Fc2 classification layer to learn the channel classification task of the video and outputs the channel categories of the video. Nclass2 is the number of channel categories of the video. The results of the channel classification layer are shown in Table 4:

[0137] Table 4. Structure of Channel Classification Layer

[0138]

[0139] It should be noted that, although the above lists the methods used respectively Figures 2-4 The structure is used to output three tasks. In practical applications, other structures can also be used, such as stacking multiple ReLU-activated fully connected layers before the final output.

[0140] 1.2) Regarding multi-task model training

[0141] 1.2.1) Parameter Initialization

[0142] The Conv1-Conv5 layers in the basic feature layer use the parameters of ResNet101 pre-trained on the ImageNet dataset. Newly added layers such as the embedding layer, Fc1 layer and Fc2 layer are initialized with a Gaussian distribution with a variance of 0.01 and a mean of 0.

[0143] 1.2.2) Setting learning parameters

[0144] Update the underlying basic features and set the learning parameters as shown in Tables 1, 2, 3, and 4.

[0145] 1.2.3) Learning Rate

[0146] To improve learning performance, this application employs an asynchronous learning rate. The basic feature layer and feature embedding layer both use a learning rate of lr1=0.0005, as do the two classification layers (content classification layer and channel classification layer). Since classification layers are more prone to overfitting to the target (because the classification task involves getting two samples of the same class to output the same predicted class, i.e., fitting the same learning target, overfitting the label predictions will result in very similar predictions for multiple dissimilar images), this semantic overfitting will also make the embedding more prone to overfitting. That is, two images of the same class will have identical embeddings (i.e., the embeddings are also overfitting the classification target), resulting in a lack of distinction between image embeddings. The asynchronous learning rate allows the embedding layer's parameter updates to be slower than the fully connected (FC) layer (update efficiency is 0.1 times that of the FC layer), effectively preventing the embedding from being overfitted by the two classification targets during each batch parameter update.

[0147] 1.2.4) Learning Process

[0148] For the full dataset, perform training epochs; each epoch processes the full dataset once, until the average loss no longer decreases in a certain epoch.

[0149] 1.2.5) The specific operations in each iteration are as follows: Divide the full sample into batches of bs samples (referring to triplet samples) into Nb batches. For each batch, obtain the triplet samples mentioned above and perform the following processing:

[0150] (1) Model forward: Set all parameters of the multi-task model to the state to be learned. During training, the multi-task model performs forward calculation on an input image frame to obtain the prediction results: em, fc1 and fc2. Among them, em represents the global features (embedded features) of the image frame, fc1 represents the content category of the image frame, and fc2 represents the channel category of the image frame.

[0151] (2) Loss calculation: Calculate the corresponding loss, calculate the triplet loss function for em, calculate the cross entropy loss function for classification for fc1 and fc2 respectively, and obtain the target loss function by weighted summation of the three loss functions. See the following description for the specific calculation process.

[0152] (3) Model parameter update: Stochastic gradient descent (SGD) is used to perform gradient backward calculation on the loss from the previous step to obtain the updated values ​​of all model parameters and update the model.

[0153] 1.3) Preparation for training multi-task models

[0154] 1.3.1) Sample pairs for triplet learning

[0155] The training data for image metric learning is labeled according to the following rules: Each labeling task involves selecting three images from the entire dataset that meet the rules to form a set of samples (anchor, positive, and negative). Anchor and positive form a positive sample pair (similar or identical images), and anchor and negative form a negative sample pair (dissimilar images), abbreviated as (a, p, n). The randomly generated triplet data contains a large number of easily distinguishable samples. These samples initially help the model learn, but quickly (e.g., after the third epoch), the model develops a good ability to distinguish easy samples. At this point, the loss from a large number of easy samples will be much greater than that from hard samples, thus burying the hard samples among the easy samples. Therefore, obtaining a sufficient number of hard samples in the mid-to-late stages has a significant impact on the model.

[0156] Based on this, the embodiments of this application use triplet learning to mine samples. The labeling is adjusted to first label positive sample pairs (obtain similar sample pairs), and use positive sample pairs as input. In each batch of sample pairs (the number of sample pairs in each batch is bs), the following mining is performed to obtain triplet pairs: For a certain sample x: randomly select an image sample i from the remaining bs-1 sample pairs, and calculate the distance between the embedding of image sample i (the output of the current model's embedding layer) and x. Sort the samples by distance from smallest to largest, remove the first K% of samples (e.g., 3%, i.e., (2*bs-2)*3 / 100, e.g., for bs=64, the first K% is the first 4), and take the first 10 samples as negative samples. These samples are then combined with the positive sample pairs in x to form triplet pairs. Therefore, each sample x generates 10 triplet pairs, and the entire batch obtains 10*bs triplet pairs.

[0157] In practical applications, bs needs to be set to a relatively large value, such as 256. Taking the first 10 samples as negative samples is based on the following considerations: Since this task requires identifying videos composed of multiple images affecting authorization, these videos are generally extremely similar. For example, two images that have undergone certain image transformations (noise addition, cropping, border addition, etc.) will have almost identical content. However, in reality, randomly selecting two images from massive datasets often results in different images. Therefore, the first 10 samples with the smallest distance can all satisfy the condition of being negative samples of positive sample pairs.

[0158] 1.3.2) Tagging based on content categories

[0159] For each image in the above-annotated positive sample pair, label the image's content category (since the content of two images in the positive sample is extremely similar, only one image can be labeled, and the other image in the sample pair will also have the same content category). For example, the image content categories include: people, text, game heroes, game maps, and others, totaling 5 content categories. If a retrieval strategy needs to be defined for other content categories later, more content categories can be added.

[0160] 1.3.3) Labeling for Channel Categories

[0161] For each image in the above-annotated positive sample pair, label the image's channel category (since the content of the two images in the positive sample is extremely similar, you can also label only one image, and then the other image in the sample pair will also have the same category). For example, the channel categories include: lifestyle videos, film and television variety shows, concerts, games, and others, a total of 5 channel categories. If you need to define search strategies for other channel categories in the future, you can add more channel categories.

[0162] 1.4) Loss function for multi-task models

[0163] The total loss (target loss function) in this application embodiment is the sum of three losses, namely the embedding loss function. Content classification loss function Channel classification loss function The sum of them is ,in, , and The corresponding weights can be set according to the actual situation, such as... , and .

[0164] 1.4.1) Triplet loss

[0165] For each image triple (a, p, n), its representation is obtained from the model. , , (The output of the embedded feature layer), calculated according to formula (1) in the preset Triple loss at the preset margin between positive and negative sample pairs .

[0166] 1.4.2) Content Classification Loss Function

[0167] Constructing the content classification loss function At that time, the cross-entropy loss function between the content category of the video sample and the content category label of the video sample is calculated. As an example, the formula for calculating the cross-entropy loss function is: Wherein, the above formula represents the average cross-entropy loss of N samples. This represents the predicted probability that sample i belongs to category c. This indicates whether the label of sample i is c. If it is c, then... =1, otherwise =0.

[0168] 1.4.3) Channel Classification Loss Function

[0169] Constructing the channel classification loss function In the same way, the content classification loss function can be constructed, that is, the cross-entropy loss function can be used for calculation, which will not be elaborated here.

[0170] 1.5) The model performs forward inference on the video.

[0171] For the input video, one image frame is uniformly extracted per second, such as N image frames. A multi-task model is used to perform forward computation on these N image frames, obtaining their global features (embedded features), content categories, and channel categories. The global feature sequence composed of the global features of the N image frames can be used as the global features of the video, and the content category sequence composed of the content categories of the N image frames can be used as the content category of the video to be retrieved. Since each video corresponds to a channel category (such as variety shows, singing, sports, games, live-streaming e-commerce, lifestyle, etc.), after channel classification processing of the N image frames of the video, the channel category corresponding to each image frame is obtained. The channel category that appears most frequently can be used as the video's channel category. For example, in a 5-second video, the channel categories corresponding to the 5 extracted images are [lifestyle, lifestyle, lifestyle, concert, concert], so the video channel can be determined as "lifestyle".

[0172] In this way, the global features of the video are a sequence of global features composed of the global features of all image frames extracted from the video; the content category of the video is a sequence of content categories composed of the content categories of N image frames; and the channel category of the video is the channel category of a specific image frame among the N image frames. For example, if the video to be retrieved is 10 seconds long, and 10 images are extracted from it, a multi-task extraction process using a multi-task model can be used to obtain the global features of the 10 image frames, the content categories of the 10 image frames, and one channel category.

[0173] 2) The retrieval process of the primary retrieval system

[0174] 2.1) Construct a retrieval inventory index

[0175] First, a multi-task extraction process is performed on each stock video in the retrieval library (i.e., the first video set) using a multi-task model to obtain the global features of multiple image frames of each stock video, the content category of multiple image frames of each stock video, and the channel category of each stock video. Then, initialize the global feature index table (db_embedding_list), time index table (db_time_list), and identifier index table (video_id_list) corresponding to the video index of the first video set to be empty. Finally, add the global features of each image frame of each stock video to db_embedding_list, add the time point of each image frame in the corresponding stock video (since one frame is extracted every second, the i-th frame represents the i-th second) to db_time_list, and add the video identifier of the video to which each image frame belongs to video_id_list. At this time, the i-th value in the three db_embedding_list, db_time_list, and video_id_list corresponds to the global feature of the same frame in the same video, the time point of the feature, and the video ID to which the feature belongs. Therefore, based on the ID of a certain global feature in db_embedding_list, you can query its time point (which second) in the video and its video ID (video_id_list[id]). Additionally, the duration (in seconds) of each video ID can be recorded as db_vid_time_list, such as db_vid_time_list[i] indicating that the duration of the i-th video is db_vid_time_list[i] seconds, where i represents the video number.

[0176] 2.2) Search process

[0177] Step 1: Retrieve image frames from the search library

[0178] 2.2.1) For the video to be retrieved (query video), it first undergoes multi-task extraction processing through a multi-task model to obtain the global features Le of each image frame in the multiple image frames of the video to be retrieved, the content category Lc of each image frame in the multiple image frames of the video to be retrieved, and the channel category of the video to be retrieved. The total duration of the video to be retrieved is recorded as query_time (since 1 frame is extracted per second, the total duration is equal to the number of global features Ne_query of the image frames of the video to be retrieved).

[0179] 2.2.2) Select different parameter configurations based on the channel category of the video to be retrieved, such as... Figure 1Configure the Lifestyle, Film & TV, and Concert channels (parameter configuration needs to be determined in advance. Generally, a customized channel is added according to business needs. The categories and parameters under this channel can be customized. For example, if some channels need to perform high-accuracy face retrieval, a face category threshold of 0.1 can be added under this channel. This is the Euclidean distance threshold. The smaller the Euclidean distance, the more similar the face is). For other categories, directly select the Lifestyle channel parameter configuration.

[0180] 2.2.3) Based on the parameter configuration, for the global feature Lei of the i-th image frame in the video to be retrieved, the top K (preset, e.g., K = 1000) image frames in the database are searched for their global features. Then, the distance threshold thr corresponding to the content category of Lci is obtained (e.g., if the content category of the i-th image frame is text in a lifestyle video, the distance threshold can be set to 0.1). Global features of image frames from the top K global features that satisfy the condition that the Euclidean distance to the query feature Lei is less than the distance threshold are selected from the retrieval database (i.e., image frames corresponding to global features that satisfy the above conditions are selected). Alternatively, a range search can be used directly, such as directly searching the database for image frames corresponding to global features whose Euclidean distance to Lei is less than the distance threshold. This method achieves customized filtering and recall based on different video channels and content categories.

[0181] 2.2.4) After obtaining multiple image frames that meet the conditions in the search library, record the video identifier of the stock video to which the image frame belongs in the search library, and then find the video identifier video_id_list[id] of the video to which the recalled image frame belongs, and use the stock video corresponding to the video identifier in the index library as the recalled video (i.e. the above candidate video).

[0182] Step 2: Sort the video based on the recalled image frames.

[0183] 2.2.5) Based on the global features of the image frames that satisfy the above recall conditions and whose Euclidean distance to the query feature Lei is less than the distance threshold, after selecting the image frames corresponding to the global features that satisfy the above conditions, count how many image frames were recalled in the video j corresponding to the video identifier of the video to which each recalled image frame belongs, denoted as Ne_db[j]. Taking the extraction of one image frame per second from the video as an example, the number of image frames selected in the recalled video j is the selection duration corresponding to the selected image frames, denoted as Dur_recall_db[j]. At the same time, count the number (or duration) of image frames that match the video to be retrieved and the recalled video j, Ne_query, that is, count the duration of the video to be retrieved and the recalled video j that match, Dur_recall_query.

[0184] 2.2.6) Recall video processing

[0185] For each recalled video j, calculate the proportion of recalled frames in the recalled video (first ratio): R_db_number = Ne_db[j] / db_vid_time_list[j], and the proportion of recalled frames in the video to be retrieved (second ratio): R_query_number = Ne_query / query_time; when R_db_number is greater than the ratio threshold (thr_number), the recalled video is retained. The ratio threshold corresponds to the content category of the image frame and can be preset, such as 0.3. That is, when the matching degree between the video to be retrieved and a certain recalled video in the search library is 30%, the recalled video is considered a valid recalled video, and the valid recalled video is the reference video mentioned above; otherwise, it is an invalid recalled video and is discarded.

[0186] 2.2.7) Recall Video Sorting

[0187] For each valid recalled video, calculate R_query_number, sort the valid recalled videos in descending order to obtain a sequence of valid recalled videos, which forms the second video set.

[0188] 3) The retrieval process of the secondary retrieval system

[0189] The effective recalled videos output by the previous-level retrieval system undergo a second-level fine matching process for re-sorting, retrieving the final target videos as the search results, thus making the final displayed TOPN target videos more accurate. This second-level fine matching process provides three modal feature deduplication modules: local feature deduplication, text feature deduplication, and audio feature deduplication. In practical applications, the appropriate modal feature deduplication module is selected based on the video channel to which the video to be retrieved belongs. For example, for lecture or lecturing videos, the audio feature deduplication module can be used; for videos with a relatively large number of targets in the frame (such as large scenes like squares or indoor global maps), the local feature deduplication module is used; for fast-moving sports videos, since both image and audio are prone to failure, the text (title) feature deduplication module can be used. Furthermore, in practical applications, deduplication can also be performed based on a combination of multiple modal features, such as using local features + audio features for combined filtering.

[0190] 3.1) Local Feature Deduplication Module

[0191] First, local feature extraction is performed on multiple image frames of the video to be retrieved, such as SIFT feature extraction, to obtain the SIFT features of each image frame as local features. Then, for each image frame in the above-mentioned effective recalled video pair (a video pair consisting of the video to be retrieved and each effective recalled video), fine matching of local features is performed, that is, the similarity between the local features of the image frames of the retrieved video and the local features of the image frames of the effective recalled videos is calculated, such as calculating the Euclidean distance between the two, and selecting image frames whose Euclidean distance is lower than the distance threshold (i.e., similar to Step 1 in 2.2). Then, referring to Step 2 in 2.2 (2.2.5)-2.2.6), local feature matching is performed to retrieve the target video from the second video set as the retrieval result for the video to be retrieved.

[0192] 3.2) Text Feature Deduplication Module

[0193] First, text feature extraction is performed on each video in the search database. For example, each video is input into a Bidirectional Encoder Representation from Transformers (BERT) model to extract text features from the text information (such as title information) of each video, resulting in text features for each video. These text features are then added to the text feature database. Next, for the video to be searched, text features are extracted in the same way, and the similarity between the text features of the video to be searched and the text features of each video in the database is calculated. For example, the Euclidean distance between them is calculated. Videos in the database with text features whose Euclidean distance is below a certain threshold are selected as target videos, and these target videos are used as the search results for the video to be searched.

[0194] 3.3) Audio Feature Deduplication Module

[0195] First, an audio frame is extracted every second from the audio data of the video to be retrieved. Audio feature extraction is then performed on each audio frame to obtain its audio features. For example, the audio data (wav file) of the video to be retrieved is extracted, and the audio data is sampled based on a sampling window to obtain audio data corresponding to multiple sampling windows. Fourier transforms are then performed on the audio data corresponding to each sampling window to obtain the audio spectrum for each sampling window. For example, the audio data is resampled to 16kHz mono audio, and a short-time Fourier transform is performed using a 25ms Hann window and a 10ms frame shift to obtain the spectrogram. The spectrogram is then mapped to a 64th-order Mel spectrum filter bank to calculate the Mel spectrum (mel spectrum), and log(mel-spectrum + 0.01) is calculated to obtain a stable Mel spectrum. The added bias of 0.01 is to avoid taking the logarithm of 0. These features are then framed with a duration of 0.96s, with no frame overlap. Each frame contains 64 Mel frequency bands and a duration of 10ms (i.e., a total of 96...). (Frame), to obtain audio features (embedding).

[0196] Then, for each audio frame in the above-mentioned effective recalled video pairs (video pairs consisting of the video to be retrieved and each effective recalled video), a fine matching of audio features is performed, that is, the similarity between the audio features of the audio frames of the retrieved video and the audio features of the audio frames of the effective recalled video is calculated, such as calculating the Euclidean distance between the two, and selecting target videos with an Euclidean distance lower than the distance threshold from the second video set as the retrieval results for the video to be retrieved.

[0197] It should be noted that for frame-level features (such as local features of image frames and audio features of audio frames), due to the large number of features, inverted file search (ivf) can be used in the search database to improve search efficiency.

[0198] 4) Customized feedback

[0199] See Figure 11 , Figure 11 This is a schematic diagram of the architecture of the video retrieval system provided in this application embodiment. For retrieval results of different channel categories, different feedback information can be output according to business needs. For example, in the video recommendation application, recommendation information for the video to be retrieved is output; in the video authorization determination, authorization prompt information for the video to be retrieved is output, such as prompt information on whether authorization is granted for the video to be retrieved; for example, for lifestyle videos, videos with high click-through rates and their titles are displayed for the user's reference; for course lecture videos, videos that affect the authorization of the video to be retrieved are returned and the degree of influence is indicated (R_db_number); for game videos, videos already in the search library are pushed to the user to learn how to play the relevant maps / heroes.

[0200] In practical applications, considering that the global features of the video to be retrieved are often insufficient, this application embodiment also provides another video retrieval system, see [link to relevant documentation]. Figure 12 , Figure 12 This is a schematic diagram of the architecture of the video retrieval system provided in this application embodiment. The architecture includes a multi-task model, a first-level retrieval system (or a first-level recall system), a second-level retrieval system (or a second-level local feature fine matching module), and a third-level retrieval system (a third-level multimodal fine matching module). In actual implementation, after the server retrieves multiple reference videos from the first video set, when the channel category indicates that the video to be retrieved belongs to a multi-target category, local features are extracted from the video to be retrieved to obtain the local features of the video to be retrieved. Based on the local features, multiple candidate target videos are retrieved from the multiple reference videos in the second video set to form a third video set. Based on the channel category of the video to be retrieved, the multimodal features of the video to be retrieved are determined, and based on the multimodal features of the video to be retrieved, the target video is retrieved from the multiple candidate target videos in the third video set.

[0201] In summary, the embodiments of this application have the following beneficial effects:

[0202] 1) A multi-level retrieval process is adopted, which successively uses the global features and content features of the video to be retrieved to perform coarse matching and deduplication, as well as the multi-modal features to perform fine matching and deduplication. Through multi-level combination, the retrieval efficiency and accuracy are improved.

[0203] 2) Adopting a channel classification model based on global features, and customizing the secondary deduplication capability of video channels, it supports the combination output of fine matching deduplication capabilities under different channel categories, thereby meeting the accuracy requirements of different channel categories. At the same time, it also supports customized channel processing requirements for video channels, reducing the manual intervention of the system and improving efficiency.

[0204] 3) A content classification model based on global features is used to distinguish the content categories of video images. Different filtering ratio thresholds are set for different content categories, making filtering configurable and model inference more efficient. This automatically triggers the filtering strategies required for different content categories, supports the retrieval needs of different image content, and improves the processing capability for different video content categories.

[0205] The video retrieval method provided in this application has been described above in conjunction with exemplary applications and implementations of the electronic devices provided in the embodiments of this application. This application also provides a video retrieval device. In practical applications, the functional modules in the video retrieval device can be collaboratively implemented by the hardware resources of the electronic device (such as a terminal or server), such as computing resources like processors, communication resources (such as those used to support various communication methods like optical fiber and cellular), and memory. Figure 2 The video retrieval device 555 stored in the memory 550 is shown. It can be software in the form of programs and plug-ins, such as software modules designed in programming languages ​​such as C / C++ and Java, application software designed in programming languages ​​such as C / C++ and Java, or dedicated software modules, application programming interfaces, plug-ins, cloud services, etc. in large software systems. Examples of different implementation methods are given below.

[0206] The video retrieval device 555 includes a series of modules, such as a task extraction module 5551, a first retrieval module 5552, and a second retrieval module 5553. The following describes how the various modules in the video retrieval device 555 provided in this application embodiment work together to implement the video retrieval scheme.

[0207] The task extraction module 5551 is used to perform multi-task extraction processing on the video to be retrieved to obtain the global features, content category, and channel category of the video to be retrieved; the first retrieval module 5551 is used to retrieve multiple reference videos from the first video set based on the global features and content category of the video to be retrieved to form a second video set; the second retrieval module 5552 is used to determine the modal features of the video to be retrieved based on the channel category of the video to be retrieved, and retrieve the target video from the multiple reference videos in the second video set based on the modal features of the video to be retrieved, so as to use the target video as the retrieval result of the video to be retrieved.

[0208] In some embodiments, the multi-task extraction processing of the video to be retrieved is implemented through a multi-task model, which includes a basic feature layer, a feature embedding layer, a content classification layer, and a channel classification layer. The task extraction module 5551 is further configured to extract basic features from the video to be retrieved through the basic feature layer to obtain the basic features of the video to be retrieved; perform embedding vector transformation processing on the basic features of the video to be retrieved through the feature embedding layer to obtain the global features of the video to be retrieved; perform content classification processing on the basic features of the video to be retrieved through the content classification layer to obtain the content category of the video to be retrieved; and perform channel classification processing on the basic features of the video to be retrieved through the channel classification layer to obtain the channel category of the video to be retrieved.

[0209] In some embodiments, the basic feature layer includes a pooling layer and multiple cascaded convolutional layers, and the video to be retrieved includes multiple image frames; the task extraction module 5551 is further configured to perform cascaded convolution processing on each of the image frames through the multiple cascaded convolutional layers to obtain convolutional features of each of the image frames; perform pooling processing on the convolutional features of each of the image frames through the pooling layer to obtain image features of each of the image frames; and perform concatenation processing on the image features of each of the image frames to obtain the basic features of the video to be retrieved.

[0210] In some embodiments, the apparatus further includes: a model training module, configured to: extract basic features from the video sample through the basic feature layer to obtain basic features of the video sample; perform embedding vector transformation processing on the basic features of the video sample through the feature embedding layer to obtain global features of the video sample; perform content classification processing on the basic features of the video sample through the content classification layer to obtain content category of the video sample; perform channel classification processing on the basic features of the video sample through the channel classification layer to obtain channel category of the video sample; construct a target loss function for the multi-task model based on the global features of the video sample, the content category of the video sample, and the channel category of the video sample; and update the model parameters of the multi-task model based on the target loss function.

[0211] In some embodiments, the model training module is further configured to: construct an embedding loss function for the multi-task model based on the global features of the video samples, the global features of the positive video samples in the video samples, and the global features of the negative video samples in the video samples; construct a content classification loss function for the multi-task model based on the content category and the content category label of the video samples; construct a channel classification loss function for the multi-task model based on the channel category and the channel category label of the video samples; and perform a weighted summation of the embedding loss function, the content classification loss function, and the channel classification loss function to obtain the target loss function of the multi-task model.

[0212] In some embodiments, the model training module is further configured to perform similarity matching between the global features of the video sample and the global features of the positive video samples in the video sample to obtain a first similarity between the video sample and the positive video sample; perform similarity matching between the global features of the video sample and the global features of the negative video samples in the video sample to obtain a second similarity between the video sample and the negative video samples; and construct the embedding loss function of the multi-task model based on the first similarity and the second similarity.

[0213] In some embodiments, before constructing the embedding loss function of the multi-task model based on the global features of the video samples, the global features of the positive video samples in the video samples, and the global features of the negative video samples in the video samples, the apparatus further includes: a sample construction module, configured to extract at least one candidate video sample from a plurality of candidate positive video sample pairs, wherein the plurality of candidate positive video sample pairs do not include the video sample; obtain the similarity between each candidate video sample and the video sample, and sort the corresponding candidate video samples in descending order of the similarity; and take the portion of the candidate video samples that are ranked later in the sorting result as the negative video samples in the video samples.

[0214] In some embodiments, the first video set includes multiple videos, each video including multiple reference image frames; the video to be retrieved includes multiple image frames, the global features of the video to be retrieved include the global features of each image frame, and the content category of the video to be retrieved includes the content category of each image frame; the first retrieval module 5552 is further configured to perform the following processing for each image frame: searching for the global features of a target number of reference image frames from the global feature index table of the video index corresponding to the first video set; obtaining the Euclidean distance between the global features of each reference image frame and the global features of the image frame in the target number of reference image frames respectively; determining the distance threshold corresponding to the content category of the image frame, and selecting multiple reference image frames from the target number of reference image frames whose Euclidean distance is less than the distance threshold; and retrieving multiple reference videos from the multiple videos in the first video set based on the selected multiple reference image frames.

[0215] In some embodiments, the first retrieval module 5552 is further configured to: search for the video identifier of the video to which each of the reference image frames belongs from the identifier index table of the video index corresponding to the first video set; select the video corresponding to each of the video identifiers as candidate videos from the plurality of videos in the first video set; obtain the selection duration corresponding to the selected reference image frames in each candidate video, and the total duration of each candidate video; obtain a first ratio of each selection duration to the total duration, and retrieve candidate videos from the plurality of candidate videos in the first video set whose first ratio is greater than a ratio threshold as the reference videos.

[0216] In some embodiments, the apparatus further includes: a video set creation module, configured to obtain the matching duration of image frames that match multiple reference video frames of each of the reference videos in the plurality of image frames of the video to be retrieved, and the duration of the video to be retrieved; obtain a second ratio of each matching duration to the duration of the video to be retrieved, and sort each of the reference videos in descending order based on the second ratio to obtain a reference video sequence, thereby forming the second video set.

[0217] In some embodiments, the apparatus further includes: a video index construction module, configured to initialize a global feature index table, a time index table, and an identifier index table for the video index corresponding to the first video set; and to perform the following processing for each reference image frame: obtaining the global features of the reference image frame, the time point of the reference image frame in the corresponding video, and the video identifier of the video to which the reference image frame belongs; adding the global features of the reference image frame to the global feature index table, adding the time point of the reference image frame in the corresponding video to the time index table, and adding the video identifier of the video to which the reference image frame belongs to the identifier index table, thereby constructing a video index corresponding to the first video set.

[0218] In some embodiments, the second retrieval module 5553 is further configured to: when the channel category indicates that the video to be retrieved is a text category, perform text feature extraction processing on the text data of the video to be retrieved to obtain text features of the video to be retrieved; when the channel category indicates that the video to be retrieved is an audio category, perform audio feature extraction processing on the audio data of the video to be retrieved to obtain audio features of the video to be retrieved; when the channel category indicates that the video to be retrieved is a multi-target category, perform local feature extraction processing on the multi-targets in the video to be retrieved to obtain local features of the video to be retrieved; and determine at least one of the text features, the audio features, and the local features as the modal features of the video to be retrieved.

[0219] In some embodiments, the second retrieval module 5553 is further configured to perform similarity matching between the modal features of the video to be retrieved and the modal features of each of the reference videos in the second video set to obtain corresponding similarity values; and to retrieve, from the multiple reference videos in the second video set, reference videos with similarity values ​​greater than a similarity threshold as the target videos.

[0220] In some embodiments of the above scheme, after retrieving multiple reference videos from the first video set, the device further includes: a third retrieval module, configured to extract local features from the video to be retrieved when the channel category indicates that the video to be retrieved is a multi-target category, to obtain local features of the video to be retrieved; based on the local features, to retrieve multiple candidate target videos from the multiple reference videos of the second video set to form a third video set; the second retrieval module is further configured to retrieve target videos from the multiple candidate target videos of the third video set.

[0221] In some embodiments, the apparatus further includes: a feedback output module, configured to combine the search results and the channel category to output feedback information for the video to be searched; wherein the feedback information is configured to provide recommendation information or authorization prompt information for the video to be searched.

[0222] This application provides a computer program product, which includes a computer program or instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer program or instructions from the computer-readable storage medium and executes the computer program or instructions, causing the computer device to perform the video retrieval method described in this application.

[0223] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the video retrieval method provided in this application. For example, ... Figure 3 The method shown.

[0224] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0225] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0226] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0227] As an example, executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0228] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A video retrieval method, characterized in that, The method is implemented through a multi-task model, which includes a basic feature layer, a feature embedding layer, a content classification layer, and a channel classification layer. During training, the multi-task model is trained using an asynchronous learning rate on the basic feature layer, the feature embedding layer, the content classification layer, and the channel classification layer. The method includes: The basic feature layer extracts basic features from the video to be retrieved to obtain the basic features of the video to be retrieved; the feature embedding layer performs embedding vector transformation on the basic features of the video to be retrieved to obtain the global features of the video to be retrieved; the content classification layer performs content classification on the basic features of the video to be retrieved to obtain the content category of the video to be retrieved; and the channel classification layer performs channel classification on the basic features of the video to be retrieved to obtain the channel category of the video to be retrieved. For each image frame in the video to be retrieved, the global features of a target number of reference image frames are searched from the global feature index table of the video index corresponding to the first video set. The first video set includes multiple videos, and each video includes multiple reference image frames. The video to be retrieved includes multiple image frames, the global features of the video to be retrieved include the global features of each image frame, and the content category of the video to be retrieved includes the content category corresponding to each image frame. Obtain the Euclidean distance between the global features of each of the reference image frames and the global features of the image frame; determine the distance threshold corresponding to the content category of the image frame, and select multiple reference image frames from the target number of reference image frames whose Euclidean distance is less than the distance threshold; Based on the selected multiple reference image frames, multiple reference videos are retrieved from the first video set to form a second video set; Based on the channel category of the video to be retrieved, the modal features of the video to be retrieved are determined, and based on the modal features of the video to be retrieved, the target video is retrieved from multiple reference videos in the second video set, and the target video is used as the retrieval result of the video to be retrieved. When the channel category indicates that the video to be retrieved is a multi-target category, local feature extraction is performed on the video to be retrieved to obtain the local features of the video to be retrieved; based on the local features, multiple candidate target videos are retrieved from multiple reference videos in the second video set to form a third video set; based on the channel category of the video to be retrieved, the multimodal features of the video to be retrieved are determined. Based on the multimodal features of the video to be retrieved, the target video is retrieved from multiple candidate target videos in the third video set.

2. The method as described in claim 1, characterized in that, The basic feature layer includes a pooling layer and multiple cascaded convolutional layers; The step of extracting basic features from the video to be retrieved through the basic feature layer to obtain the basic features of the video to be retrieved includes: Each of the image frames is subjected to cascaded convolutional processing through the multiple cascaded convolutional layers to obtain the convolutional features of each of the image frames. The convolutional features of each image frame are pooled through the pooling layer to obtain the image features of each image frame; The image features of each of the image frames are stitched together to obtain the basic features of the video to be retrieved.

3. The method as described in claim 1, characterized in that, The method further includes: The basic features of the video samples are obtained by extracting basic features from the basic feature layer. The basic features of the video sample are transformed by the feature embedding layer to obtain the global features of the video sample. The content classification layer performs content classification processing on the basic features of the video sample to obtain the content category of the video sample; The basic features of the video sample are classified by the channel classification layer to obtain the channel category of the video sample. Based on the global features of the video samples, the content category of the video samples, and the channel category of the video samples, a target loss function for the multi-task model is constructed, and the model parameters of the multi-task model are updated based on the target loss function.

4. The method as described in claim 3, characterized in that, The objective loss function of the multi-task model, constructed based on the global features of the video samples, the content category of the video samples, and the channel category of the video samples, includes: Based on the global features of the video samples, the global features of the positive video samples in the video samples, and the global features of the negative video samples in the video samples, the embedding loss function of the multi-task model is constructed. Based on the content category and content category label of the video sample, the content classification loss function of the multi-task model is constructed. Based on the channel category and channel category label of the video sample, the channel classification loss function of the multi-task model is constructed; The embedding loss function, the content classification loss function, and the channel classification loss function are weighted and summed to obtain the target loss function of the multi-task model.

5. The method as described in claim 4, characterized in that, The embedding loss function of the multi-task model is constructed based on the global features of the video samples, the global features of the positive video samples in the video samples, and the global features of the negative video samples in the video samples, including: The global features of the video sample are matched with the global features of the positive video sample in the video sample to obtain the first similarity between the video sample and the positive video sample. The global features of the video sample are matched with the global features of the negative video sample in the video sample to obtain the second similarity between the video sample and the negative video sample. Based on the first similarity and the second similarity, the embedding loss function of the multi-task model is constructed.

6. The method according to claim 4, characterized in that, Before constructing the embedding loss function of the multi-task model based on the global features of the video samples, the global features of the positive video samples in the video samples, and the global features of the negative video samples in the video samples, the method further includes: At least one candidate video sample is extracted from a plurality of candidate positive video sample pairs, wherein the plurality of candidate positive video sample pairs do not include the video sample; Obtain the similarity between each candidate video sample and the video sample, and sort the corresponding candidate video samples in descending order of similarity. The candidate video samples that are ranked later in the sorting results are taken as negative video samples in the video samples.

7. The method as described in claim 1, characterized in that, The step of retrieving multiple reference videos from the first video set based on the selected multiple reference image frames includes: From the identifier index table corresponding to the video index of the first video set, find the video identifier of the video to which each of the reference image frames belongs; From the multiple videos in the first video set, select the video corresponding to each video identifier as a candidate video; Obtain the selection duration corresponding to the selected reference image frame in each of the candidate videos, and the total duration of each of the candidate videos; Obtain a first ratio of each selected duration to the total duration, and retrieve candidate videos from multiple candidate videos in the first video set whose first ratio is greater than a ratio threshold as the reference videos.

8. The method as described in claim 7, characterized in that, The method further includes: The matching duration of the image frames that match the multiple image frames of the video to be retrieved and the multiple reference video frames of each of the reference videos is obtained, as well as the duration of the video to be retrieved. A second ratio is obtained between the matching duration of each video and the duration of the video to be retrieved. Based on the second ratio, the reference videos are sorted in descending order to obtain a reference video sequence, thereby forming the second video set.

9. The method as described in claim 1, characterized in that, The method further includes: Initialize the global feature index table, time index table, and identifier index table of the video index corresponding to the first video set; For each of the reference image frames, the following processing is performed: Obtain the global features of the reference image frame, the time point of the reference image frame in the corresponding video, and the video identifier of the video to which the reference image frame belongs; The global features of the reference image frame are added to the global feature index table, the time point of the reference image frame in the corresponding video is added to the time index table, and the video identifier of the video to which the reference image frame belongs is added to the identifier index table to construct the video index corresponding to the first video set.

10. The method as described in claim 1, characterized in that, Determining the modal features of the video to be retrieved based on its channel category includes: When the channel category indicates that the video to be retrieved is a text category, text feature extraction processing is performed on the text data of the video to be retrieved to obtain the text features of the video to be retrieved; When the channel category indicates that the video to be retrieved is an audio category, audio feature extraction processing is performed on the audio data of the video to be retrieved to obtain the audio features of the video to be retrieved. When the channel category indicates that the video to be retrieved is a multi-target category, local feature extraction processing is performed on the multi-targets in the video to be retrieved to obtain the local features of the video to be retrieved. At least one of the text features, the audio features, and the local features is determined as the modal feature of the video to be retrieved.

11. The method as described in claim 1, characterized in that, The step of retrieving the target video from multiple reference videos in the second video set based on the modal features of the video to be retrieved includes: The modal features of the video to be retrieved are matched with the modal features of each of the reference videos in the second video set to obtain the corresponding similarity values. From the multiple reference videos in the second video set, the reference videos with similarity values ​​greater than a similarity threshold are retrieved as the target videos.

12. The method as described in claim 1, characterized in that, The method further includes: Based on the search results and the channel category, output feedback information for the video to be searched; The feedback information is used to provide recommendation information or authorization prompts for the video to be searched.

13. A video retrieval device, characterized in that, The device performs video retrieval through a multi-task model, which includes a basic feature layer, a feature embedding layer, a content classification layer, and a channel classification layer. During training, the multi-task model is trained using an asynchronous learning rate on the basic feature layer, the feature embedding layer, the content classification layer, and the channel classification layer. The device includes: The task extraction module is used to extract basic features from the video to be retrieved through the basic feature layer to obtain the basic features of the video to be retrieved; to perform embedding vector transformation on the basic features of the video to be retrieved through the feature embedding layer to obtain the global features of the video to be retrieved; to perform content classification on the basic features of the video to be retrieved through the content classification layer to obtain the content category of the video to be retrieved; and to perform channel classification on the basic features of the video to be retrieved through the channel classification layer to obtain the channel category of the video to be retrieved. The first retrieval module is configured to, for each image frame in the video to be retrieved, search for the global features of a target number of reference image frames from the global feature index table of the video index corresponding to the first video set, wherein the first video set includes multiple videos, each video includes multiple reference image frames; the video to be retrieved includes multiple image frames, the global features of the video to be retrieved include the global features of each image frame, and the content category of the video to be retrieved includes the content category corresponding to each image frame; obtain the Euclidean distance between the global features of each reference image frame and the global features of the image frame; determine the distance threshold corresponding to the content category of the image frame, and select multiple reference image frames from the target number of reference image frames whose Euclidean distance is less than the distance threshold; Based on the selected multiple reference image frames, multiple reference videos are retrieved from the first video set to form a second video set; The second retrieval module is used to determine the modal features of the video to be retrieved based on the channel category of the video to be retrieved, and to retrieve the target video from multiple reference videos in the second video set based on the modal features of the video to be retrieved, so as to use the target video as the retrieval result of the video to be retrieved; The third retrieval module is used to extract local features of the video to be retrieved when the channel category indicates that the video to be retrieved is a multi-target category, to obtain the local features of the video to be retrieved; based on the local features, to retrieve multiple candidate target videos from multiple reference videos in the second video set to form a third video set; and to determine the multimodal features of the video to be retrieved based on the channel category of the video to be retrieved. The second retrieval module is further configured to retrieve the target video from multiple candidate target videos in the third video set based on the multimodal features of the video to be retrieved.

14. The apparatus as claimed in claim 13, characterized in that, The basic feature layer includes pooling layers and multiple cascaded convolutional layers. The task extraction module is also used for: Each of the image frames is subjected to cascaded convolutional processing through the multiple cascaded convolutional layers to obtain the convolutional features of each of the image frames. The convolutional features of each image frame are pooled through the pooling layer to obtain the image features of each image frame; The image features of each of the image frames are stitched together to obtain the basic features of the video to be retrieved.

15. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the video retrieval method according to any one of claims 1 to 12.

16. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the video retrieval method according to any one of claims 1 to 12 when executed by a processor.

17. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method described in any one of claims 1 to 12.

Images