Video retrieval method and device, storage medium, computer device and program product

Abstract

The application provides a video retrieval method and device, a storage medium, a computer device and a program product. The method comprises: performing video frame extraction processing on a first video according to a video frame time sequence to obtain a plurality of candidate key video frames; performing clustering processing on the plurality of candidate key video frames to obtain a plurality of clusters, and extracting target candidate key video frames from the plurality of clusters respectively to serve as key video frames; performing image feature extraction processing on the plurality of extracted key video frames respectively to obtain image features corresponding to the plurality of key video frames respectively; performing image feature fusion processing on the image features corresponding to the plurality of key video frames respectively to obtain a video feature of the first video; and based on the video feature of the first video and video features corresponding to a plurality of candidate videos respectively, searching for a second video matching the first video from the plurality of candidate videos. The application can extract accurate and comprehensive video features from a video, thereby improving the accuracy of video retrieval.

Description

Technical Field

[0001] This application relates to the field of computers, and in particular to a video retrieval method, apparatus, storage medium, computer equipment, and program product. Background Technology

[0002] In today's digital age, image content is experiencing rapid growth. From personal photos and videos shared on social media platforms, to images used by businesses for product demonstrations and marketing, and to massive amounts of image data used for data analysis in scientific research, the application scenarios for image content are ubiquitous and their quantity is increasing dramatically. Against this backdrop, the need for retrieval has become extremely urgent, especially video retrieval, which plays a crucial role in numerous scenarios.

[0003] Among the solutions provided by related technologies, keyword-based video retrieval is commonly used. This involves matching keywords input by the user with the video's metadata (such as title, description, and tags) to achieve video retrieval. However, the image content in videos is usually rich and diverse, and the metadata is difficult to fully and comprehensively summarize the video, resulting in low accuracy in video retrieval. Summary of the Invention

[0004] This application provides a video retrieval method, apparatus, storage medium, computer equipment, and program product that can extract accurate and comprehensive video features from videos, thereby improving the accuracy of video retrieval.

[0005] To solve the above-mentioned technical problems, this application provides the following technical solution: According to one aspect of this application, a video retrieval method is provided, comprising: The first video is processed by extracting video frames according to the video frame time sequence to obtain multiple candidate key video frames; Clustering is performed on the multiple candidate key video frames to obtain multiple clusters, and target candidate key video frames are extracted from the multiple clusters as key video frames. Image feature extraction processing is performed on the extracted key video frames to obtain the image features corresponding to the key video frames respectively. Image feature fusion processing is performed on the image features corresponding to the multiple key video frames to obtain the video features of the first video. Based on the video features of the first video and the video features corresponding to multiple candidate videos, a second video matching the first video is retrieved from the multiple candidate videos.

[0006] In the above scheme, the step of extracting target candidate key video frames from the plurality of clusters as key video frames includes: The multiple clusters are traversed, and the following processing is performed on the traversed clusters: Subtitle recognition processing is performed on multiple candidate key video frames in the traversed clusters respectively; Extract target candidate key video frames, including subtitles, from the traversed clusters to serve as key video frames.

[0007] In the above scheme, the step of extracting target candidate key video frames from the plurality of clusters as key video frames includes: The multiple clusters are traversed, and the following processing is performed on the traversed clusters: Target recognition processing is performed on multiple candidate key video frames in the traversed clusters respectively to obtain the number of targets included in each of the multiple candidate key video frames in the traversed clusters. Extract the target candidate key video frames that include the maximum number of targets from the traversed clusters, and use them as key video frames.

[0008] In the above scheme, the number of key video frames is M; after extracting target candidate key video frames from the multiple clusters as key video frames, the video retrieval method further includes: The M key video frames are sampled according to the video frame timing to obtain N key video frames, where N is less than M and both N and M are integers greater than 0. The step of performing image feature extraction processing on the extracted key video frames to obtain the image features corresponding to the key video frames respectively includes: Image feature extraction is performed on each of the N key video frames to obtain the image features corresponding to each of the N key video frames.

[0009] According to one aspect of this application, a video retrieval device is provided, comprising: The first extraction module is used to extract video frames from the first video according to the video frame time sequence to obtain multiple candidate key video frames. The second extraction module is used to perform clustering processing on the multiple candidate key video frames to obtain multiple clusters, and extract target candidate key video frames from the multiple clusters respectively as key video frames. The feature extraction module is used to perform image feature extraction processing on the extracted key video frames respectively to obtain the image features corresponding to the key video frames respectively. The feature fusion module is used to perform image feature fusion processing on the image features corresponding to the multiple key video frames respectively to obtain the video features of the first video. The feature retrieval module is used to retrieve a second video that matches the first video from among the multiple candidate videos, based on the video features of the first video and the video features corresponding to the multiple candidate videos.

[0010] In the above scheme, the first extraction module is further configured to: The first video is processed by sliding a window according to the video frame sequence. After each slide of the sliding window, candidate key video frames are extracted from the sliding window based on the degree of difference between different video frames in the sliding window.

[0011] In the above scheme, the first extraction module is further configured to: The pixel values ​​corresponding to multiple pixels of the video frame in the sliding window are fused to obtain fused pixel values. The degree of difference between different video frames in the sliding window is obtained by performing an absolute difference calculation on the fused pixel values ​​corresponding to different video frames in the sliding window.

[0012] In the above scheme, the first extraction module is further configured to: The maximum difference is determined based on the difference between different video frames in the sliding window; The video frame corresponding to the maximum difference is extracted from the sliding window as a candidate key video frame.

[0013] In the above scheme, the second extraction module is further used for: Each of the multiple candidate key video frames is subjected to feature statistical processing to obtain the statistical features corresponding to each of the multiple candidate key video frames. Determine the feature distances between the statistical features corresponding to different candidate key video frames; The candidate key video frames are clustered based on the feature distance to obtain multiple clusters.

[0014] In the above scheme, the second extraction module is further used for: The plurality of candidate key video frames are traversed, and the following processing is performed on the traversed candidate key video frames: A pixel value vector is constructed based on the pixel values ​​corresponding to multiple pixels in the traversed candidate key video frames. The pixel value vector is determined as the statistical feature of the candidate key video frames traversed.

[0015] In the above scheme, the second extraction module is further used for: The multiple clusters are traversed, and the following processing is performed on the traversed clusters: Smoothing detection processing is performed on multiple candidate key video frames in the traversed cluster to obtain smoothing parameters corresponding to the multiple candidate key video frames in the traversed cluster. The smoothing parameters are negatively correlated with the smoothing degree. Extract the target candidate key video frame corresponding to the maximum smoothness parameter from the traversed clusters, and use it as the key video frame.

[0016] In the above scheme, the second extraction module is further used for: The process involves iterating through multiple candidate key video frames within the traversed clusters, and performing the following processing on each candidate key video frame: Pixel value difference processing is performed on multiple pixels in the traversed candidate key video frames to obtain the difference pixel values ​​corresponding to the multiple pixels in the traversed candidate key video frames respectively. Differential pixel value fusion processing is performed on the differential pixel values ​​corresponding to multiple pixels in the traversed candidate key video frames to obtain the smoothing parameters of the traversed candidate key video frames.

[0017] In the above scheme, the second extraction module is further used for: The multiple clusters are traversed, and the following processing is performed on the traversed clusters: Subtitle recognition processing is performed on multiple candidate key video frames in the traversed clusters respectively; Extract target candidate key video frames, including subtitles, from the traversed clusters to serve as key video frames.

[0018] In the above scheme, the second extraction module is further used for: The multiple clusters are traversed, and the following processing is performed on the traversed clusters: Target recognition processing is performed on multiple candidate key video frames in the traversed clusters respectively to obtain the number of targets included in each of the multiple candidate key video frames in the traversed clusters. Extract the target candidate key video frames that include the maximum number of targets from the traversed clusters, and use them as key video frames.

[0019] In the above scheme, the feature retrieval module is further configured to: The video features of the first video are combined with the features of the first video in other modalities to obtain the multimodal features of the first video. Based on the multimodal features of the first video and the multimodal features corresponding to the multiple candidate videos, a second video matching the first video is retrieved from the multiple candidate videos.

[0020] In the above scheme, the features of the first video in other modalities include the content description features of the first video; the feature extraction module is further used for: Obtain the content description text of the first video; Text feature extraction processing is performed on the content description text to obtain the content description features of the first video.

[0021] In the above scheme, the feature extraction module is further used for: Image feature extraction is performed on multiple key video frames extracted by an image encoder to obtain the image features corresponding to each of the multiple key video frames. The text features of the first video are obtained by extracting text features from the content description text using a text encoder. The image encoder and the text encoder are obtained through joint training.

[0022] In the above scheme, the video retrieval device further includes a comparison learning module, used for: Obtain image-text pairs and sample labels, wherein the image-text pairs include sample images and sample text, and the sample labels are used to indicate whether there is a correspondence between the sample images and the sample text; The image encoder performs image feature extraction processing on the sample image to obtain the image features of the sample image; The text encoder is used to extract text features from the sample text to obtain the text features of the sample text. Determine the similarity between the image features of the sample image and the text features of the sample text, and determine the loss value based on the similarity and the sample label; The image encoder and the text encoder are trained based on the loss value.

[0023] In the above scheme, the features of the first video in other modalities include the duration feature of the first video; the feature extraction module is further used for: Determine the video duration of the first video; Construct a duration description text based on the video duration; The duration description text is subjected to text feature extraction processing to obtain the duration feature of the first video.

[0024] In the above scheme, the features of the first video in other modalities include the duration feature of the first video; the feature extraction module is further used for: Determine the video duration of the first video; The duration of the video is one-hot encoded to obtain the duration feature of the first video.

[0025] In the above scheme, the feature fusion module is further used for: Perform any of the following processes: The image features corresponding to the multiple key video frames are averaged across multiple dimensions to obtain the video features of the first video. The maximum value of the image features corresponding to the multiple key video frames is taken in multiple dimensions to obtain the video features of the first video. The image features corresponding to the multiple key video frames are averaged in multiple dimensions to obtain the average feature, and the image features corresponding to the multiple key video frames are maximized in multiple dimensions to obtain the maximum feature. The average feature and the maximum feature are summed to obtain the video feature of the first video.

[0026] In the above scheme, the feature retrieval module is further configured to: Multiple vector indices are established based on the video features corresponding to the multiple candidate videos; Determine the first similarity between the video features of the first video and the plurality of vector indices, and filter the plurality of vector indices according to the first similarity to obtain the filtered vector indices; A second similarity is determined between the video features of the first video and the video features of a plurality of third videos, and the plurality of third videos are filtered according to the second similarity to obtain a second video that matches the first video. The video features of the plurality of third videos are obtained by indexing the filtered vector index, and the plurality of third videos are a subset of the plurality of candidate videos.

[0027] In the above scheme, the number of key video frames is M; the video retrieval device further includes a sampling module, used for: The M key video frames are sampled according to the video frame timing to obtain N key video frames, where N is less than M and both N and M are integers greater than 0. The feature extraction module is also used for: Image feature extraction is performed on each of the N key video frames to obtain the image features corresponding to each of the N key video frames.

[0028] According to one aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute the video retrieval method described above.

[0029] According to one aspect of this application, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video retrieval method described above.

[0030] According to one aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium, and the processor executes the computer instructions to implement the above-described video retrieval method.

[0031] In related technologies, video retrieval is achieved by matching received keywords with video metadata (such as title, description, tags, etc.). However, the image content in videos is usually rich and diverse, and the metadata is difficult to fully and comprehensively summarize the video. To address this, this application extracts multiple representative key video frames from a first video, performs image feature fusion processing on the image features corresponding to the multiple key video frames respectively, obtains the video features of the first video, and uses the video features of the first video to represent the first video. The process of extracting key video frames includes two stages: coarse screening and fine screening. In the coarse screening stage, the first video is processed by extracting video frames according to their temporal order to obtain multiple candidate key video frames. This ensures that the multiple candidate key video frames are evenly distributed over time, meaning that each candidate key video frame can represent a video segment of the first video within a specific time period. In the fine screening stage, the multiple candidate key video frames are first clustered into multiple clusters. Since different candidate key video frames with similar image content will be clustered into one cluster, target candidate key video frames are extracted from multiple clusters to serve as key video frames. This ensures that the number of key video frames is not excessive, reducing the waste of computing resources and further improving the representativeness of the key video frames. After obtaining the video features of the first video, this application retrieves a second video that matches the first video from among the multiple candidate videos based on the video features of the first video and the video features corresponding to the multiple candidate videos. Since video features can comprehensively and completely summarize the video, the accuracy of video retrieval can be improved, that is, the probability of the second video actually matching the first video can be increased.

[0032] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0033] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0034] Figure 1 This is a schematic diagram of the architecture of the video retrieval system provided in an embodiment of this application; Figure 2 This is a schematic diagram of a scenario for the video retrieval method provided in an embodiment of this application; Figure 3 This is another scenario illustration of the video retrieval method provided in the embodiments of this application; Figure 4 This is another scenario illustration of the video retrieval method provided in the embodiments of this application; Figure 5 This is a flowchart illustrating a video retrieval method provided in an embodiment of this application; Figure 6 This is a schematic diagram of extracting candidate key video frames provided in an embodiment of this application; Figure 7 This is a schematic diagram of extracting key video frames according to an embodiment of this application; Figure 8 This is another flowchart illustrating the video retrieval method provided in this application embodiment; Figure 9 This is a schematic diagram of a sliding window provided in an embodiment of this application; Figure 10 This is another flowchart illustrating the video retrieval method provided in this application embodiment; Figure 11 This is another flowchart illustrating the video retrieval method provided in this application embodiment; Figure 12 This is another flowchart illustrating the video retrieval method provided in this application embodiment; Figure 13 This is a schematic diagram of feature extraction provided in an embodiment of this application; Figure 14 This is another schematic diagram of feature extraction provided in the embodiments of this application; Figure 15 This is another flowchart illustrating the video retrieval method provided in this application embodiment; Figure 16 This is a schematic diagram of a vector index provided in an embodiment of this application; Figure 17 This is a schematic diagram of an interface of the video platform provided in an embodiment of this application; Figure 18 This is a schematic diagram of the video retrieval results provided in an embodiment of this application; Figure 19 This is another flowchart illustrating the video retrieval method provided in this application embodiment; Figure 20 This is a schematic diagram of the video retrieval device provided in an embodiment of this application; Figure 21 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application; Figure 22 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] It should be noted that while some processes described in the specification, claims, and accompanying drawings contain multiple steps that appear in a specific order, it should be clearly understood that these steps may not be performed in the order they appear herein, or may be performed in parallel. The step numbers are merely used to distinguish different steps and do not represent any particular order of execution. Furthermore, descriptions such as "first," "second," or "objective" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0037] It is worth noting that in the specific embodiments of this application, video and other related data are involved. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the subject is required, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards. For example, when an embodiment of this application needs to obtain video data, separate permission or consent from the subject can be obtained through pop-up windows or redirection to a confirmation page. After obtaining the subject's separate permission or consent, the necessary video data for enabling the embodiment of this application to function properly can then be obtained.

[0038] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows: 1) Video Retrieval: This refers to the process of finding videos relevant to a query from a large number of videos. Related technologies have limited implementations of video retrieval, and most focus on text-based queries, such as matching keywords entered by an object with video metadata. This application's embodiment uses video-to-video retrieval, that is, retrieving other videos using a specific video.

[0039] 2) Video Frame: A video is a still image in a video. A video is a dynamic image formed by switching between a series of consecutive still images. For example, if a video has a frame rate of 24 frames per second, it means that 24 different video frames will switch every second.

[0040] 3) Key Frame: This refers to a frame in a video that is of great significance. These frames are usually used to represent the main changes or important events in a scene and can effectively capture the key moments of the video content.

[0041] 4) Clustering: This is an unsupervised learning method used to divide data objects (such as video frames) in a dataset into different clusters. Data objects within a cluster have high similarity, while data objects belonging to different clusters have high differences.

[0042] 5) Sliding window: A sliding window is a fixed-size window defined on an ordered data sequence. This window can slide along the data sequence with a certain step size. After each slide, various operations can be performed on the data within the window. The sliding window has two key parameters: size and step size. The size determines the amount of data that the sliding window can process each time, and the step size determines the distance the sliding window slides each time.

[0043] 6) Features: These are attributes or variables used to describe and distinguish different data objects, and are also an abstract representation of data objects.

[0044] 7) Multimodal: This refers to the processing and analysis of multiple different types of data or information sources. In the fields of machine learning and artificial intelligence, multimodal usually refers to using multiple modalities (such as text, images, etc.) simultaneously for learning and reasoning, thereby enabling the model to achieve better performance.

[0045] Faced with the need for video retrieval, the solutions provided by related technologies typically employ keyword-based video retrieval. This relies on keywords input by the user and matching them with the video's metadata (such as title, description, and tags) to achieve video retrieval. However, the image content in videos is usually rich and diverse, making it difficult for metadata to fully and comprehensively summarize the video, resulting in low accuracy in video retrieval.

[0046] To address this, embodiments of this application provide a video retrieval method, apparatus, storage medium, computer device, and program product. The method proposes first extracting multiple candidate key video frames according to the temporal sequence of the video frames, and then extracting multiple key video frames from these candidate key video frames through clustering. This improves the representativeness of the key video frames. Next, image feature extraction processing is performed on each of the multiple key video frames to obtain the image features corresponding to each key video frame. These image features are then fused into a single video feature. Compared to metadata, video features can comprehensively and completely summarize a video. Therefore, by comparing the video features of different videos, embodiments of this application can improve the accuracy of video retrieval, i.e., increase the probability that the retrieved second video actually matches the first video.

[0047] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a video retrieval system provided in an embodiment of this application. It includes a terminal 140, an Internet connection 130, a gateway 120, a server 110, etc.

[0048] Terminal 140 can take various forms, including desktop computers, laptops, PDAs (personal digital assistants), mobile phones, vehicle terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. For example, multiple desktop computers can be interconnected via a local area network, sharing a single monitor to work collaboratively, forming a single terminal 140. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0049] Server 110 refers to a computer system capable of providing certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Server 110 can also communicate with the Internet 130 via wired or wireless means to exchange data.

[0050] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.

[0051] This embodiment of the application can be implemented by terminal 140, by server 110, or by both terminal 140 and server 110. Examples will be provided below, illustrating various scenarios.

[0052] (a) Photo album management scenario.

[0053] As an example, the following are provided: Figure 2 The local photo album interface of terminal 140 shown includes multiple videos such as Video 1, Video 2, ..., Video 6. When terminal 140 receives a search operation triggered for the first video in the local photo album, it treats all other videos in the local photo album as candidate videos. Figure 2 The example provided is video 1, which is the first video in the text. The triggering method for the search operation is not limited, such as... Figure 2 As shown, when video 1 is selected, terminal 140 will display send option, search option, add option and delete option. Terminal 140 will determine the received trigger operation (such as click operation) for the search option as the search operation.

[0054] Then, terminal 140 performs video frame extraction processing on the first video according to the video frame time sequence to obtain multiple candidate key video frames; it performs clustering processing on the multiple candidate key video frames to obtain multiple clusters, and extracts target candidate key video frames from each cluster as key video frames; it performs image feature extraction processing on the extracted multiple key video frames to obtain image features corresponding to each key video frame; it performs image feature fusion processing on the image features corresponding to each key video frame to obtain the video features of the first video; based on the video features of the first video and the video features corresponding to each of the multiple candidate videos, it retrieves a second video that matches the first video from the multiple candidate videos. Terminal 140 can display the retrieved second video. Figure 2 In the example, the second video includes video 2, video 5, and video 6.

[0055] By employing the above method, a second video can be accurately retrieved based on the first video in the local photo album, thereby improving the efficiency and convenience of album management and meeting the personalized management needs of the local photo album. For example, if an user of terminal 140 is interested in the first video in the local photo album, terminal 140 can retrieve a matching second video using the above method so that the user can watch the second video; conversely, if an user of terminal 140 is not interested in the first video in the local photo album, terminal 140 can retrieve a matching second video using the above method so that the user can delete both the first and second videos, thus streamlining the local photo album.

[0056] (ii) Online video retrieval scenario.

[0057] For example, server 110 is the backend server of a video platform, and terminal 140 runs a client of the video platform. Terminal 140 accesses server 110 to obtain relevant video data, thereby enabling online video playback. When terminal 140 plays the first video, server 110 can consider other videos in the video platform as candidate videos. Then, server 110 performs video frame extraction processing on the first video according to the video frame sequence to obtain multiple candidate key video frames; it performs clustering processing on the multiple candidate key video frames to obtain multiple clusters, and extracts target candidate key video frames from each cluster as key video frames; it performs image feature extraction processing on the extracted multiple key video frames to obtain image features corresponding to each key video frame; it performs image feature fusion processing on the image features corresponding to each key video frame to obtain the video features of the first video; based on the video features of the first video and the video features corresponding to the multiple candidate videos, it retrieves a second video that matches the first video from the multiple candidate videos. After retrieving the second video, server 110 sends the relevant data of the second video to terminal 140 for display. Figure 3 The video playback interface of terminal 140 is shown. The first video being played in the video playback interface is video 1. The second videos (i.e. matching videos) displayed in the video playback interface include video 2, video 5 and video 6. For the second video, only the cover image, name and other data can be displayed to help the user quickly understand the general situation of the second video.

[0058] By using the methods described above, other videos closely related to the currently playing video can be recommended, improving the accuracy of video recommendations, satisfying the user's further viewing needs based on current interests, and increasing the probability that the user will discover content of interest on the video platform.

[0059] (III) Video classification scenarios.

[0060] For example, server 110 is the backend server of a video platform. The video platform includes multiple videos. Server 110 can use a video that has been categorized (e.g., manually labeled) as the first video, and all other videos that have not been categorized as candidate videos. Then, server 110 performs video frame extraction processing on the first video according to the video frame time sequence to obtain multiple candidate key video frames; it performs clustering processing on the multiple candidate key video frames to obtain multiple clusters, and extracts target candidate key video frames from each cluster as key video frames; it performs image feature extraction processing on the extracted multiple key video frames to obtain the image features corresponding to each key video frame; it performs image feature fusion processing on the image features corresponding to each key video frame to obtain the video features of the first video; based on the video features of the first video and the video features corresponding to each of the multiple candidate videos, it retrieves the second video that matches the first video from the multiple candidate videos. After retrieving the second video, server 110 assigns the second video the same category as the first video. Figure 4 In the example, taking the first video as Video 1 and Video 1 as food, and the second video including Video 2, Video 5 and Video 6 as an example, Video 2, Video 5 and Video 6 are also assigned the food category.

[0061] The above methods enable intelligent classification of the massive amounts of videos managed by video platforms. While ensuring classification accuracy, this greatly improves the efficiency of video classification, eliminating the need for manual labeling of each video, thus saving significant manpower and time costs. It also allows massive amounts of videos to be organized into accurate categories, facilitating subsequent management, retrieval, and recommendation, and providing a more accurate data foundation for the formulation of various category-based operational strategies.

[0062] It is worth noting that, Figure 1 The video retrieval system shown and Figures 2 to 4 The scenarios shown are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of video retrieval technology and the emergence of new business scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0063] In this embodiment, the description will be from the perspective of a video retrieval device. Specifically, the video retrieval device can be integrated into a computer device that has a storage unit and is equipped with a microprocessor and has computing capabilities. The computer device can be a server or a terminal, and there is no limitation on the latter.

[0064] Please see Figure 5 , Figure 5This is a flowchart illustrating the video retrieval method provided in an embodiment of this application. The video retrieval method can be implemented by a server and / or a terminal. Figure 5 The video retrieval methods shown include: Step 510: Extract video frames from the first video according to the video frame time sequence to obtain multiple candidate key video frames; Step 520: Cluster the multiple candidate key video frames to obtain multiple clusters, and extract the target candidate key video frames from the multiple clusters as key video frames. Step 530: Perform image feature extraction processing on the extracted key video frames to obtain the image features corresponding to the key video frames respectively. Step 540: Perform image feature fusion processing on the image features corresponding to multiple key video frames to obtain the video features of the first video; Step 550: Based on the video features of the first video and the video features corresponding to multiple candidate videos, retrieve the second video that matches the first video from the multiple candidate videos.

[0065] Steps 510 to 550 are described in detail below.

[0066] In step 510, the first video is processed by extracting video frames according to the video frame time sequence to obtain multiple candidate key video frames.

[0067] For ease of understanding, the video that triggers the search will be referred to as the first video. The first video consists of multiple video frames, which are switched according to a specific video frame sequence during playback, thus presenting a dynamic effect of video playback.

[0068] Here, the first video is processed by extracting video frames in chronological order to obtain multiple candidate key video frames. This process retains key information from the first video while reducing the amount of data processing required later. Step 510 is equivalent to coarsely screening the multiple video frames in the first video. The extracted candidate key video frames are relatively evenly distributed in time, meaning that each candidate key video frame can represent a video segment of the first video within a specific time period. This effectively avoids the problem of over-extracting video frames from certain time periods or ignoring video frames from other time periods.

[0069] In some embodiments, the first video is processed by extracting video frames according to the video frame time sequence to obtain multiple candidate key video frames, including: sampling multiple video frames in the first video according to the video frame time sequence to obtain multiple candidate key video frames.

[0070] Here, multiple video frames in the first video can be sampled according to the video frame sequence. That is, a video frame is extracted every preset number of video frames as a candidate key video frame. The preset number can be set according to the actual application scenario. For example... Figure 6 As shown, the first video includes video frames 1 to 10 (subsequent video frames are not shown). Video frames 1 to 10 can be sampled according to their temporal sequence. For example, starting from video frame 1, one video frame is extracted every three video frames as a candidate key video frame. The resulting candidate key video frames include video frame 1, video frame 5, and video frame 9. This method performs uniform sampling based on the temporal sequence of video frames, which can, to some extent, preserve the key information and content development of the first video. The extracted candidate key video frames can serve as representative samples of the first video, reflecting its content at different time periods.

[0071] In step 520, multiple candidate key video frames are clustered to obtain multiple clusters, and target candidate key video frames are extracted from multiple clusters as key video frames.

[0072] After extracting multiple candidate key video frames, clustering is performed on these frames to group similar frames into a single cluster. The clustering algorithm is not limited; it can be density-based spatial clustering of applications with noise (DBSCAN) or hierarchical density-based spatial clustering of applications with noise (HDBSCAN). In some embodiments, multiple candidate key video frames can be mapped to Euclidean space (i.e., mapped to vectors). The Euclidean distance between different candidate key video frames is determined in Euclidean space, and clustering is performed based on this distance. This improves the accuracy and interpretability of the clustering process. It is worth noting that Euclidean space is a finite-dimensional real vector space with favorable metric properties, allowing the definition of vector lengths and distances between two vectors.

[0073] After obtaining multiple clusters through clustering, target candidate key video frames are extracted from each cluster as key video frames. A target candidate key video frame refers to one or more candidate key video frames within a cluster; for example, it can refer to a randomly selected candidate key video frame within the cluster. Figure 7 As shown, multiple candidate key video frames (in Euclidean space) Figure 7Clustering (represented by points) yields two clusters, cluster 1 and cluster 2. A candidate key video frame (i.e., the target candidate key video frame) is randomly selected from cluster 1 as the key video frame, and a candidate key video frame (i.e., the target candidate key video frame) is randomly selected from cluster 2 as the key video frame. Based on step 510, step 520 is equivalent to fine-tuning multiple candidate key video frames, thereby ensuring that the extracted key video frames have a certain degree of difference, which improves the representativeness of the key video frames and ensures that the number of key video frames is not excessive, effectively reducing the amount of subsequent data processing.

[0074] In some embodiments, after clustering multiple candidate key video frames to obtain multiple clusters, the video retrieval method further includes: identifying candidate key video frames outside the multiple clusters as key video frames.

[0075] Here, after clustering, multiple candidate key video frames may appear outside of any cluster; these are outliers. Such candidate key video frames are directly identified as key video frames. Figure 7 As shown, there is a candidate key video frame outside of clusters 1 and 2, which can be identified as a key video frame. This method avoids missing crucial information in the first video due to the limitations of clustering algorithms. For example, in a suspense movie, a brief, highly suspenseful candidate key video frame may differ significantly from candidate key video frames in other regular scenes, becoming an outlier during clustering. However, this candidate key video frame is crucial to the movie's plot, and identifying it as a key video frame helps to fully preserve the movie's key information.

[0076] In step 530, image feature extraction processing is performed on the extracted key video frames to obtain the image features corresponding to the key video frames respectively.

[0077] For each extracted key video frame, image feature extraction is performed to obtain image features, which are represented in vector form. For example, statistical features can be obtained by performing feature statistical processing on the key video frame, and these statistical features are used as image features. In this case, the image features reflect more superficial and local information in the key video frame, such as the distribution of grayscale, color distribution, or texture. The method of feature statistical processing will be introduced later. Alternatively, the key video frame can be encoded using an image encoder to obtain encoded features, and these encoded features are used as image features. In this case, the image features can reflect the deeper semantics of the key video frame. The image encoder is a machine learning model built based on artificial intelligence principles to implement image encoding, such as the Convolutional Neural Network (CNN) model.

[0078] In some embodiments, the number of key video frames is M. After extracting target candidate key video frames from multiple clusters as key video frames, the video retrieval method further includes: sampling the M key video frames according to the video frame time sequence to obtain N key video frames, where N is less than M and both N and M are integers greater than 0; and performing image feature extraction processing on the extracted key video frames to obtain image features corresponding to the multiple key video frames, including: performing image feature extraction processing on the N key video frames to obtain image features corresponding to the N key video frames.

[0079] Here, there are M key video frames. Considering that the number of key video frames may still be too large, the M key video frames are sampled according to the video frame time sequence to obtain N key video frames. The sampled N key video frames are used for image feature extraction processing, where N is less than M, and both N and M are integers greater than 0. For example, if M=5 and N=3, and the M key video frames are arranged sequentially as key video frame 1, key video frame 2, key video frame 3, key video frame 4, and key video frame 5, then key video frames 1, 3, and 5 can be sampled sequentially for image feature extraction processing. The above method of further sampling the M key video frames according to the video frame time sequence can reduce the number of key video frames, reduce the amount of subsequent data processing, and at the same time retain as much key information in the first video as possible.

[0080] It's worth noting that a smaller value for N increases the likelihood of missing crucial information from the first video; a larger value for N increases the likelihood of introducing noise. Therefore, the value of N needs to be set based on the actual application scenario, and it's not simply a matter of smaller or larger values ​​being better.

[0081] In step 540, image feature fusion processing is performed on the image features corresponding to multiple key video frames to obtain the video features of the first video.

[0082] After extracting the image features corresponding to multiple key video frames, image feature fusion processing is performed on these features to obtain the video features of the first video. In this way, the video features of the first video can completely and comprehensively summarize the first video.

[0083] In some embodiments, image feature fusion processing is performed on the image features corresponding to multiple key video frames to obtain video features of a first video, including: performing any one of the following processes: averaging the image features corresponding to multiple key video frames in multiple dimensions to obtain video features of the first video; taking the maximum value of the image features corresponding to multiple key video frames in multiple dimensions to obtain video features of the first video; averaging the image features corresponding to multiple key video frames in multiple dimensions to obtain an average value feature; taking the maximum value of the image features corresponding to multiple key video frames in multiple dimensions to obtain a maximum value feature; and summing the average value feature and the maximum value feature to obtain video features of the first video.

[0084] For ease of understanding, key video frames include key video frame 1 and key video frame 2, and the image features of key video frame 1 are... The image features of key video frame 2 are For example, we will illustrate several methods of image feature fusion processing.

[0085] 1) The image features corresponding to multiple key video frames are averaged across multiple dimensions to obtain the video features of the first video. That is, the video features of the first video are... This approach focuses on the main distribution (overall features) of image features corresponding to multiple key video frames.

[0086] 2) The image features corresponding to multiple key video frames are maximized across multiple dimensions to obtain the video features of the first video. For example, , , Then the video features of the first video are This approach focuses on the specific and saliency features of the image features corresponding to multiple key video frames.

[0087] 3) Average the image features corresponding to multiple key video frames across multiple dimensions to obtain the average feature, and maximize the image features corresponding to multiple key video frames across multiple dimensions to obtain the maximum feature. Summate the average feature and the maximum feature to obtain the video features of the first video. For example, , , Then the characteristic of the average value is The maximum value characteristic is The video features of the first video are This method focuses not only on the main distribution of image features corresponding to multiple key video frames, but also on the specificity and saliency of these features. It's worth noting that summation is used because it's a linear combination method that maintains the independence of the average and maximum features to a certain extent, offering strong interpretability. Furthermore, summation is more intuitive in information fusion, stably combining the average and maximum features without the significant fluctuations caused by changes in feature values, unlike multiplication, which is less sensitive to noise.

[0088] In practical applications, you can choose any of the above methods to determine the video features of the first video, depending on the focus of your considerations. Alternatively, you can choose the method with the best experimental results to determine the video features of the first video after conducting experiments.

[0089] In some embodiments, after performing image feature fusion processing on the image features corresponding to multiple key video frames to obtain the video features of the first video, the video retrieval method further includes: performing regularization processing on the video features of the first video.

[0090] Here, the video features of the first video can be regularized, such as by L2 norm regularization or other regularization methods. Regularization normalizes the length of the video features to 1, ensuring that the video features of different videos are on a relatively uniform scale, allowing for more accurate comparison of video features across different videos.

[0091] In step 550, based on the video features of the first video and the video features corresponding to the multiple candidate videos, a second video matching the first video is retrieved from the multiple candidate videos.

[0092] Here, for each candidate video, the video features of the candidate video can also be determined by referring to steps 510 to 540 above. Based on the video features of the first video and the video features corresponding to the multiple candidate videos, a second video matching the first video is retrieved from the multiple candidate videos. For example, the similarity (such as cosine similarity) between the video features of the first video and the video features of each candidate video can be determined individually, and the candidate videos corresponding to the K largest similarities are determined as the second videos matching the first video, where K is an integer greater than 0 and can be arbitrarily set. It is worth noting that similarity is inversely proportional to feature distance (such as Euclidean distance, Manhattan distance, etc.). The above process can also be described as: determining the feature distance between the video features of the first video and the video features of each candidate video individually, and determining the candidate videos corresponding to the K smallest feature distances as the second videos matching the first video.

[0093] In steps 510 to 550 above, multiple representative key video frames are extracted from the first video. Image feature fusion processing is performed on the image features corresponding to the multiple key video frames to obtain the video features of the first video, which then represent the first video. The process of extracting key video frames includes two stages: coarse screening and fine screening. In the coarse screening stage, video frames are extracted from the first video according to their temporal order to obtain multiple candidate key video frames, ensuring that the distribution of these candidate key video frames is uniform over time, meaning that each candidate key video frame can represent a video segment of the first video within a specific time period. In the fine screening stage, the multiple candidate key video frames are clustered into multiple clusters. Since different candidate key video frames with similar image content will be clustered into one cluster, target candidate key video frames are extracted from each of these clusters to serve as key video frames. This ensures that the number of key video frames is not excessive, reducing the waste of computing resources and further improving the representativeness of the key video frames. After obtaining the video features of the first video, this embodiment of the application retrieves a second video that matches the first video from among the multiple candidate videos based on the video features of the first video and the video features corresponding to the multiple candidate videos. Since the video features can summarize the video more completely and comprehensively, the accuracy of video retrieval can be improved, that is, the probability of the second video actually matching the first video can be increased.

[0094] In some embodiments, refer to Figure 8 , Figure 8 This is a schematic diagram of an optional flowchart of the video retrieval method provided in the embodiments of this application, such as... Figure 8 As shown, Figure 5 Step 510 shown includes: Step 810: Perform sliding processing in the first video according to the video frame sequence using a sliding window; Step 820: After each slide of the sliding window, extract candidate key video frames from the sliding window based on the degree of difference between different video frames in the sliding window.

[0095] Steps 810 to 820 are described in detail below.

[0096] In step 810, a sliding window is used to slide through the first video according to the video frame sequence.

[0097] Here, a sliding window is used to slide through the first video frame in sequence. The size and step size of the sliding window can be set according to the actual application scenario.

[0098] In step 820, after each sliding of the sliding window, candidate key video frames are extracted from the sliding window based on the degree of difference between different video frames in the sliding window.

[0099] Here, after each slide of the sliding window, the degree of difference between different video frames within the sliding window is determined. This degree of difference refers to the difference between any two video frames (not limited to adjacent video frames). The method for determining the degree of difference between different video frames within the sliding window is not limited; it can be calculated based on the pixel values, statistical features, or image features of the video frames.

[0100] Then, based on the degree of difference between different video frames in the sliding window, candidate key video frames are extracted from the sliding window. For example, the largest number of differences are determined based on the degree of difference between different video frames in the sliding window, and the video frames corresponding to the largest number of differences are determined as candidate key video frames. In this way, the extracted candidate key video frames can reflect the changes in key content in the sliding window and have a certain degree of representativeness.

[0101] For ease of understanding, we will use an example where the sliding window size is 4 video frames and the step size is 4 video frames. See [link / reference] Figure 9 After the sliding window slides for the first time, it covers video frames 1 to 4. Therefore, candidate key video frames are extracted from video frames 1 to 4 based on the degree of difference between different video frames. After the sliding window slides for the second time, it covers video frames 5 to 8. Therefore, candidate key video frames are extracted from video frames 5 to 8 based on the degree of difference between different video frames.

[0102] In some embodiments, before extracting candidate key video frames from the sliding window based on the degree of difference between different video frames in the sliding window, the video retrieval method further includes: performing pixel value fusion processing on the pixel values ​​corresponding to multiple pixels of the video frames in the sliding window to obtain fused pixel values; and performing absolute difference calculation on the fused pixel values ​​corresponding to different video frames in the sliding window to obtain the degree of difference between different video frames in the sliding window.

[0103] Here is an example of how to determine the differences between different video frames. First, for each video frame in the sliding window, pixel value fusion processing is performed on the pixel values ​​corresponding to multiple pixels in the video frame to obtain a fused pixel value. Here, pixel value refers to the information contained in a pixel, used to represent a certain attribute of the pixel, such as color or grayscale information; pixel value fusion processing, such as averaging, results in a fused pixel value that reflects the overall situation of pixel values ​​in the video frame. Of course, pixel value fusion processing can also take the maximum value or other methods; there are no limitations on this.

[0104] Then, the absolute difference is calculated on the fused pixel values ​​corresponding to different video frames in the sliding window to obtain the degree of difference between the different video frames in the sliding window. The absolute difference calculation calculates the absolute value of the difference, which avoids the influence of positive and negative signs and ensures that the degree of difference accurately reflects the changes in the fused pixel values. For example, the degree of difference between video frame 1 and video frame 2 = |fused pixel value of video frame 1 - fused pixel value of video frame 2|.

[0105] By using the above method, the numerous pixel values ​​of each video frame are fused into a single fused pixel value, and the absolute difference between the fused pixel values ​​corresponding to different video frames is determined as the degree of difference. This can significantly reduce the amount of computation and quickly filter out candidate key video frames from a large number of video frames, which can well meet the efficiency-first screening requirements in the coarse screening stage.

[0106] In some embodiments, extracting candidate key video frames from a sliding window based on the degree of difference between different video frames in the sliding window includes: determining the maximum degree of difference based on the degree of difference between different video frames in the sliding window; and extracting the video frame corresponding to the maximum degree of difference from the sliding window as a candidate key video frame.

[0107] by Figure 9For example, after the sliding window slides for the first time, covering video frames 1 to 4, it's necessary to determine the differences between video frames 1 and 2, 1 and 3, 1 and 4, 2 and 3, 2 and 4, and 3 and 4. Then, the largest difference is determined among these differences; for example, if the largest difference is between video frames 1 and 4, then both video frames 1 and 4 are identified as candidate key video frames. This method captures the most critical and significant content changes within the sliding window, improving the representativeness of the extracted candidate key video frames.

[0108] In steps 810 to 820 above, the sliding window is used to slide through the first video in the order of video frames, which can scan all video frames of the first video segment by segment. Since the video frames are constantly changing in time sequence, this sliding process based on the video frame time sequence can ensure that no key information in any time period is missed. After each slide of the sliding window, candidate key video frames are extracted from the sliding window according to the degree of difference between different video frames in the sliding window. The degree of difference reflects the degree of change of video content between different video frames. Video frames with greater degree of difference often contain richer information or mark more important content changes. Therefore, extracting candidate key video frames from the sliding window according to the degree of difference can ensure that the candidate key video frames are representative of the multiple video frames covered by the sliding window.

[0109] In some embodiments, refer to Figure 10 , Figure 10 This is a schematic diagram of an optional flowchart of the video retrieval method provided in the embodiments of this application, such as... Figure 10 As shown, Figure 5 Step 520 shown includes: Step 1010: Perform feature statistical processing on multiple candidate key video frames respectively to obtain the statistical features corresponding to each candidate key video frame; Step 1020: Determine the feature distance between the statistical features corresponding to different candidate key video frames; Step 1030: Cluster multiple candidate key video frames based on feature distance to obtain multiple clusters; Step 1040: Extract target candidate key video frames from multiple clusters to serve as key video frames.

[0110] Steps 1010 to 1030 are described in detail below.

[0111] In step 1010, feature statistical processing is performed on multiple candidate key video frames to obtain the statistical features corresponding to each candidate key video frame.

[0112] Here, for each extracted candidate key video frame, feature statistical processing is performed to obtain statistical features. Feature statistical processing refers to statistical methods that do not rely on machine learning models. For example, color histogram statistics are performed on candidate key video frames to obtain color histogram features, which can reflect the color distribution in the candidate key video frames; another example is texture statistics are performed on candidate key video frames to obtain texture features, which can reflect the texture in the candidate key video frames.

[0113] It's worth noting that statistical features are used instead of coded features (i.e., coded features obtained through image encoding) in the fine screening stage because the number of candidate key video frames is usually large. Coded features require model processing, which is labor-intensive. Statistical features, on the other hand, are easier and faster to obtain. Therefore, using statistical features in the fine screening stage improves efficiency. Of course, if workload and efficiency are not a concern, coded features can also be used in the fine screening stage.

[0114] In some embodiments, feature statistical processing is performed on multiple candidate key video frames to obtain statistical features corresponding to each candidate key video frame. This includes: traversing multiple candidate key video frames and performing the following processing on the traversed candidate key video frames: constructing a pixel value vector based on the pixel values ​​corresponding to multiple pixels in the traversed candidate key video frames; and determining the pixel value vector as the statistical features of the traversed candidate key video frames.

[0115] For each candidate key video frame, a pixel value vector can be constructed based on the pixel values ​​corresponding to multiple pixels in the candidate key video frame. This involves converting an m×n pixel value matrix into an m×n dimensional pixel value vector. For example, the pixel values ​​corresponding to multiple pixels in a candidate key video frame can be represented as follows:

[0116] in, Let represent the pixel value of the pixel at coordinate (1, 1) in the candidate key video frame, and so on. The constructed pixel value vector can then be... That is, converting a 2×2 pixel value matrix into a 4-dimensional pixel value vector.

[0117] Then, the constructed pixel value vector is determined as the statistical feature of the candidate key video frames. This statistical feature reflects the pixel value distribution (color distribution / grayscale distribution) of the candidate key video frames. In this way, the scattered pixel values ​​in the candidate key video frames can be integrated into a single vector, which facilitates subsequent processing and analysis, such as making it easier to calculate distances for clustering.

[0118] In some embodiments, after performing feature statistical processing on multiple candidate key video frames to obtain statistical features corresponding to each candidate key video frame, the video retrieval method further includes: performing regularization processing on the statistical features corresponding to each candidate key video frame.

[0119] Here, the statistical features corresponding to multiple candidate key video frames can be regularized (e.g., L2 norm-based regularization) to enable a fair comparison of the statistical features of different candidate key video frames on a uniform scale, thereby improving the accuracy of the subsequently determined feature distances.

[0120] In step 1020, the feature distance between the statistical features corresponding to different candidate key video frames is determined.

[0121] Here, the feature distances between the statistical features corresponding to different candidate key video frames are determined, such as Euclidean distance or Manhattan distance.

[0122] In step 1030, multiple candidate key video frames are clustered based on feature distance to obtain multiple clusters.

[0123] Understandably, statistical features are used as representations of candidate key video frames. The feature distance between the statistical features corresponding to two candidate key video frames is actually the distance between those two candidate key video frames. Therefore, given the distances between different candidate key video frames, multiple candidate key video frames can be clustered to obtain multiple clusters.

[0124] In steps 1010 to 1040 above, considering that the number of candidate key video frames is usually large, statistical features of the candidate key video frames are extracted, so that the candidate key video frames can be quickly mapped to Euclidean space, thereby improving the processing efficiency of the fine screening stage; the feature distance between different statistical features is determined in Euclidean space, and the feature distance between different statistical features is used as the distance between the candidate key video frames represented by different statistical features, thereby improving the accuracy of clustering processing.

[0125] In some embodiments, refer to Figure 11 , Figure 11 This is a schematic diagram of an optional flowchart of the video retrieval method provided in the embodiments of this application, such as... Figure 11 As shown, Figure 5 Step 520 shown includes: Step 1110: Cluster the multiple candidate key video frames to obtain multiple clusters; Step 1120: Perform smoothing detection processing on multiple candidate key video frames in the traversed clusters respectively to obtain the smoothing parameters corresponding to the multiple candidate key video frames in the traversed clusters. The smoothing parameters are negatively correlated with the smoothing degree. Step 1130: Extract the target candidate key video frame corresponding to the maximum smoothness parameter from the traversed clusters, and use it as the key video frame.

[0126] Steps 1120 to 1130 are described in detail below.

[0127] In step 1120, smoothing detection processing is performed on multiple candidate key video frames in the traversed cluster to obtain the smoothing parameters corresponding to the multiple candidate key video frames in the traversed cluster. The smoothing parameters are negatively correlated with the smoothing degree.

[0128] After clustering multiple candidate key video frames to obtain multiple clusters, a random candidate key video frame from each cluster can be used as the target candidate key video frame. Alternatively, the target candidate key video frame can be extracted according to other specific strategies. An example strategy will be introduced next.

[0129] First, for each cluster obtained through clustering, smoothing detection is performed on multiple candidate key video frames within the cluster to obtain smoothing parameters corresponding to each candidate key video frame. There is a negative correlation between the smoothing parameter and the smoothing degree; that is, the larger the smoothing parameter of a candidate key video frame, the lower the smoothing degree of that candidate key video frame, and the richer the details and variations within that candidate key video frame. The type of smoothing parameter is not limited; it can be, for example, the Laplacian smoothing parameter, the mean gradient magnitude, or the standard deviation of the gradient direction.

[0130] In some embodiments, smoothing detection processing is performed on multiple candidate key video frames in the traversed cluster to obtain smoothing parameters corresponding to each candidate key video frame in the traversed cluster. This includes: traversing multiple candidate key video frames in the traversed cluster, and performing the following processing on the traversed candidate key video frames: performing pixel value difference processing on multiple pixels in the traversed candidate key video frames to obtain difference pixel values ​​corresponding to each pixel in the traversed candidate key video frames; and performing difference pixel value fusion processing on the difference pixel values ​​corresponding to each pixel in the traversed candidate key video frames to obtain smoothing parameters for the traversed candidate key video frames.

[0131] This section provides an example method for determining the smoothing parameters of candidate key video frames. For each candidate key video frame in the cluster, pixel value differencing is performed on multiple pixels within the candidate key video frame to obtain the difference pixel values ​​corresponding to each pixel. Pixel value differencing refers to performing a difference operation between the pixel value of a given pixel and the pixel values ​​of its neighboring pixels. A neighboring pixel can include one or more pixels. If there are multiple neighboring pixels, the difference pixel values ​​between the pixel value and the pixel values ​​of its multiple neighboring pixels can be merged (e.g., by summing or averaging) into a single difference pixel value. It is worth noting that for pixels located at the boundaries of candidate key video frames, neighboring pixels may not exist. In this case, the pixel values ​​of the neighboring pixels can be set to 0 or other preset values ​​to avoid calculation errors caused by boundary issues.

[0132] For ease of understanding, let's take a pixel (x, y) in a candidate key video frame with x as the horizontal axis and y as the vertical axis as an example. For instance, if there are four neighboring pixels: the pixels directly above, to the left, below, and to the right of pixel (x, y), then the difference pixel value of pixel (x, y) = f(x, y-1) + f(x-1, y) + f(x, y+1) + f(x+1, y) - 4f(x, y), where f(x, y-1) represents the pixel value of pixel (x, y-1). Of course, the above is merely an example of difference operations and does not constitute a limitation on the embodiments of this application. For example, if there are other selection rules for neighboring pixels, the direction of the difference operation can be reversed, such as the difference pixel value of pixel (x, y) = 4f(x, y) - f(x, y-1) - f(x-1, y) - f(x, y+1) - f(x+1, y).

[0133] For example, if there are eight neighboring pixels, namely the pixels directly above, to the upper left, to the left, to the lower left, to the lower right, to the right, and to the upper right of pixel (x, y), then the difference pixel value of pixel (x, y) = f(x, y-1) + f(x-1, y-1) + f(x-1, y) + f(x-1, y+1) + f(x, y+1) + f(x+1, y+1) + f(x+1, y) + f(x+1, y-1) - 8f(x, y). This case considers the eight neighboring pixels around pixel (x, y), and the measurement of pixel value changes is more comprehensive, but the amount of computation will also increase.

[0134] The difference pixel value of a pixel reflects the local smoothness of that pixel relative to its neighboring pixels; that is, the larger the difference pixel value, the lower the local smoothness (the higher the degree of pixel value variation). After determining the difference pixel value of each pixel in the candidate key video frame, the difference pixel values ​​corresponding to multiple pixels in the candidate key video frame are fused (e.g., summation, averaging, taking the absolute value and then summing), to obtain the smoothing parameter of the candidate key video frame. In this way, the smoothing parameter can reflect the overall smoothness of the candidate key video frame; that is, the larger the smoothing parameter, the lower the overall smoothness.

[0135] The above method can accurately capture the local smoothness around each pixel by performing pixel value difference processing on multiple pixels in the candidate key video frame. By performing difference pixel value fusion processing on the difference pixel values ​​corresponding to multiple pixels in the candidate key video frame, the smoothing parameter of the candidate key video frame is obtained. This makes the smoothing parameter accurately reflect the overall smoothness of the candidate key video frame, so that the target candidate key video frame can be extracted based on the smoothing parameter in the future.

[0136] In step 1130, the target candidate key video frame corresponding to the maximum smoothing parameter is extracted from the traversed clusters and used as the key video frame.

[0137] After obtaining the smoothing parameters corresponding to multiple candidate key video frames within the cluster, the target candidate key video frame corresponding to the maximum smoothing parameter is extracted as the key video frame. Compared with other candidate key video frames within the cluster, the key video frame has richer details and variations, thus representing the entire cluster. Subsequently, image features with rich semantics can be extracted from the key video frame, which helps to improve the accuracy of video retrieval.

[0138] In steps 1110 to 1130 above, smoothing detection processing is performed on multiple candidate key video frames in the cluster to obtain the smoothing parameters corresponding to each candidate key video frame. Then, the target candidate key video frame corresponding to the maximum smoothing parameter is extracted from the cluster as the key video frame. Since the key video frame contains more valuable information than other candidate key video frames in the cluster, it can more comprehensively represent the entire cluster and improve the accuracy of the fine screening stage.

[0139] Of course, the above is only one example of how to extract target candidate key video frames, and does not constitute a limitation on the embodiments of this application. Next, other example methods will be described.

[0140] In some embodiments, extracting target candidate key video frames from multiple clusters as key video frames includes: traversing multiple clusters and performing the following processing on the traversed clusters: performing subtitle recognition processing on multiple candidate key video frames in the traversed clusters; and extracting target candidate key video frames including subtitles from the traversed clusters as key video frames.

[0141] Here, for each cluster obtained through clustering, subtitle recognition processing is performed on multiple candidate key video frames within the cluster. The result of the subtitle recognition processing is either "including subtitles" or "excluding subtitles." The method of subtitle recognition processing is not limited. For example, Optical Character Recognition (OCR) can be performed on the candidate key video frames; if characters are recognized, the candidate key video frame is determined to include subtitles. Alternatively, threshold segmentation can be performed on the candidate key video frames; if the grayscale value of a pixel is higher than a grayscale threshold (or lower, depending on the grayscale relationship between the subtitle and the background), that pixel is marked as a candidate subtitle pixel. Then, if multiple candidate subtitle pixels form a region conforming to a preset shape, the candidate key video frame is determined to include subtitles. The preset shape is a general shape of subtitles, such as a rectangle. Another example is that a subtitle recognition model can be used to perform subtitle recognition processing on the candidate key video frames to obtain the result of whether the candidate key video frame includes subtitles.

[0142] Then, target candidate key video frames including subtitles are extracted from the cluster as key video frames. Subtitles in the first video often contain key semantic information, such as dialogue content and narration. By extracting target candidate key video frames including subtitles as key video frames, this key information can be accurately extracted. In addition, subtitles in key video frames help to better understand the semantics of key video frames and improve the accuracy of subsequently extracted image features.

[0143] In some embodiments, extracting target candidate key video frames from multiple clusters as key video frames includes: traversing multiple clusters and performing the following processing on the traversed clusters: performing target recognition processing on multiple candidate key video frames in the traversed clusters to obtain the number of targets included in each of the multiple candidate key video frames in the traversed clusters; and extracting the target candidate key video frame that includes the maximum number of targets from the traversed clusters as a key video frame.

[0144] Here, for each cluster obtained through clustering, target recognition processing is performed on multiple candidate key video frames within the cluster to obtain the number of targets included in each candidate key video frame. The method of target recognition processing is not limited; for example, it can be implemented using a target recognition model. Similarly, the type of target to be recognized is not limited; for example, it can be a face.

[0145] Then, the candidate key video frames containing the largest number of targets are extracted from the cluster as key video frames. This ensures that the extracted key video frames contain richer key information and facilitates a better understanding of their semantics, thus improving the accuracy of subsequently extracted image features. For example, in sports event videos, targets include people, balls, etc., so the candidate key video frames containing the largest number of targets are more likely to correspond to key scenes in the sports event, such as intense player confrontations or goal moments, fully showcasing the core content of the sports event. Similarly, in movies, targets include faces, so the candidate key video frames containing the largest number of targets are more likely to correspond to key scenes in the movie, such as core plot conflicts.

[0146] In some embodiments, refer to Figure 12 , Figure 12 This is a schematic diagram of an optional flowchart of the video retrieval method provided in the embodiments of this application, such as... Figure 12 As shown, Figure 5 Following step 540, the following is also included: Step 1210: Perform feature cross processing on the video features of the first video and the features of the first video in other modalities to obtain the multimodal features of the first video; like Figure 12 As shown, Figure 5 Step 550 shown can be updated to step 1220: Step 1220: Based on the multimodal features of the first video and the multimodal features corresponding to multiple candidate videos, retrieve the second video that matches the first video from the multiple candidate videos.

[0147] Steps 1210 to 1220 are described in detail below.

[0148] In step 1210, the video features of the first video are subjected to feature cross-processing with the features of the first video in other modalities to obtain the multimodal features of the first video.

[0149] Here, in addition to determining the video features of the first video in the image modality, features of the first video in other modalities can also be determined. The video features of the first video in the image modality are then combined with the features of the first video in other modalities to obtain the multimodal features of the first video. Feature cross-processing is used to combine different features to generate new features, thereby uncovering potential relationships in the data. This application does not limit the method of feature cross-processing; for example, it can be multiplicative cross-processing, additive cross-processing, division cross-processing, etc.

[0150] In some embodiments, performing feature cross processing on the video features of the first video and the features of the first video in other modalities to obtain multimodal features of the first video includes: performing Hadamard feature cross processing on the video features of the first video and the features of the first video in other modalities to obtain multimodal features of the first video.

[0151] Hadamard feature cross processing refers to multiplying two vectors of the same dimension by their corresponding elements in the same dimension to obtain a new vector. For example, the video features of the first video are... The first video has the following characteristics in other modalities: The multimodal features obtained by performing Hadamard feature cross processing are: Hadamard feature cross processing emphasizes the interaction between different features by performing inner product-like crosses (i.e., multiplicative crosses), which is applicable to situations where features of different modalities influence each other in the embodiments of this application; at the same time, it is beneficial for capturing nonlinear relationships between features.

[0152] In some embodiments, the features of the first video in other modalities include the content description features of the first video; before performing feature cross processing on the video features of the first video and the features of the first video in other modalities, the video retrieval method further includes: obtaining the content description text of the first video; performing text feature extraction processing on the content description text to obtain the content description features of the first video.

[0153] Here, in addition to determining the video features of the first video in the image modality, we can also determine the features of the first video in the text modality, such as content description features (caption features). Next, we will describe an example of how to determine content description features.

[0154] First, obtain the content description text of the first video. This text describes the video content, for example, the content description text for a pet video might be: "A cute puppy is playing on the grass, chasing a colorful frisbee, accidentally tripping over some grass, but quickly getting up and continuing to run happily, looking very playful." The content description text can be manually annotated or obtained through a machine learning model performing video annotation tasks. The latter enables automatic annotation, saving manpower. The type of machine learning model used to perform the video annotation task is not limited, such as a Contrastive Language-Image Pretraining (CLIP) model or a large language model.

[0155] Then, text feature extraction is performed on the content description text, and the extracted text features are used as the content description features of the first video. This text feature extraction can be achieved using machine learning models or other methods (such as the bag-of-words model), and there are no restrictions on which method is used.

[0156] The above method obtains the content description text of the first video and extracts text features as content description features, thereby supplementing the text modality features. In this way, video features and content description features can complement each other. After feature cross-processing, the feature expressive ability can be improved. The resulting multimodal features can more comprehensively and accurately express the video content, which helps to improve the accuracy of video retrieval.

[0157] In some embodiments, image feature extraction processing is performed on multiple extracted key video frames to obtain image features corresponding to each key video frame, including: performing image encoding processing on the multiple extracted key video frames using an image encoder to obtain image features corresponding to each key video frame; and performing text feature extraction processing on the content description text to obtain the content description features of the first video, including: performing text encoding processing on the content description text using a text encoder to obtain the content description features of the first video; wherein the image encoder and the text encoder are obtained through joint training.

[0158] Here, the image encoder and text encoder are pre-trained jointly. Joint training refers to simultaneously training multiple related model components or models of different modalities to improve overall performance. Its core lies in utilizing the correlation and complementarity between different components or modalities, enabling them to learn from and promote each other. In this embodiment, by jointly training the image encoder and text encoder, the semantic relationships between images and text are learned, thereby improving the model training effect. It is worth noting that the image encoder is a model component used to encode images into image features (or can be understood as a separate model; the text encoder is similar). The text encoder is a model component used to encode text into text features.

[0159] After completing the joint training of the image encoder and the text encoder, as follows: Figure 13 As shown, on the one hand, an image encoder performs image encoding processing on multiple extracted key video frames (i.e., an example implementation of image feature extraction processing) to obtain image features corresponding to each key video frame; on the other hand, a text encoder performs text encoding processing on the content description text of the first video (i.e., an example implementation of text feature extraction processing) to obtain the content description features of the first video. Then, image feature fusion processing is performed on the image features corresponding to the multiple key video frames to obtain the video features of the first video. The video features of the first video are then combined with the content description features through feature cross-processing to obtain the multimodal features of the first video.

[0160] The above approach, by jointly training the image encoder and text encoder, enables them to learn the deep semantic relationships between images and text. This ensures that during the model inference stage, the image features extracted by the image encoder and the text features (content description features) extracted by the text encoder are semantically closely related, providing a more accurate information foundation for video retrieval tasks. Furthermore, compared to training the image encoder and text encoder independently, the joint training method can achieve collaborative optimization, thereby improving training efficiency and reducing training time and computational resource consumption.

[0161] In some embodiments, before performing image encoding processing on the extracted key video frames respectively by the image encoder, the video retrieval method further includes: obtaining image-text pairs and sample labels, wherein the image-text pairs include sample images and sample text, and the sample labels are used to indicate whether there is a correspondence between the sample images and sample text; performing image encoding processing on the sample images by the image encoder to obtain image features of the sample images; performing text encoding processing on the sample text by the text encoder to obtain text features of the sample text; determining the similarity between the image features of the sample images and the text features of the sample text, and determining a loss value based on the similarity and the sample labels; and training the image encoder and the text encoder based on the loss value.

[0162] Here, contrastive learning can be used to jointly train the image encoder and the text encoder. Contrastive learning is an unsupervised learning method that aims to learn effective representations by comparing the similarities and differences between images and text. Its core idea is to bring similar images and text closer together, while pushing dissimilar images and text further apart, so that the model can learn meaningful feature representations.

[0163] To facilitate understanding, the joint training process is illustrated using an image-text pair as an example. An image-text pair includes a sample image and sample text. Sample labels indicate whether there is a correspondence between the sample image and sample text; for example, a sample label of 0 indicates no correspondence, while a sample label of 1 indicates a correspondence (meaning the sample text is a description of the sample image). Based on this, an image encoder performs image encoding on the sample image to obtain its image features, and a text encoder performs text encoding on the sample text to obtain its text features. Then, the similarity (e.g., cosine similarity) between the image features of the sample image and the text features of the sample text is determined, and a loss value is determined based on the similarity and sample labels. Specifically, when the sample label is 0, the loss value is positively correlated with the similarity; when the sample label is 1, the loss value is negatively correlated with the similarity. This embodiment does not limit the specific type of the loss function used to calculate the loss value; for example, it can be the InfoNCE loss function. Then, the image encoder and text encoder are trained based on the loss value, i.e., the model parameters of the image encoder and the text encoder are updated. Of course, the above is just an example of joint training. In actual training, a large number of positive and negative samples can be prepared for training to improve the training effect. Positive samples refer to image-text pairs with a sample label of 1, and negative samples refer to image-text pairs with a sample label of 0.

[0164] The above method uses contrastive learning for joint training. During joint training, the model learns to bring image features and text features that describe the same content closer together in the feature space, while pushing image features and text features that describe different content further apart. This allows the model to focus on features that can effectively distinguish different content, thereby improving the accuracy of the extracted image features and content description features during the model inference stage.

[0165] In some embodiments, the features of the first video in other modalities include the duration feature of the first video; before performing feature cross processing on the video features of the first video and the features of the first video in other modalities, the video retrieval method further includes: determining the video duration of the first video; constructing duration description text based on the video duration; and performing text feature extraction processing on the duration description text to obtain the duration feature of the first video.

[0166] Here, in addition to determining the video features of the first video in the image modality, we can also determine the features of the first video in the text modality, such as duration features. Next, we will illustrate an example of how to determine duration features.

[0167] First, determine the video length of the first video and construct a duration description text based on the video length. For example, the video length can be substituted into a preset text template to obtain the duration description text. For instance, if the video length is 10 minutes and the text template is "video length is xxxx", then substituting the video length into the text template will yield the duration description text "video length is 10 minutes". Of course, the above text template is only an example and does not constitute a limitation on the embodiments of this application.

[0168] Then, text feature extraction is performed on the duration description text, and the resulting text features are used as the duration features of the first video. The text feature extraction process for the duration description text can refer to the text feature extraction process for the content description text described above; for example, it can also be implemented using a machine learning model or a bag-of-words model. In some embodiments, see... Figure 14 The image encoder can extract the image features of key video frames, and the text encoder can extract the duration features of the duration description text. The image encoder and the text encoder are obtained through joint training.

[0169] The above approach introduces duration features, which complement video features to provide a more comprehensive description of the video. This adds a time dimension to video retrieval, further improving its accuracy. For example, if the second retrieved video is similar in duration to the first, they are more likely to be a match. Furthermore, constructing duration description text based on video duration presents the video duration in a more natural language format. This allows for accurate capture of the semantics inherent in the video duration during text feature extraction, achieving accurate conversion from video duration to duration features.

[0170] In some embodiments, the features of the first video in other modalities include the duration feature of the first video; before performing feature cross processing on the video features of the first video and the features of the first video in other modalities, the video retrieval method further includes: determining the video duration of the first video; performing one-hot encoding on the video duration to obtain the duration feature of the first video.

[0171] Here, we provide another example of how to determine duration features: one-hot encoding of the first video's duration yields its duration feature. The principle of one-hot encoding is as follows: for a categorical variable with n distinct values, n binary features (each with a value of 0 or 1) are created. Each feature corresponds to a value of the categorical variable. When the categorical variable of a sample takes a specific value, the corresponding feature position is set to 1, and other feature positions are set to 0.

[0172] For example, three duration intervals can be pre-defined: (0, 10 minutes), [10 minutes, 30 minutes), and [30 minutes, 60 minutes). (0, 10 minutes) refers to 0-10 minutes (excluding 10 minutes), [10 minutes, 30 minutes) refers to 10-30 minutes (including 10 minutes, but excluding 30 minutes), and so on. Then, one-hot encoding is performed on these three duration intervals. For example, the duration feature corresponding to the duration interval (0, 10 minutes) is [1, 0, 0], the duration feature corresponding to the duration interval [10 minutes, 30 minutes) is [0, 1, 0], and the duration feature corresponding to the duration interval [30 minutes, 60 minutes) is [0, 0, 1]. For example, if the first video's duration is 12 minutes, then since 12 minutes falls within the duration interval [10 minutes, 30 minutes), the duration feature of the first video is determined to be [0, 1, 0]. Of course, the above duration intervals are just examples; the number and length of duration intervals can be specifically set according to the actual application scenario.

[0173] The above method, through one-hot encoding, can quickly convert the video duration of the first video into a duration feature, which helps to improve the efficiency of video retrieval; at the same time, one-hot encoding has a small computational load, which can save computing resources.

[0174] In some embodiments, the video features of the first video are subjected to feature cross-processing with the features of the first video in other modalities to obtain the multimodal features of the first video. This includes: performing feature cross-processing on the video features, content description features, and duration features of the first video to obtain the multimodal features of the first video. Here, both content description features and duration features can be considered simultaneously to further improve the comprehensiveness of the obtained multimodal features, which helps to improve the accuracy of video retrieval.

[0175] Of course, in addition to the features of the image modality and the text modality, features of other modalities can also be considered. For example, the audio in the first video can be extracted, and audio feature extraction processing can be performed on the audio to obtain the audio features of the first video (i.e., the features of the audio modality).

[0176] In step 1220, based on the multimodal features of the first video and the multimodal features corresponding to the multiple candidate videos, a second video matching the first video is retrieved from the multiple candidate videos.

[0177] For example, the similarity (such as cosine similarity) between the multimodal features of the first video and the multimodal features of each candidate video can be determined separately, and the candidate videos corresponding to the K largest similarities can be determined as the second videos that match the first video, where K is an integer greater than 0 and can be set arbitrarily.

[0178] In steps 1210 to 1220 above, the video features of the first video are cross-processed with the features of the first video in other modalities to obtain the multimodal features of the first video. In this way, the potential relationships between features of different modalities can be explored, and the first video can be described more comprehensively and accurately through multimodal features. Based on multimodal features, the similarity between different videos can be measured more accurately, thereby further improving the accuracy of video retrieval.

[0179] In some embodiments, refer to Figure 15 , Figure 15 This is a schematic diagram of an optional flowchart of the video retrieval method provided in the embodiments of this application, such as... Figure 15 As shown, Figure 5 The illustrated step 550 includes: Step 1510: Establish multiple vector indices based on the video features corresponding to the multiple candidate videos; Step 1520: Determine the first similarity between the video features of the first video and multiple vector indices, and filter the multiple vector indices according to the first similarity to obtain the filtered vector indices; Step 1530: Determine the second similarity between the video features of the first video and the video features of multiple third videos, and filter the multiple third videos according to the second similarity to obtain the second video that matches the first video. The video features of the multiple third videos are obtained by indexing the vector after filtering. The multiple third videos are some of the candidate videos.

[0180] Steps 1510 to 1530 are described in detail below.

[0181] In step 1510, multiple vector indices are established based on the video features corresponding to the multiple candidate videos.

[0182] In this embodiment, the similarity (such as cosine similarity) between the video features of the first video and the video features of each candidate video can be determined individually, and the candidate videos corresponding to the K largest similarities are determined as the second videos matching the first video. However, in scenarios with a large number of candidate videos, the computational cost of this method is very high. To address this, vector indexing can be used to reduce the computational cost and accelerate the video retrieval process.

[0183] First, multiple vector indices are established based on the video features corresponding to multiple candidate videos. For example, for the video features corresponding to multiple candidate videos, similar video features are grouped together and represented by a single vector index, while dissimilar video features are stored separately. Therefore, the number of vector indices is less than the number of video features. This application does not limit the vector indexing technology used; for example, Locality Sensitive Hashing (LSH) indexing, Inverted File Index (IVF), Multi-Stage Tree Graph (MSTG) indexing, and Hierarchical Navigable Small World Graph (HNSW) indexing are all acceptable. Taking the Inverted File Index as an example, the video features corresponding to multiple candidate videos can be clustered to obtain multiple clusters. For each cluster, the cluster center is the vector index, which is used to index the video features within that cluster.

[0184] like Figure 16 As shown, two vector indices are established based on the video features corresponding to the eight candidate videos. Vector index 1 is used to index the video features of candidate video 1, candidate video 2, candidate video 3, and candidate video 4, while vector index 2 is used to index the video features of candidate video 5, candidate video 6, candidate video 7, and candidate video 8.

[0185] In step 1520, the first similarity between the video features of the first video and the multiple vector indices is determined, and the multiple vector indices are filtered according to the first similarity to obtain the filtered vector indices.

[0186] Here, the first similarity between the video features of the first video and multiple vector indices is determined, and the vector indices corresponding to the J largest first similarities are determined as the filtered vector indices, where J is an integer greater than 0 and can be set arbitrarily, such as 1.

[0187] Compared to the video features of candidate videos, the number of vector indices is significantly reduced, thus ensuring that the computational load of step 1520 is not too large and that the filtered vector indices can be found quickly.

[0188] In step 1530, the second similarity between the video features of the first video and the video features of multiple third videos is determined, and the multiple third videos are filtered according to the second similarity to obtain the second video that matches the first video. The video features of the multiple third videos are obtained by indexing the vector after filtering, and the multiple third videos are some of the candidate videos.

[0189] Here, the video features of multiple third videos are obtained by indexing based on the filtered vector indices, and the second similarity between the video features of the first video and the video features of the multiple third videos is determined. For example... Figure 16 As shown, for example, if the vector index after filtering is vector index 1, then the third video refers to candidate videos 1, 2, 3, and 4. Then, multiple third videos are filtered based on the second similarity score; for example, the third video corresponding to the K largest second similarities is determined as the second video matching the first video.

[0190] In steps 1510 to 1530 above, similar video features are grouped together and represented by a vector index. First, the vector index is used for filtering, and then the third video is used for filtering. This strategy of gradually narrowing the scope avoids calculating the similarity of video features for all candidate videos, which significantly reduces the amount of computation and improves the efficiency of video retrieval. It is suitable for scenarios with a large number of candidate videos, such as scenarios where video retrieval needs to be performed on a large number of candidate videos managed by a video platform.

[0191] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0192] As an example, embodiments of this application provide, for instance, the following: Figure 17The diagram shown is an interface illustration of a video platform. This embodiment of the application can be applied to a video platform to provide a video retrieval method. For example, a video uploaded to the video platform can be used as the first video, and a second video matching the first video can be retrieved from multiple candidate videos managed by the video platform. Figure 17 This shows multiple second videos retrieved based on a first video uploaded by an object.

[0193] As an example, embodiments of this application provide, for instance, the following: Figure 18 The illustration shows an example of video retrieval. After receiving the first video, this embodiment of the application can accurately retrieve a second video (i.e., video that is completely identical to the first video in terms of screen display) from an existing database. Figure 18 The second video shown at the top also adaptively identified other second videos that closely resembled the city skyline, sunset, sky, and clouds in the first video. It's worth noting that... Figure 18 The image shows multiple retrieved second videos, arranged in descending order of similarity to the first video.

[0194] Next, the video retrieval method provided in the embodiments of this application will be explained from the perspective of underlying implementation. For example... Figure 19 As shown, the video retrieval method mainly includes three steps: key video frame extraction, video feature encoding, and database storage / retrieval. These will be explained separately below.

[0195] 1) Extraction of key video frames.

[0196] Current frame extraction methods mainly include FFmpeg-based methods and random sampling methods. However, for FFmpeg-based methods, changes to the extraction script parameters may result in the extraction of non-critical video frames or even the complete failure to extract critical video frames. In addition, the number of critical video frames extracted by FFmpeg may be extremely large, making it impossible to extract typical and representative critical video frames. For random sampling methods, the extracted critical video frames may be unrepresentative, or critical video frames in some time periods may be over-extracted while those in other time periods are ignored, resulting in uneven temporal distribution of critical video frames.

[0197] To address the aforementioned issues, the embodiments of this application can achieve adaptive key video frame extraction, ensuring that the extracted key video frames are representative. The specific process is as follows: a) Extracting candidate key video frames: A sliding window is used to process the video frames sequentially. After each slide, the difference between different video frames within the sliding window is calculated. The two video frames with the largest difference are extracted as candidate key video frames to simulate key changes in the video frame. This can be achieved by averaging the grayscale values ​​of multiple pixels in each video frame to obtain an average grayscale value (corresponding to the fused pixel value mentioned above). Then, the absolute difference between the average grayscale values ​​of different video frames is calculated to obtain the difference between them.

[0198] b) Clustering: The extracted candidate key video frames are clustered to obtain multiple clusters. In this way, the candidate key video frames in each cluster are similar in terms of screen content.

[0199] c) Extracting the optimal video frame within a cluster: For each cluster, calculate the smoothing parameters corresponding to multiple candidate key video frames within the cluster, and extract the candidate key video frame corresponding to the largest smoothing parameter as the key video frame (optimal video frame). The larger the smoothing parameter of the candidate key video frame, the lower the smoothness and the richer the details and variations in the image.

[0200] d) For a single candidate key video frame that cannot be clustered during the clustering process, it is directly used as the key video frame.

[0201] 2) Video feature encoding.

[0202] Considering the strong image encoding capability and robust generalization effect of the image encoder in the CLIP model, video feature encoding can be achieved through the image encoder in the CLIP model. This application provides the following three video feature encoding methods: a) mean: The image encoder in the CLIP model encodes multiple key video frames separately to obtain image features corresponding to each key video frame. These image features are then averaged across multiple dimensions to obtain the video features. This method focuses on the main distribution (overall features) of the image features corresponding to the multiple key video frames. In some embodiments, this averaging process can be implemented using the torch.mean function.

[0203] b) max: The image encoder in the CLIP model encodes multiple key video frames to obtain image features corresponding to each key video frame. Then, the maximum value of these image features across multiple dimensions is calculated to obtain the video features. This method focuses on the specificity and saliency of the image features corresponding to the multiple key video frames. In some embodiments, this maximum value calculation can be implemented using the torch.max function.

[0204] c) mean+max: The CLIP model uses an image encoder to encode multiple key video frames, obtaining image features for each frame. These features are then averaged across multiple dimensions to obtain the mean feature, and the maximum feature is obtained by maximizing these features across multiple dimensions. Finally, the mean and maximum features are summed to obtain the final video feature. This approach considers both the main distribution of image features across multiple key video frames and the specificity and saliency of those features.

[0205] In some embodiments, the video features can be combined with the video's features in other modalities using Hadamard feature cross-processing to obtain the video's multimodal features, which are then used in subsequent database insertion / retrieval steps. These other modal features include, but are not limited to, Caption features and Duration features. For Caption features, they can be extracted from the video's content description text using a text encoder (such as the text encoder in the CLIP model or the text encoder in a large language model). For Duration features, a duration description text can be constructed based on the video's length, such as "video length is 10 minutes," and then the Duration feature can be extracted from the duration description text using a text encoder (such as the text encoder in the CLIP model or the text encoder in a large language model); alternatively, the video's length can be one-hot encoded to obtain the Duration feature.

[0206] 3) Bookmark / Search.

[0207] a) Data Storage: If the videos involved in the above steps are candidate videos, then after performing L2-norm-based regularization on the video features corresponding to multiple candidate videos, a vector index is created and stored in the database. Here, "stored in the database" refers to storing the data in a vector database.

[0208] b) Retrieval: If the video involved in the above steps is the first video that triggers the retrieval, then the top K matching second videos are queried in the vector database based on the video features of the first video. For example, the first similarity between the video features of the first video and multiple vector indices in the vector database is determined, and the vector index corresponding to the largest first similarity is used as the filtered vector index. The second similarity between the video features of the first video and the video features of multiple third videos is determined, and the top K third videos corresponding to the second similarity are used as the second videos matching the first video. The video features of the multiple third videos are obtained by indexing the filtered vector index.

[0209] This application proposes a concise and efficient end-to-end video retrieval method. First, several representative key video frames are extracted from the video. Then, the image encoder in the CLIP model extracts the image features corresponding to each key video frame. Finally, these image features are fused into a single video feature, enabling the video feature to comprehensively and completely summarize the video, thereby improving the accuracy of video retrieval. Furthermore, Hadamard feature cross-referencing can be used to introduce features from other modalities (such as caption and duration features) into the original video features, further enhancing the accuracy of video retrieval.

[0210] The inventors conducted experimental verification based on the embodiments of this application. The experimental verification process used video features, and the obtained video retrieval indicators are shown in Table 1.

[0211] Table 1 Video Retrieval Metrics

[0212] In Table 1, #Frames represents the number of key video frames extracted from the video (corresponding to the N key video frames mentioned above); Top K represents the top K retrieved second videos; Recall@K is the recall rate, representing the proportion of videos that actually match the first video that were retrieved, calculated as Recall@K = number of videos that actually match the first video among the top K second videos / number of videos that actually match the first video in the database; Precision@K is the precision rate, representing the proportion of videos that actually match the first video among the top K second videos, calculated as Precision@K = number of videos that actually match the first video among the top K second videos / K; F1-score is the harmonic mean of recall and precision. Furthermore, the internal CLIP model in Table 1 refers to the CLIP model trained by the inventors themselves, while the HuggingFace CLIP model refers to the open-source CLIP model provided by HuggingFace.

[0213] As can be seen from Table 1, the embodiments of this application can achieve excellent video retrieval performance, and the video retrieval performance can be further improved if a better model is adopted.

[0214] The inventors conducted further experimental verification based on the internal CLIP model. During the experimental verification process, Caption features were introduced, that is, video features and Caption features were fused into multimodal features to achieve video retrieval. The resulting video retrieval metrics are shown in Table 2.

[0215] Table 2 Video Retrieval Metrics

[0216] As shown in Table 2, the video retrieval performance was further improved after the introduction of the Caption feature.

[0217] To facilitate better implementation of the video retrieval method provided in this application, this application also provides an apparatus based on the above-described video retrieval method. The meanings of the terms used are the same as in the video retrieval method described above, and specific implementation details can be found in the descriptions within the method embodiments.

[0218] Please see Figure 20 , Figure 20 This is a schematic diagram of the structure of a video retrieval device provided in an embodiment of this application. The video retrieval device is applied to a computer device and may include a first extraction module 2010, a second extraction module 2020, a feature extraction module 2030, a feature fusion module 2040, and a feature retrieval module 2050, wherein: The first extraction module 2010 is used to perform video frame extraction processing on the first video according to the video frame time sequence to obtain multiple candidate key video frames. The second extraction module 2020 is used to perform clustering processing on multiple candidate key video frames to obtain multiple clusters, and extract target candidate key video frames from multiple clusters as key video frames. The feature extraction module 2030 is used to perform image feature extraction processing on the extracted key video frames respectively to obtain the image features corresponding to the key video frames respectively. The feature fusion module 2040 is used to perform image feature fusion processing on the image features corresponding to multiple key video frames to obtain the video features of the first video. The feature retrieval module 2050 is used to retrieve a second video that matches the first video from multiple candidate videos based on the video features of the first video and the video features corresponding to multiple candidate videos.

[0219] In some embodiments, the first extraction module 2010 is further configured to: The sliding window is used to slide the video frame in sequence within the first video. After each slide of the sliding window, candidate key video frames are extracted from the sliding window based on the degree of difference between different video frames in the sliding window.

[0220] In some embodiments, the first extraction module 2010 is further configured to: Pixel values ​​are fused for the pixel values ​​corresponding to multiple pixels in the video frame within the sliding window to obtain fused pixel values. The degree of difference between different video frames in the sliding window is obtained by performing an absolute difference calculation on the fused pixel values ​​corresponding to different video frames in the sliding window.

[0221] In some embodiments, the first extraction module 2010 is further configured to: The maximum difference is determined based on the degree of difference between different video frames in the sliding window; The video frame with the largest difference is extracted from the sliding window and used as a candidate key video frame.

[0222] In some embodiments, the second extraction module 2020 is further configured to: Feature statistical processing was performed on multiple candidate key video frames to obtain the statistical features corresponding to each candidate key video frame. Determine the feature distances between the statistical features corresponding to different candidate key video frames; Multiple candidate key video frames are clustered based on feature distance to obtain multiple clusters.

[0223] In some embodiments, the second extraction module 2020 is further configured to: Multiple candidate key video frames are traversed and processed, and the following processing is performed on the traversed candidate key video frames: Construct a pixel value vector based on the pixel values ​​corresponding to multiple pixels in the candidate key video frames that have been traversed; The pixel value vector is determined as the statistical feature of the candidate key video frames traversed.

[0224] In some embodiments, the second extraction module 2020 is further configured to: The process iterates through multiple clusters and performs the following processing on each of the traversed clusters: Smoothing detection is performed on multiple candidate key video frames in the traversed clusters to obtain the smoothing parameters corresponding to the multiple candidate key video frames in the traversed clusters. The smoothing parameters are negatively correlated with the smoothing degree. Extract the target candidate key video frame corresponding to the maximum smoothness parameter from the traversed clusters, and use it as the key video frame.

[0225] In some embodiments, the second extraction module 2020 is further configured to: The process involves iterating through multiple candidate key video frames within the traversed clusters and performing the following processing on each candidate key video frame: Pixel value difference processing is performed on multiple pixels in the traversed candidate key video frames to obtain the difference pixel values ​​corresponding to each pixel in the traversed candidate key video frames. Differential pixel value fusion processing is performed on the differential pixel values ​​corresponding to multiple pixels in the traversed candidate key video frames to obtain the smoothing parameters of the traversed candidate key video frames.

[0226] In some embodiments, the second extraction module 2020 is further configured to: The process iterates through multiple clusters and performs the following processing on each of the traversed clusters: Subtitle recognition processing is performed on multiple candidate key video frames in the traversed clusters respectively; Extract target candidate key video frames, including subtitles, from the traversed clusters to serve as key video frames.

[0227] In some embodiments, the second extraction module 2020 is further configured to: The process iterates through multiple clusters and performs the following processing on each of the traversed clusters: Target recognition processing is performed on multiple candidate key video frames in the traversed clusters to obtain the number of targets included in each candidate key video frame in the traversed clusters. Extract the target candidate key video frame containing the maximum number of targets from the traversed clusters, and use it as the key video frame.

[0228] In some embodiments, the feature retrieval module 2050 is further configured to: The video features of the first video are combined with the features of the first video in other modalities to obtain the multimodal features of the first video. Based on the multimodal features of the first video and the multimodal features corresponding to multiple candidate videos, a second video matching the first video is retrieved from the multiple candidate videos.

[0229] In some embodiments, the features of the first video in other modalities include content description features of the first video; the feature extraction module 2030 is further configured to: Get the content description text of the first video; Text feature extraction is performed on the content description text to obtain the content description features of the first video.

[0230] In some embodiments, the feature extraction module 2030 is further configured to: Image feature extraction is performed on multiple key video frames extracted by an image encoder to obtain the image features corresponding to each key video frame. The text features of the first video are obtained by extracting text features from the content description text using a text encoder. The image encoder and text encoder are obtained through joint training.

[0231] In some embodiments, the video retrieval device further includes a contrast learning module (not shown), for: Obtain image-text pairs and sample labels. Image-text pairs include sample images and sample text. Sample labels are used to indicate whether there is a correspondence between sample images and sample text. The image features of the sample image are obtained by performing image feature extraction processing on the sample image through an image encoder; The text features of the sample text are obtained by extracting text features from the sample text using a text encoder. Determine the similarity between the image features of the sample image and the text features of the sample text, and determine the loss value based on the similarity and the sample label; The image encoder and text encoder are trained based on the loss values.

[0232] In some embodiments, the features of the first video in other modalities include the duration feature of the first video; the feature extraction module 2030 is further configured to: Determine the duration of the first video; Construct duration description text based on video duration; Text feature extraction is performed on the duration description text to obtain the duration features of the first video.

[0233] In some embodiments, the features of the first video in other modalities include the duration feature of the first video; the feature extraction module 2030 is further configured to: Determine the duration of the first video; One-hot encoding is performed on the video duration to obtain the duration feature of the first video.

[0234] In some embodiments, the feature fusion module 2040 is further configured to: Perform any of the following processes: The image features corresponding to multiple key video frames are averaged across multiple dimensions to obtain the video features of the first video. The maximum value of the image features corresponding to multiple key video frames is taken in multiple dimensions to obtain the video features of the first video. The average feature is obtained by averaging the image features corresponding to multiple key video frames in multiple dimensions, and the maximum feature is obtained by taking the maximum value of the image features corresponding to multiple key video frames in multiple dimensions. The video features of the first video are obtained by summing the average feature and the maximum feature.

[0235] In some embodiments, the feature retrieval module 2050 is further configured to: Multiple vector indices are created based on the video features corresponding to multiple candidate videos; Determine the first similarity between the video features of the first video and multiple vector indices, and filter the multiple vector indices based on the first similarity to obtain the filtered vector indices; The second similarity between the video features of the first video and the video features of multiple third videos is determined, and the multiple third videos are filtered according to the second similarity to obtain the second video that matches the first video. The video features of the multiple third videos are obtained by indexing the vector after filtering. The multiple third videos are a subset of the multiple candidate videos.

[0236] In some embodiments, the number of key video frames is M; the video retrieval device further includes a sampling module (not shown) for: Based on the video frame timing sequence, M key video frames are sampled to obtain N key video frames, where N is less than M, and both N and M are integers greater than 0. The feature extraction module 2030 is also used for: Image feature extraction is performed on N key video frames to obtain the image features corresponding to each of the N key video frames.

[0237] For details on the implementation of each of the above modules, please refer to the previous examples, which will not be repeated here.

[0238] Reference Figure 21 , Figure 21 To implement the structural block diagram of a portion of the terminal 140 in this application embodiment, the terminal 140 includes: a radio frequency (RF) circuit 2110, a memory 2115, an input unit 2130, a display unit 2140, a sensor 2150, an audio circuit 2160, a wireless fidelity (WiFi) module 2170, a processor 2180, and a power supply 2190, among other components. Those skilled in the art will understand that... Figure 21 The terminal 140 structure shown does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0239] The RF circuit 2110 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 2180; in addition, it transmits uplink data to the base station.

[0240] The memory 2115 can be used to store software programs and modules. The processor 2180 executes various terminal functions and video retrieval processing by running the software programs and modules stored in the memory 2115.

[0241] The input unit 2130 can be used to receive input numeric or character information, and to generate key signal inputs related to the settings and function control of the terminal. Specifically, the input unit 2130 may include a touch panel 2131 and other input devices 2132.

[0242] Display unit 2140 can be used to display input or provided information, as well as various menus of the terminal. Display unit 2140 may include display panel 2141.

[0243] Audio circuitry 2160, speaker 2161, and microphone 2162 provide an audio interface.

[0244] In this embodiment, the processor 2180 included in the terminal 140 can execute the video retrieval method of the previous embodiment.

[0245] The terminal 140 in this application embodiment includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. This application embodiment can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0246] Figure 22 This is a partial structural block diagram of a server 110 implementing an embodiment of this application. The server 110 can vary significantly due to different configurations or performance, and may include one or more Central Processing Units (CPUs) 2222 (e.g., one or more processors) and a memory 2232, and one or more storage media 2230 (e.g., one or more mass storage devices) for storing application programs 2242 or data 2244. The memory 2232 and storage media 2230 can be temporary or persistent storage. The program stored in the storage media 2230 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server 110. Furthermore, the CPU 2222 may be configured to communicate with the storage media 2230 and execute the series of instruction operations in the storage media 2230 on the server 110.

[0247] Server 110 may also include one or more power supplies 2226, one or more wired or wireless network interfaces 2250, one or more input / output interfaces 2258, and / or one or more operating systems 2241, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0248] The central processing unit 2222 in server 110 can be used to execute the video retrieval method of the embodiments of this application.

[0249] This application also provides a computer-readable storage medium for storing program code for executing the video retrieval methods of the foregoing embodiments.

[0250] This application also provides a computer program product, which includes a computer program. The processor of a computer device reads and executes the computer program, causing the computer device to perform the video retrieval method described above.

[0251] Furthermore, the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0252] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0253] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0254] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0255] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0256] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0257] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0258] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0259] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0260] The above is a detailed description of the embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A video retrieval method, characterized in that, include: The first video is processed by extracting video frames according to the video frame time sequence to obtain multiple candidate key video frames; Clustering is performed on the multiple candidate key video frames to obtain multiple clusters, and target candidate key video frames are extracted from the multiple clusters as key video frames. Image feature extraction processing is performed on the extracted key video frames to obtain the image features corresponding to the key video frames respectively. Image feature fusion processing is performed on the image features corresponding to the multiple key video frames to obtain the video features of the first video. Based on the video features of the first video and the video features corresponding to multiple candidate videos, a second video matching the first video is retrieved from the multiple candidate videos.

2. The video retrieval method according to claim 1, characterized in that, The first video is processed by extracting video frames according to the video frame time sequence to obtain multiple candidate key video frames, including: The first video is processed by sliding a window according to the video frame sequence. After each slide of the sliding window, candidate key video frames are extracted from the sliding window based on the degree of difference between different video frames in the sliding window.

3. The video retrieval method according to claim 2, characterized in that, Before extracting candidate key video frames from the sliding window based on the degree of difference between different video frames in the sliding window, the video retrieval method further includes: The pixel values ​​corresponding to multiple pixels of the video frame in the sliding window are fused to obtain fused pixel values. The degree of difference between different video frames in the sliding window is obtained by performing an absolute difference calculation on the fused pixel values ​​corresponding to different video frames in the sliding window.

4. The video retrieval method according to claim 2, characterized in that, The step of extracting candidate key video frames from the sliding window based on the degree of difference between different video frames in the sliding window includes: The maximum difference is determined based on the difference between different video frames in the sliding window; The video frame corresponding to the maximum difference is extracted from the sliding window as a candidate key video frame.

5. The video retrieval method according to claim 1, characterized in that, The clustering process of the multiple candidate key video frames yields multiple clusters, including: Each of the multiple candidate key video frames is subjected to feature statistical processing to obtain the statistical features corresponding to each of the multiple candidate key video frames. Determine the feature distances between the statistical features corresponding to different candidate key video frames; The candidate key video frames are clustered based on the feature distance to obtain multiple clusters.

6. The video retrieval method according to claim 5, characterized in that, The step of performing feature statistical processing on the plurality of candidate key video frames to obtain the statistical features corresponding to the plurality of candidate key video frames includes: The plurality of candidate key video frames are traversed, and the following processing is performed on the traversed candidate key video frames: A pixel value vector is constructed based on the pixel values ​​corresponding to multiple pixels in the traversed candidate key video frames. The pixel value vector is determined as the statistical feature of the candidate key video frames traversed.

7. The video retrieval method according to claim 1, characterized in that, The step of extracting target candidate key video frames from the plurality of clusters as key video frames includes: The multiple clusters are traversed, and the following processing is performed on the traversed clusters: Smoothing detection processing is performed on multiple candidate key video frames in the traversed cluster to obtain smoothing parameters corresponding to the multiple candidate key video frames in the traversed cluster. The smoothing parameters are negatively correlated with the smoothing degree. Extract the target candidate key video frame corresponding to the maximum smoothness parameter from the traversed clusters, and use it as the key video frame.

8. The video retrieval method according to claim 7, characterized in that, The step of performing smoothing detection processing on multiple candidate key video frames in the traversed clusters to obtain smoothing parameters corresponding to each candidate key video frame in the traversed clusters includes: The process involves iterating through multiple candidate key video frames within the traversed clusters, and performing the following processing on each candidate key video frame: Pixel value difference processing is performed on multiple pixels in the traversed candidate key video frames to obtain the difference pixel values ​​corresponding to the multiple pixels in the traversed candidate key video frames respectively. Differential pixel value fusion processing is performed on the differential pixel values ​​corresponding to multiple pixels in the traversed candidate key video frames to obtain the smoothing parameters of the traversed candidate key video frames.

9. The video retrieval method according to claim 1, characterized in that, After performing image feature fusion processing on the image features corresponding to the multiple key video frames to obtain the video features of the first video, the video retrieval method further includes: The video features of the first video are combined with the features of the first video in other modalities to obtain the multimodal features of the first video. The step of retrieving a second video matching the first video from among the multiple candidate videos based on the video features of the first video and the video features corresponding to the multiple candidate videos includes: Based on the multimodal features of the first video and the multimodal features corresponding to the multiple candidate videos, a second video matching the first video is retrieved from the multiple candidate videos.

10. The video retrieval method according to claim 9, characterized in that, The features of the first video in other modalities include the content description features of the first video; Before performing feature cross-processing on the video features of the first video with features of the first video in other modalities, the video retrieval method further includes: Obtain the content description text of the first video; Text feature extraction processing is performed on the content description text to obtain the content description features of the first video.

11. The video retrieval method according to claim 10, characterized in that, The step of performing image feature extraction processing on the extracted key video frames to obtain the image features corresponding to the key video frames respectively includes: The extracted key video frames are processed by an image encoder to obtain the image features corresponding to each key video frame. The step of performing text feature extraction processing on the content description text to obtain the content description features of the first video includes: The content description text is encoded using a text encoder to obtain the content description features of the first video. The image encoder and the text encoder are obtained through joint training.

12. The video retrieval method according to claim 11, characterized in that, Before performing image encoding processing on the extracted key video frames using an image encoder, the video retrieval method further includes: Obtain image-text pairs and sample labels, wherein the image-text pairs include sample images and sample text, and the sample labels are used to indicate whether there is a correspondence between the sample images and the sample text; The image encoder performs image encoding processing on the sample image to obtain the image features of the sample image; The text encoder performs text encoding on the sample text to obtain the text features of the sample text; Determine the similarity between the image features of the sample image and the text features of the sample text, and determine the loss value based on the similarity and the sample label; The image encoder and the text encoder are trained based on the loss value.

13. The video retrieval method according to claim 9, characterized in that, The features of the first video in other modalities include the duration feature of the first video; before performing feature cross-processing on the video features of the first video and the features of the first video in other modalities, the video retrieval method further includes: Determine the video duration of the first video; Construct a duration description text based on the video duration; The duration description text is subjected to text feature extraction processing to obtain the duration feature of the first video.

14. The video retrieval method according to claim 13, characterized in that, The features of the first video in other modalities include the duration feature of the first video; before performing feature cross-processing on the video features of the first video and the features of the first video in other modalities, the video retrieval method further includes: Determine the video duration of the first video; The duration of the video is one-hot encoded to obtain the duration feature of the first video.

15. The video retrieval method of claim 1, wherein, The step of performing image feature fusion processing on the image features corresponding to the multiple key video frames to obtain the video features of the first video includes: Perform any of the following processes: The image features corresponding to the multiple key video frames are averaged across multiple dimensions to obtain the video features of the first video. The maximum value of the image features corresponding to the multiple key video frames is taken in multiple dimensions to obtain the video features of the first video. The image features corresponding to the multiple key video frames are averaged in multiple dimensions to obtain the average feature, and the image features corresponding to the multiple key video frames are maximized in multiple dimensions to obtain the maximum feature. The average feature and the maximum feature are summed to obtain the video feature of the first video.

16. The video retrieval method of claim 1, wherein, The step of retrieving a second video matching the first video from among the multiple candidate videos based on the video features of the first video and the video features corresponding to the multiple candidate videos includes: Multiple vector indices are established based on the video features corresponding to the multiple candidate videos; Determine the first similarity between the video features of the first video and the plurality of vector indices, and filter the plurality of vector indices according to the first similarity to obtain the filtered vector indices; A second similarity is determined between the video features of the first video and the video features of a plurality of third videos, and the plurality of third videos are filtered according to the second similarity to obtain a second video that matches the first video. The video features of the plurality of third videos are obtained by indexing the filtered vector index, and the plurality of third videos are a subset of the plurality of candidate videos.

17. A video retrieval apparatus characterized by comprising: include: The first extraction module is used to extract video frames from the first video according to the video frame time sequence to obtain multiple candidate key video frames. The second extraction module is used to perform clustering processing on the multiple candidate key video frames to obtain multiple clusters, and extract target candidate key video frames from the multiple clusters respectively as key video frames. The feature extraction module is used to perform image feature extraction processing on the extracted key video frames respectively to obtain the image features corresponding to the key video frames respectively. The feature fusion module is used to perform image feature fusion processing on the image features corresponding to the multiple key video frames respectively to obtain the video features of the first video. The feature retrieval module is used to retrieve a second video that matches the first video from among the multiple candidate videos, based on the video features of the first video and the video features corresponding to the multiple candidate videos.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the video retrieval method according to any one of claims 1 to 16.

19. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the video retrieval method according to any one of claims 1 to 16.

20. A computer program product comprising computer programs or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the video retrieval method according to any one of claims 1 to 16.

Images