Video retrieval methods, devices, equipment and storage media

Abstract

This application provides a video retrieval method, apparatus, device, and storage medium, applicable at least to the fields of artificial intelligence and video deduplication. The method includes: sequentially extracting video frames from all video frames in the video to be retrieved based on the embedding features of every two adjacent video frames, obtaining at least one extracted video frame; determining frame information of the extracted video frame based on its embedding features and attribute parameters within the video to be retrieved; constructing a video pair between the video to be retrieved and each stored video in a preset video library; performing video frame time segment matching on each video pair based on the frame information of the extracted video frame in the video to be retrieved and the stored frame information of the stored video frames in each stored video; and determining duplicate videos of the video to be retrieved from the preset video library based on the matching results. This application improves the efficiency and accuracy of video retrieval and deduplication.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and includes, but is not limited to, a video retrieval method, apparatus, device, and storage medium. Background Technology

[0002] Current video retrieval and deduplication methods typically involve uniformly extracting frames from a video and extracting the embedding features of each frame to characterize the video. During the video retrieval and deduplication process, the number of similar frames between two videos is counted, and the similarity ratio is used to determine whether there are any overlapping parts between the two videos.

[0003] However, the methods in these related technologies do not support variable-length frame extraction, which can easily consume too much feature storage space for slow videos. Furthermore, the frame extraction method is inconsistent with the training image acquisition method, which can lead to discrepancies between the image generation method and the image acquisition method for embedded feature extraction, resulting in biased image representation. Simultaneously, video deduplication methods do not support deduplication after temporal changes in images, treating such changes as duplicates and failing to accurately distinguish duplicate content. Therefore, these related technologies not only consume more storage space and reduce the efficiency of video retrieval and deduplication, but also suffer from low accuracy in video retrieval and deduplication. Summary of the Invention

[0004] This application provides a video retrieval method, apparatus, device, and storage medium, which are applied at least in the fields of artificial intelligence and video deduplication. It can improve the efficiency of video retrieval and deduplication by non-uniformly extracting video frames from the video to be retrieved, and can accurately obtain the matching results between the video to be retrieved and the stored videos based on the frame information of the extracted video frames, thereby improving the accuracy of video retrieval and deduplication.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides a video retrieval method, comprising: extracting embedding features from a video to be retrieved to obtain embedding features of each video frame in the video to be retrieved; extracting video frames from all video frames in the video to be retrieved based on the embedding features of each two adjacent video frames to obtain at least one extracted video frame; determining frame information of the extracted video frame based on the embedding features of each extracted video frame and the attribute parameters of the extracted video frame in the video to be retrieved; constructing a video pair between the video to be retrieved and each stock video in a preset video library; wherein the stock video includes at least one stock video frame, and each stock video frame corresponds to stock frame information; performing video frame time segment matching on each video pair based on the frame information of the extracted video frame in the video to be retrieved and the stock frame information of the stock video frames in each stock video to obtain a matching result; and determining duplicate videos of the video to be retrieved from the preset video library based on the matching result.

[0006] This application provides a video retrieval device, comprising: a feature extraction module for extracting embedding features from a video to be retrieved, obtaining embedding features of each video frame in the video to be retrieved; a video frame extraction module for sequentially extracting video frames from all video frames in the video to be retrieved based on the embedding features of each two adjacent video frames, obtaining at least one extracted video frame; a first determination module for determining frame information of the extracted video frame based on the embedding features of each extracted video frame and attribute parameters of the extracted video frame in the video to be retrieved; a video pair construction module for constructing a video pair between the video to be retrieved and each stock video in a preset video library; wherein the stock video includes at least one stock video frame, and each stock video frame corresponds to stock frame information; a matching module for performing video frame time segment matching on each video pair based on the frame information of the extracted video frame in the video to be retrieved and the stock frame information of the stock video frames in each stock video, obtaining a matching result; and a second determination module for determining duplicate videos of the video to be retrieved from the preset video library based on the matching result.

[0007] This application provides a video retrieval device, including: a memory for storing executable instructions; and a processor for executing the executable instructions stored in the memory to implement the video retrieval method described above.

[0008] This application provides a computer program product, which includes a computer program or executable instructions stored in a computer-readable storage medium; wherein, when the processor of the video retrieval device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, it implements the above-mentioned video retrieval method.

[0009] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the aforementioned video retrieval method.

[0010] The embodiments of this application have the following beneficial effects: During video retrieval, video frames are extracted sequentially based on the embedding features of every two adjacent video frames in the video to be retrieved, resulting in at least one extracted video frame. This achieves non-uniform extraction of video frames from the video to be retrieved, thus avoiding the problem of duplicate features occupying excessive storage space and improving the efficiency of video retrieval and deduplication. Furthermore, since the frame information of each extracted video frame is determined based on its embedding features and attribute parameters in the video to be retrieved, and video pair matching is further performed based on this frame information, accurate matching results between the video to be retrieved and the stored videos can be obtained through matching video pairs using the frame information determined by the embedding features of the extracted video frames, thereby improving the accuracy of video retrieval and deduplication. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of an optional architecture of the video retrieval system provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the video retrieval device provided in the embodiments of this application; Figure 3 This is an optional flowchart illustrating the video retrieval method provided in this application embodiment; Figure 4 This is another optional flowchart illustrating the video retrieval method provided in the embodiments of this application; Figure 5 This is a flowchart illustrating the feature extraction model training method provided in the embodiments of this application; Figure 6 This is a flowchart illustrating the comparison model training method provided in the embodiments of this application; Figure 7 This is a schematic diagram illustrating the implementation process of the frame-skipping model provided in this application embodiment in an application; Figure 8 This is a schematic diagram of the structure of the characterization model in the embodiments of this application; Figure 9 This is a schematic diagram of the frame-skipping model provided in the embodiments of this application; Figure 10 These are effect diagrams of extracting four images from each of the two scenes of the two videos provided in the embodiments of this application; Figure 11 This is a schematic diagram of the matching segment information provided in the embodiments of this application; Figures 12A to 12C This is a schematic diagram illustrating the processing of three intersection scenarios provided in the embodiments of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application.

[0014] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0015] Before explaining the embodiments of this application, the technical terms involved in the embodiments of this application will be explained first.

[0016] (1) Image recognition: In object category-level recognition in images, the specific instance of the object is not considered, but only the category of the object (such as human, dog, cat, bird, etc.) is considered to identify and give the category to which the object belongs. A typical example is the recognition task in the large-scale general object recognition open source dataset (Imagenet), which can identify which of 1000 categories an object belongs to.

[0017] (2) ImageNet: A large-scale, general-purpose open-source dataset for object recognition. It is a large-scale visualization database used for research on visual object recognition software.

[0018] (3) ImageNet pre-trained model: A deep learning network model is trained based on the ImageNet dataset, and the parameter weights of the deep learning network model are the ImageNet pre-trained model.

[0019] (5) Video similarity ranking: In the task of obtaining similar videos in the preset video library for the video to be retrieved, the search results need to be ranked so that the videos in the library at the top are more similar to the video to be retrieved.

[0020] This application provides a video retrieval method, which proposes automatic frame extraction based on a deep learning frame extraction model and video retrieval and deduplication methods based on non-uniform frame extraction. This application designs an image frame extraction model (i.e., a comparison model) based on image pair comparison, achieving free frame selection through the model; it designs a model structure and learning method in which the image frame extraction model and the representation model (i.e., the feature extraction model) share underlying parameters and learn alternately, thereby improving inference efficiency and the self-consistency of the two models in application; and it designs a video deduplication scheme based on automatic frame extraction, enabling the effective application of the above two models in practice and improving the retrieval efficiency and storage performance of the video deduplication system.

[0021] In the video retrieval method provided in this application embodiment, firstly, embedding features are extracted from the video to be retrieved to obtain the embedding features of each video frame in the video to be retrieved; then, based on the embedding features of each two adjacent video frames, video frames are extracted from all video frames in the video to be retrieved to obtain at least one extracted video frame; based on the embedding features of each extracted video frame and the attribute parameters of the extracted video frame in the video to be retrieved, the frame information of the extracted video frame is determined; then, the video to be retrieved is constructed into a video pair with each stock video in a preset video library; wherein, the stock video includes at least one stock video frame, and each stock video frame corresponds to a stock frame information; based on the frame information of the extracted video frame in the video to be retrieved and the stock frame information of the stock video frames in each stock video, video frame time period matching is performed on each video pair to obtain a matching result; finally, based on the matching result, duplicate videos of the video to be retrieved are determined from the preset video library. In this way, the problem of duplicate features occupying too much storage space can be avoided, thereby improving the efficiency of video retrieval and deduplication; and by matching video pairs based on the frame information determined by the embedded features of extracted video frames, the matching results between the video to be retrieved and the stored videos can be accurately obtained, thereby improving the accuracy of video retrieval and deduplication.

[0022] The following describes exemplary applications of the video retrieval device according to embodiments of this application. The video retrieval device provided in this application can be implemented as a terminal or a server. In one implementation, the video retrieval device provided in this application can be implemented as any terminal with video processing capabilities, such as a laptop, tablet, desktop computer, mobile device (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device), smart robot, smart home appliance, and smart vehicle device, or a terminal capable of extracting and matching video frames of a given video to be retrieved to obtain duplicate videos of the video to be retrieved. In another implementation, the video retrieval device provided in this application can also be implemented as a server, wherein the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this application embodiment. The following describes an exemplary application when the video retrieval device is implemented as a server.

[0023] See Figure 1 , Figure 1 This is a schematic diagram of an optional architecture of the video retrieval system provided in this application embodiment. This application embodiment uses the application of the video retrieval method to a video application as an example for illustration. To support any video application, the terminal in this application embodiment has at least one video application installed, through which videos can be played. For any video to be played, the video to be played can be used as the video to be retrieved. First, the video to be retrieved is searched from a preset video library. When no duplicate video to be retrieved is found in the preset video library, it indicates that the current video to be retrieved is a normal video and can be played normally through the video application.

[0024] In this embodiment, the video retrieval system 10 includes at least a terminal 100, a network 200, and a server 300, wherein the server 300 may be a server for a video application. The server 300 may constitute the video retrieval device of this embodiment. The terminal 100 connects to the server 300 via the network 200, which may be a wide area network (WAN), a local area network (LAN), or a combination of both. Before running the video application and playing the video to be retrieved, the video retrieval method provided in this embodiment is used to generate a video retrieval request. The video retrieval request includes at least the video to be retrieved, and the terminal 100 sends the video retrieval request to the server 300 via the network 200. Server 300 performs embedding feature extraction on the video to be retrieved, obtaining the embedding features of each video frame in the video to be retrieved; and sequentially extracts video frames from all video frames in the video to be retrieved based on the embedding features of each two adjacent video frames, obtaining at least one extracted video frame; based on the embedding features of each extracted video frame and the attribute parameters of the extracted video frame in the video to be retrieved, the frame information of the extracted video frame is determined; then, the video to be retrieved is constructed into a video pair with each stock video in the preset video library; wherein, the stock video includes at least one stock video frame, and each stock video frame corresponds to a stock frame information; then, based on the frame information of the extracted video frame in the video to be retrieved and the stock frame information of the stock video frames in each stock video, video frame time segment matching is performed on each video pair to obtain a matching result; finally, based on the matching result, duplicate videos of the video to be retrieved are determined from the preset video library. If server 300 determines that the video to be searched is a duplicate video, it means that the preset video library already stores a similar video to be searched. Therefore, the duplicate video is returned to terminal 100, and terminal 100 plays the duplicate video. If server 300 does not find a duplicate video to be searched, it means that the preset video library does not store a similar video to be searched. Therefore, the search result is fed back to terminal 100, and terminal 100 responds to the search result by playing the video to be searched.

[0025] The video retrieval method provided in this application embodiment can also be implemented based on a cloud platform and through cloud technology. For example, the server 300 mentioned above can be a cloud server. The cloud server can perform embedded feature extraction on the video to be retrieved, or extract video frames from all video frames in the video to be retrieved, or determine the frame information of the extracted video frames, and perform video frame time segment matching on each video pair, etc.

[0026] In some embodiments, a cloud storage system may also be included, where the video to be retrieved can be stored, or where existing video frames can be stored, or where the matching results for each video pair can be stored. Thus, when the video to be retrieved is played on the current interface of the terminal, the matching results for each video pair can be retrieved from the cloud storage system to perform video duplication detection and deduplication.

[0027] It's important to clarify that cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a collective term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies applied in the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0028] Figure 2 This is a schematic diagram of the structure of the video retrieval device provided in the embodiments of this application. Figure 2 The video retrieval device shown includes at least one processor 310, a memory 350, at least one network interface 320, and a user interface 330. The various components in the video retrieval device are coupled together via a bus system 340. It is understood that the bus system 340 is used to implement communication between these components. In addition to a data bus, the bus system 340 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 340.

[0029] The processor 310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0030] User interface 330 includes one or more output devices 331 that enable the presentation of media content, and one or more input devices 332.

[0031] Memory 350 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 350 may optionally include one or more storage devices physically located remote from processor 310. Memory 350 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 350 described in this application embodiment is intended to include any suitable type of memory. In some embodiments, memory 350 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as illustrated below.

[0032] Operating system 351 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 352 is used to reach other computing devices via one or more (wired or wireless) network interfaces 320, exemplary network interfaces 320 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. The input processing module 353 is used to detect and translate one or more user inputs or interactions from one or more input devices 332.

[0033] In some embodiments, the apparatus provided in this application may be implemented in software. Figure 2 A video retrieval device 354 stored in memory 350 is shown. This video retrieval device 354 can be a video retrieval device within a video retrieval equipment. It can be software in the form of programs and plugins, including the following software modules: a feature extraction module 3541, a video frame extraction module 3542, a first determination module 3543, a video pair construction module 3544, a matching module 3545, and a second determination module 3546. These modules are logically connected and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0034] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video retrieval method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0035] The video retrieval methods provided in the embodiments of this application can be executed by a video retrieval device, which can be a terminal or a server. That is, the video retrieval methods in the embodiments of this application can be executed by a terminal, by a server, or by interaction between a terminal and a server.

[0036] See Figure 3 , Figure 3 This is an optional flowchart illustrating the video retrieval method provided in this application embodiment. The following will be combined with... Figure 3 The steps shown will be explained. It should be noted that... Figure 3 The video retrieval method in this example is illustrated by using a server as the execution entity.

[0037] Step S301: Extract the embedding features of the video to be retrieved to obtain the embedding features of each video frame in the video to be retrieved.

[0038] The video to be searched can be any type of video, such as a long video or a short video, an original video, or a video obtained by re-editing and producing an existing video. In this embodiment, a search can be performed on the video to be searched to determine whether the same or similar videos are stored in a preset video library, thereby achieving video deduplication processing to avoid wasting storage space by storing duplicate videos in the library.

[0039] Embedded feature extraction of the video to be retrieved refers to extracting features at different levels from the video frames of the video to obtain the embedded features of each video frame. These different levels of features include shallow features and deep features. Shallow features include, but are not limited to, features such as corner points, texture, and brightness of the image; deep features are more abstract features in the image, including, but not limited to, features such as eyes, nose, mouth, and limbs.

[0040] In some embodiments, the embedding feature extraction process can be implemented using a pre-trained feature extraction model, which has multiple convolutional layers, pooling layers, and fully connected layers. The multiple convolutional layers are used to extract features at different levels from the video frame. For example, there can be N convolutional layers: Conv1, Conv2…ConvN. Shallow layers like Conv1 and Conv2 extract shallow features from the video frame, such as corners, textures, and brightness; deeper layers like ConvN-1 and ConvN extract more abstract features, such as eyes, nose, mouth, and limbs. The pooling layers perform max pooling on the features output from the multiple convolutional layers. During computation, the input video frame is divided into several rectangular regions, and the maximum value is output for each sub-region. Intuitively, max pooling is effective because, after a feature is discovered, its precise location is far less important than its relative position to other features. Pooling layers continuously reduce the spatial size of the data, thus decreasing the number of parameters and computational cost, which to some extent controls overfitting. Therefore, inserting a pooling layer after the convolutional layer can avoid overfitting after the convolutional processing. The fully connected layer can be used to combine the learned features, thereby learning how to distinguish objects in a video frame, such as people, cars, cats, and dogs.

[0041] In some embodiments, the video to be retrieved can first be segmented into frames to obtain multiple video frames corresponding to the video to be retrieved. Then, embedding feature extraction can be performed on each video frame. Alternatively, embedding feature extraction can be performed every certain number of video frames to reduce the amount of data for video frame embedding feature extraction and improve video retrieval efficiency. For example, embedding feature extraction can be performed on one video frame every three frames from all extracted video frames.

[0042] In this embodiment, the embedded features of the video frame are stored in the form of embedded feature vectors. The embedded feature vectors have a certain dimension. In the subsequent data processing, data calculation is performed based on the embedded feature vectors, thereby realizing the video retrieval process.

[0043] Step S302: Based on the embedding features of each pair of adjacent video frames, extract video frames from all video frames in the video to be retrieved to obtain at least one extracted video frame.

[0044] Here, after obtaining the embedding features of each video frame, video frames are extracted sequentially based on the embedding features of every two adjacent video frames, according to the order of the video frames in the video to be retrieved. During video frame extraction, one video frame is determined from two video frames at a time.

[0045] In the implementation process, the similarity between the embedded features of two video frames can be determined first, i.e., by calculating the distance between the two embedded feature vectors, which can be calculated using the Euclidean distance. When the distance between the two embedded feature vectors is less than a distance threshold, it indicates that the similarity between the embedded features of the two video frames is relatively high, i.e., the similarity between the embedded features of the two video frames is greater than the similarity threshold; when the distance between the two embedded feature vectors is greater than or equal to the distance threshold, it indicates that the similarity between the embedded features of the two video frames is relatively low, i.e., the similarity between the embedded features of the two video frames is less than or equal to the similarity threshold.

[0046] When the similarity is less than or equal to the similarity threshold, the video frame that appears earlier in the video to be retrieved, i.e., the one with the smaller timestamp, is selected as the extracted video frame. In other words, if the similarity is less than or equal to the similarity threshold, it indicates that the two video frames have low similarity and are significantly different; therefore, the earlier video frame can be selected for extraction. When the similarity is greater than the similarity threshold, the two video frames can be compared based on their embedding features to determine which frame is more suitable for extraction. During the comparison, a comparison score is determined, and the video frame with the higher score is selected as the extracted video frame. In other words, if the similarity is greater than the similarity threshold, it means that the two video frames are highly similar and have little difference. Therefore, one of the two similar video frames with a higher comparison score can be selected to obtain the extracted video frame. That is, a more suitable extracted video frame for subsequent matching is extracted from the similar video frames.

[0047] Since when the similarity is greater than the similarity threshold, the video frame with the higher comparison score will be extracted from two adjacent video frames as the extracted video frame, when multiple consecutive video frames (greater than or equal to 3 frames) are similar video frames, it is possible to extract only one video frame from these multiple video frames as the final extracted video frame.

[0048] Step S303: Determine the frame information of the extracted video frame based on the embedding features of each extracted video frame and the attribute parameters of the extracted video frame in the video to be retrieved.

[0049] Here, frame information is used to characterize the extracted video frame and distinguish it from other extracted video frames. After extracting the embedding features of the extracted video frame, the frame information can be determined based on the embedding features and the attribute parameters of the extracted video frame in the video to be retrieved.

[0050] The attribute parameters of the extracted video frames in the video to be retrieved include, but are not limited to: the video identifier of the video to be retrieved, the time corresponding to the extracted video frame in the video to be retrieved (which can also be understood as the start time, because the video frames are extracted using a non-uniform frame extraction method, so there is a certain time interval between two adjacent extracted video frames in the video to be retrieved. That is to say, each extracted video frame corresponds to a video segment in the video to be retrieved, and the extracted video frame is the starting video frame of the video segment. Therefore, the time corresponding to the extracted video frame in the video to be retrieved can be the start time corresponding to the extracted video frame in the video to be retrieved), and the duration of the video segment corresponding to the extracted video frame (i.e., the duration of the video frame represented by the extracted video frame in the video to be retrieved, which can also be understood as the duration of the video segment from the extracted video frame to the next adjacent extracted video frame).

[0051] In this embodiment of the application, after obtaining the embedding features of the extracted video frame and the aforementioned attribute parameters of the extracted video frame in the video to be retrieved, the embedding features of the extracted video frame and the aforementioned attribute parameters of the extracted video frame in the video to be retrieved can be determined as the frame information of the extracted video frame.

[0052] In some embodiments, during video frame segmentation, a timestamp can be added to each video frame according to its position information within the video to be retrieved, and the video identifier of the video to be retrieved can be recorded. Thus, after the extracted video frames are obtained, the aforementioned attribute parameters of the extracted video frames can be determined based on their timestamps.

[0053] Step S304: Construct a video pair by combining the video to be retrieved with each stock video in the preset video library; wherein, the stock video includes at least one stock video frame, and each stock video frame corresponds to a stock frame information.

[0054] Here, the inventory frame information includes, but is not limited to, at least one of the following: the video identifier of the inventory video, the video duration of the inventory video, the embedding features of the inventory video frame, the start time corresponding to the inventory video frame in the inventory video, and the video frame duration represented by the inventory video frame in the inventory video.

[0055] In this embodiment, the inventory frame information of the inventory video frames can be obtained in the same way as obtaining the frame information of the extracted video frames. When performing video frame segmentation on the inventory video, a timestamp can be added to each video frame of the inventory video according to its position within the inventory video, and the video identifier of the inventory video can be recorded. Thus, after extracting the inventory video frames, the inventory frame information can be determined based on the timestamps of the inventory video frames.

[0056] Step S305: Based on the frame information of the extracted video frames in the video to be retrieved and the inventory frame information of the inventory video frames in each inventory video, perform video frame time segment matching on each video pair to obtain the matching result.

[0057] In this embodiment, time-segment matching based on automatic frame extraction can be performed on the video to be retrieved and the inventory video in each video pair to achieve the video frame time-segment matching process. During implementation, the time segment (i.e., video frame duration) represented by the embedding feature of each inventory video frame can be recorded when each inventory video is added to the database. When matching the video to be retrieved and the inventory video, each matching of two embedding features (one for the video to be retrieved and one for the inventory video) indicates that two time segments of the two videos have matched. Subsequently, by merging the matched time segments, the sparsely extracted video matching result can be obtained.

[0058] Step S306: Based on the matching results, identify duplicate videos of the video to be retrieved from the preset video library.

[0059] In this embodiment of the application, the matching result includes the matching duration of the video to be retrieved and the inventory video. The inventory video with the longest matching duration can be identified as the duplicate video of the video to be retrieved from the preset video library.

[0060] In other embodiments, multiple inventory videos in a preset video library can be sorted in descending order of matching duration between the video to be retrieved and each inventory video to form an inventory video sequence. Then, the top N inventory videos in the inventory video sequence are filtered out according to their arrangement order and returned to the user for deduplication. Alternatively, inventory videos with matching durations greater than a duration threshold can be filtered out and returned to the user for deduplication. This avoids the problem of misjudging or incorrectly identifying duplicate videos due to errors in matching duration calculation caused by the accuracy of the entire video retrieval process. At the same time, filtering out multiple potentially matching (the top N inventory videos in the inventory video sequence or inventory videos with matching durations greater than the duration threshold are all high-matching inventory videos) for the user to judge not only reduces the amount of inventory videos that the user needs to manually judge but also ensures the accuracy of the final video deduplication process.

[0061] The video retrieval method provided in this application extracts video frames from all video frames in the video to be retrieved based on the embedding features of every two adjacent video frames during video retrieval. This results in at least one extracted video frame, achieving non-uniform extraction of video frames from the video to be retrieved. This avoids the problem of duplicate features occupying excessive storage space, thereby improving the efficiency of video retrieval and deduplication. Furthermore, since the frame information of each extracted video frame is determined based on its embedding features and attribute parameters in the video to be retrieved, and video pair matching is further performed based on this frame information, accurate matching results between the video to be retrieved and the stored videos can be obtained through matching video pairs based on the frame information determined by the embedding features of the extracted video frames, thus improving the accuracy of video retrieval and deduplication.

[0062] The following examples illustrate possible application scenarios of the embodiments of this application. The video retrieval method of the embodiments of this application can be applied to any of the following scenarios: Scenario 1: Video Playback Scenario. When playing a video, for the video currently to be played on the terminal, this video can be used as the search target. A search is performed in a preset video library to determine if there are any duplicate videos matching the video to be played. If so, the duplicate video can be directly retrieved from the preset video library for playback. Since the videos in the preset video library are all pre-loaded and playable in real-time without requiring video encoding, this avoids the need for secondary retrieval of the video to be played, reducing bandwidth consumption during video playback. If not, the video to be played can be loaded and played during the loading process. After loading, the video to be played can be stored in the preset video library as part of its inventory, thus updating the videos in the preset video library. When searching the preset video library for duplicate videos matching the video to be played, the video retrieval method provided in this application embodiment can be used for duplicate video retrieval.

[0063] Scenario 2: Video Entry Scenario. To avoid wasting storage space by storing duplicate or similar videos during video entry, the video retrieval method provided in this application embodiment can be used. First, a duplicate video search is performed on the video to be entered to check if there are any duplicate videos matching the video to be entered (i.e., the video to be retrieved) in the preset video library. If so, the video to be retrieved is deduplicated. If so, the video to be entered into the library is prohibited from being stored, and the user is prompted that a duplicate video of the video to be entered into the preset video library is already stored, and the video to be entered does not need to be stored. If not, the video to be entered into the preset video library.

[0064] Scenario 3: Similar Video Determination. When determining whether a video to be judged is a similar video to a video in the database, the video retrieval method provided in this application embodiment can be used to determine whether the video to be judged is identical or similar to normal videos in the preset video library. That is, it determines whether there are duplicate videos of the video to be judged (i.e., the video to be retrieved) in the preset video library. If so, it indicates that the video to be judged is a similar video, and an alert can be issued or legal support can be sought. If not, it indicates that the video to be judged is a normal video, and it can be played normally.

[0065] In some embodiments, the video retrieval method is applied to a video retrieval system, which includes at least a terminal and a server. The terminal has a video application installed, which may be a video playback application. This application uses the video playback scenario in Scenario 1 above as an example to illustrate the implementation process of the video retrieval method based on the video retrieval system.

[0066] Figure 4 This is another optional flowchart illustrating the video retrieval method provided in the embodiments of this application, such as... Figure 4 As shown, the method includes the following steps: Step S401: The terminal obtains the user's video playback operation.

[0067] In this embodiment, users can perform video playback operations on a video playback application. Each video playback operation corresponds to a video to be played. Before playing the video, the method described in this embodiment needs to be used to search for the video to be played to determine whether a duplicate video of the video to be played is stored in a preset video library.

[0068] In step S402, the terminal responds to the video playback operation by generating a video retrieval request.

[0069] The video search request includes the video to be searched, which is the video the user has requested to play. The video search request is used to request a search for the video to be searched and to determine whether a duplicate video of the video to be searched is stored in the preset video library.

[0070] In step S403, the terminal sends a video retrieval request to the server.

[0071] In step S404, the server responds to the video retrieval request by inputting the video to be retrieved into the feature extraction model. The feature extraction model performs embedding feature extraction on the video to be retrieved, thereby obtaining the embedding features of each video frame in the video to be retrieved.

[0072] In this embodiment of the application, the feature extraction module includes at least two convolutional layers, a pooling layer, and a fully connected layer connected in sequence.

[0073] After the video to be retrieved is input into the feature extraction model, the feature extraction model performs embedded feature extraction through the following steps S4041 to S4043 (not shown in the figure): Step S4041: Extract features from different levels of each video frame sequentially through at least two convolutional layers to obtain the convolutional features of each video frame.

[0074] Here, there are at least two convolutional layers, such as Conv1, Conv2...ConvN, a total of N convolutional layers. Each convolutional layer is used to extract a type of feature at a certain level in the video frame. For example, shallow features such as corners, textures, and brightness can be extracted, or deep features such as eyes, nose, mouth, and limbs can be extracted.

[0075] Step S4042: Max pooling is performed on the convolutional features through a pooling layer to obtain the pooled convolutional features for each video frame.

[0076] Step S4043: The pooled convolutional features are combined through a fully connected layer to obtain the embedded features for each video frame.

[0077] Here, the fully connected layer is used to combine the learned features, thereby learning how to distinguish objects in video frames.

[0078] In step S405, the server sequentially extracts video frames from all video frames in the video to be retrieved based on the embedding features of each pair of adjacent video frames, obtaining at least one extracted video frame.

[0079] In some embodiments, the video frame extraction process in step S405 can be implemented by the following steps S4051 to S4055 (not shown in the figure): Step S4051: For two adjacent video frames, determine the similarity between the embedded features of the two video frames.

[0080] Step S4052: When the similarity is greater than the similarity threshold, the comparison score of the two video frames is determined based on the embedding features of the two video frames.

[0081] In some embodiments, the alignment score of two video frames can be determined using an alignment model. This alignment model includes a mapping layer, a difference layer, and a prediction layer. When the embedded features of two video frames are input into the alignment model, the alignment score of the two video frames can be determined through the following steps S11 to S13 (not shown in the figure): Step S11: By comparing the mapping layer of the model, the embedded features are mapped to obtain the key features in the embedded features of each video frame in the two video frames.

[0082] In one implementation, the mapping layer can be a fully connected layer (FC). Through the mapping layer, information more suitable for comparison can be extracted from the embedded features; that is, information more conducive to similarity differentiation using the embedded features. Since this information in the embedded features is not intuitively known, generally, images more similar in distribution to the training samples are more conducive to ensuring that the image comparison effect is consistent with the training performance (e.g., having the same accuracy or an image distribution histogram). In this embodiment, the extracted video frames obtained by the comparison model better meet the feature extraction model's metric requirements: the better performance of the extracted video frames after frame extraction by the comparison model in embedded feature metric learning (i.e., the two-stage training of the comparison model) serves as indirect supervision information, making the frame extraction by the comparison model more consistent with the metric learning requirements.

[0083] Step S12: By comparing the difference layers of the model, the difference vector of the key features corresponding to the two video frames is determined.

[0084] Here, the difference vector can be obtained by subtracting the key features corresponding to two video frames through the difference layer.

[0085] In one implementation, the difference layer can be implemented using a diff layer based on the diff algorithm.

[0086] Step S13: By comparing the prediction layer of the model, the difference between the two video frames is scored based on the difference vector to obtain the comparison score of the two video frames.

[0087] Here, the prediction layer is a scoring module in the comparison model, used to score the differences between two video frames. The comparison score can include 1 and -1, where a comparison score of 1 indicates that the corresponding video frame is relatively better, and a comparison score of -1 indicates that the corresponding video frame is relatively worse. It should be noted that "better" here refers to the feature extraction model. For example, if we select an image from consecutive frames of scenes A and B in the video, the image whose embedded features are closer in similar images (i.e., scenes A and B are similar) and farther apart in dissimilar images (scenes A and B are dissimilar) is considered the better image.

[0088] Step S4053: Among the two video frames, the one with the higher comparison score is determined as the extracted video frame.

[0089] In this embodiment of the application, each video frame corresponds to a timestamp in the video to be retrieved.

[0090] Step S4054: For two adjacent video frames, if the similarity is less than or equal to the similarity threshold, the video frame with the smaller timestamp is identified as the extracted video frame.

[0091] Step S4055: When any video frame is determined to be the extracted video frame, continue to extract video frames from the two video frames that are adjacent to the extracted video frame and whose timestamps are greater than the timestamps of the extracted video frame, and obtain the next extracted video frame.

[0092] In this embodiment of the application, starting from the first video frame of the video to be detected, the above-mentioned judgment and extraction operations are performed on each video frame one by one, and finally multiple extracted video frames are obtained.

[0093] Step S406: The server determines the frame information of the extracted video frame based on the embedding features of each extracted video frame and the attribute parameters of the extracted video frame in the video to be retrieved.

[0094] In some embodiments, the frame information extracted from the video frame includes, but is not limited to, at least one of the following: video identifier, video duration, embedding features, start time, and video frame duration. Here, the video identifier refers to the video identifier of the video to be retrieved; the video duration refers to the video duration of the video to be retrieved, which can be obtained from the attribute parameters of the video to be retrieved; the start time refers to the start time corresponding to the extracted video frame in the video to be retrieved; and the video frame duration refers to the duration of the video segment from the extracted video frame to the next adjacent extracted video frame, which can determine the start time corresponding to the extracted video frame in the video to be retrieved and the video frame duration represented by the extracted video frame in the video to be retrieved.

[0095] In step S407, the server constructs a video pair between the video to be retrieved and each stock video in the preset video library; wherein, the stock video includes at least one stock video frame, and each stock video frame corresponds to a stock frame information.

[0096] In step S408, the server performs video frame time segment matching on each video pair based on the frame information of the extracted video frames in the video to be retrieved and the inventory frame information of the inventory video frames in each inventory video, and obtains the matching result.

[0097] In step S409, the server determines the duplicate videos of the video to be retrieved from the preset video library based on the matching results.

[0098] In step S410, the server sends the repeating video to the terminal.

[0099] Step S411: The terminal plays the repeating video on the client of the video application.

[0100] The video retrieval method provided in this application uses a feature extraction model to extract embedded features from the video to be retrieved, and a comparison model to determine the comparison score of two video frames. Then, video frames are extracted based on the comparison score. In this way, by matching video pairs based on the frame information determined by the embedded features of the extracted video frames, the matching results between the video to be retrieved and the videos in the inventory can be accurately obtained, improving the accuracy of video retrieval and deduplication. Moreover, the two models share the underlying network structure, which can reduce the parameter calculation amount of model inference during application.

[0101] In some embodiments, the server performs video frame time segment matching on each video pair based on the frame information of the extracted video frames in the video to be retrieved and the inventory frame information of the inventory video frames in each inventory video to obtain the matching result. This can be achieved through the following steps S21 to S26 (not shown in the figure): Step S21: Obtain the embedding features of each extracted video frame in the video to be retrieved and the embedding features of each stock video frame in the stock videos.

[0102] Here, the same method used for extracting embedding features from the videos to be retrieved can be applied to extract embedding features from the videos in the inventory, thus obtaining the embedding features of the videos in the inventory.

[0103] In some embodiments, after obtaining the video to be retrieved, the video to be retrieved and the stored videos can be constructed into a video pair. Then, embedding feature extraction is used to extract embedding features from both the video to be retrieved and the stored videos in the video pair. Alternatively, embedding features can be extracted from the stored videos when they are stored in the database, and the embedding feature mappings between the stored videos and their frames can be stored in a preset video library. In this way, after constructing the video pair, the embedding features of the stored video frames can be directly obtained from the preset video library.

[0104] In this embodiment of the application, video frames can also be extracted from the stock videos in advance to obtain stock video frames, wherein the stock video frames are obtained after extracting video frames from the stock videos.

[0105] Step S22: Based on the embedding features of the extracted video frames and the embedding features of the stock video frames, determine the similar video frames of each extracted video frame in the video to be retrieved from at least one stock video frame in the stock video.

[0106] Here, the Euclidean distance between the embedded features of the extracted video frame and the embedded features of the stock video frame can be calculated. Stock video frames whose Euclidean distance is less than the distance threshold are identified as similar video frames to the extracted video frame.

[0107] Step S23: Determine the first start time and first video frame duration corresponding to each extracted video frame in the retrieved video, and the second start time and second video frame duration corresponding to each similar video frame in the inventory video.

[0108] Step S24: When the first start time of any extracted video frame in the video to be retrieved matches the second start time of any similar video frame in the inventory video, and the duration of the first video frame of the extracted video frame matches the duration of the second video frame of the similar video frame, the similar video frame is determined to be the matching video frame of the extracted video frame.

[0109] Here, matching the first start time of any extracted video frame in the video to be retrieved with the second start time of any similar video frame in the inventory video means that the time deviation determined based on the first start time of the extracted video frame is the same as the time deviation determined based on the second start time of the similar video frame. Here, the time deviation refers to the time offset of the extracted video frame.

[0110] Step S25: Based on the second video frame duration of the matching video frame corresponding to each extracted video frame in the video to be retrieved, determine the matching duration between the video to be retrieved and the stored videos.

[0111] Here, the duration of the second video frame corresponding to all extracted video frames can be summed to obtain the matching duration between the video to be retrieved and the stored videos.

[0112] Step S26: Determine the matching duration as the matching result.

[0113] Correspondingly, based on the matching results, the duplicate videos of the video to be retrieved are determined from the preset video library, which can be achieved through the following step S27 (not shown in the figure): Step S27: From the preset video library, identify the stock videos with the longest matching duration, or the stock videos with a matching duration greater than the matching duration threshold, as duplicate videos of the video to be retrieved.

[0114] In some embodiments, before performing video frame time period matching on each video pair, inventory video frames and inventory frame information can be extracted from the inventory videos in the video pair. The inventory video frame extraction and inventory frame information extraction can be implemented through the following steps S31 to S33 (not shown in the figure): Step S31: Extract the embedding features of the inventory videos in each video pair to obtain the inventory embedding features of each video frame in the inventory videos.

[0115] Step S32: Based on the inventory embedding features of every two adjacent video frames in the inventory video, extract video frames from all video frames in the inventory video to obtain at least one inventory video frame.

[0116] Step S33: Based on the inventory embedding features of each inventory video frame and the attribute parameters of the inventory video frame in the inventory video, determine the inventory frame information of the inventory video frame.

[0117] In this embodiment, the same method as for embedding feature extraction and video frame extraction of the video to be retrieved is used to perform embedding feature extraction and video frame extraction on the inventory videos. Embedding feature extraction and video frame extraction can be performed after embedding feature extraction and video frame extraction of the video to be retrieved, and after the video pairs are constructed, or they can be performed when the inventory videos are added to the database; this embodiment does not impose any limitations.

[0118] This application embodiment further provides a training method for a feature extraction model. This training method can be implemented using any feature extraction model training device. The feature extraction model training device can be the same device as the video retrieval device in this application embodiment, or it can be a different device. Figure 5 This is a flowchart illustrating the feature extraction model training method provided in this application embodiment, as shown below. Figure 5 As shown, the method includes the following steps: Step S501: The feature extraction model training device performs segmentation processing on the sample video to obtain multiple segmentation sample groups; wherein, each segmentation sample group includes at least two sample images, and every two sample images in each segmentation sample group form a similar sample pair.

[0119] Here, shot segmentation refers to dividing a long video into multiple consecutive shots, with each shot containing approximately the same images. After shot segmentation, multiple sample images are randomly selected to form a shot sample group.

[0120] In step S502, the feature extraction model training device determines multiple triples from multiple segmentation sample groups based on similar sample pairs in each segmentation sample group.

[0121] In some embodiments, the method for determining the triplet in step S502 can be implemented by the following steps S5021 to S5027 (not shown in the figure): Step S5021: Randomly select a similar sample pair from each storyboard sample group to obtain multiple similar sample pairs with the same number of groups as the multiple storyboard sample groups.

[0122] Step S5022: Divide multiple similar sample pairs into equal batches to obtain multiple batches of similar sample pairs; wherein each batch of similar sample pairs contains a first number of similar sample pairs.

[0123] Step S5023: From each batch of similar sample pairs, determine any one of the similar sample pairs as the target sample pair, and determine the other similar sample pairs in the batch other than the target sample pair as the remaining sample pairs.

[0124] Step S5024: Determine the image distance between any sample image in the target sample pair and any sample image in each remaining sample pair.

[0125] Here, we can randomly select one sample image from each of the remaining sample pairs and calculate the image distance between this sample image and any sample image in the target sample pair.

[0126] Step S5025: Sort the remaining sample pairs according to the order of image distance from largest to smallest to form a sequence of remaining sample pairs.

[0127] Step S5026: Select the second number of sample images in sequence according to the order of the remaining sample pairs in the remaining sample pair sequence to form a negative sample image set.

[0128] For example, 10 sample images can be selected from the remaining sample pair sequence to form a negative sample image set.

[0129] Step S5027: Construct a triplet from two sample images in the target sample pair and one sample image in the negative sample image set.

[0130] In this embodiment of the application, a triplet with the same number of sample images as the negative sample image set can be constructed. That is, a triplet can be constructed by each sample image in the negative sample image set and two sample images in the target sample pair. If the negative sample image set has M sample images, then M triplets can be constructed.

[0131] In step S503, the feature extraction model training device randomly selects a similar sample pair from each segment sample group as the input sample of the feature extraction model, and inputs the input sample of the feature extraction model into the feature extraction model.

[0132] In step S504, the feature extraction model training device extracts features from different levels of each sample image in the input sample through at least two convolutional layers of the feature extraction model, thereby obtaining the sample convolutional features of each sample image.

[0133] In step S505, the feature extraction model training device performs max pooling on the sample convolution features through the pooling layer of the feature extraction model, thereby obtaining the pooled sample convolution features of each sample image.

[0134] In step S506, the feature extraction model training device combines the pooled sample convolution features through the fully connected layer of the feature extraction model to obtain the sample embedding features of each sample image.

[0135] In step S507, the feature extraction model training device inputs the sample embedded features as the prediction result of the feature extraction model into the triplet loss model. Through the triplet loss model, the loss is calculated based on multiple triplets on the prediction result of the feature extraction model to obtain the triplet loss.

[0136] Step S508: The feature extraction model training device corrects the parameters in the feature extraction model based on triplet loss to obtain the trained feature extraction model.

[0137] The feature extraction model training method provided in this application constructs triples and calculates the loss of the feature extraction model prediction results based on multiple triples to obtain the triple loss. Then, based on the triple loss, the parameters in the feature extraction model are corrected to obtain the trained feature extraction model, which can improve the accuracy of the model parameters.

[0138] This application embodiment further provides a method for training a comparison model. This method can be implemented using any comparison model training device. The comparison model training device can be the same device as the video retrieval device in this application embodiment, or it can be a different device. Alternatively, the comparison model training device can be the same device as the feature extraction model training device in this application embodiment, or it can be a different device. Figure 6 This is a flowchart illustrating the comparison model training method provided in the embodiments of this application, as shown below. Figure 6 As shown, the method includes the following steps: Step S601: The comparison model training device performs segmentation processing on the sample video to obtain multiple segmentation sample groups; wherein, each segmentation sample group includes at least two sample images, and every two sample images in each segmentation sample group form a similar sample pair.

[0139] In step S602, the comparison model training device inputs each sample image in each segment sample group into the feature extraction model to obtain at least two image embedding features.

[0140] Step S603: The comparison model training device determines the best sample corresponding to the segmentation sample group based on at least two image embedding features; and generates label samples based on the best sample.

[0141] In some embodiments, the optimal sample can be determined through the following steps S6031 to S6033 (not shown in the figure): Step S6031: Determine the feature centers of at least two image embedding features.

[0142] Here, data center calculations can be performed based on all image embedding features to obtain feature centers for at least two image embedding features.

[0143] Step S6032: Determine the distance between each image embedding feature and the feature center.

[0144] Step S6033: The sample image corresponding to the embedding feature of the image with the smallest distance is determined as the best sample.

[0145] In step S604, the comparison model training device uses the image embedding features of each similar sample pair in each segment sample group as the input sample of the comparison model, and inputs the input sample of the comparison model into the comparison model.

[0146] Step S605: The comparison model training device performs feature mapping on the input samples through the mapping layer of the comparison model to obtain the key features of the samples in the image embedding features of each sample image in the similar sample pair.

[0147] Step S606: The comparison model training device determines the sample difference vector of the key features of the two sample images by comparing the difference layer of the comparison model.

[0148] In step S607, the comparison model training device uses the prediction layer of the comparison model to score the difference between two sample images based on the sample difference vector, thereby obtaining the sample comparison score of the two sample images.

[0149] In step S608, the comparison model training device inputs the sample comparison score as the comparison model prediction result into the regression loss model. Based on the best sample and the label sample, the regression loss is calculated by comparing the comparison model prediction result through the regression loss model to obtain the regression loss.

[0150] Step S609: The comparison model training device corrects the parameters in the comparison model based on the regression loss to obtain the trained comparison model.

[0151] The comparison model training method provided in this application determines the best sample and label sample, and calculates the loss of the model prediction results based on the comparison of the best sample and label sample to obtain the regression loss. Then, based on the regression loss, the parameters in the model are corrected to obtain the trained comparison model, which can improve the accuracy of the model parameters.

[0152] In some embodiments, the feature extraction model and the comparison model can share the underlying network structure, thereby reducing the parameter computation cost of model inference during application. Furthermore, the feature extraction model and the comparison model can learn alternately during training, or the training results of the comparison model can regressively update the parameters of the feature extraction model, achieving recursive updates between the two models. This ensures that the final frame extraction results generated by the comparison model are compatible with the feature extraction model, thus avoiding significant differences between the actual image representation results in practical applications and those during model training.

[0153] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0154] The conventional approach to video deduplication tasks (i.e., identification, retrieval, and deduplication) involves extracting images (i.e., video frames) from the video at regular intervals, also known as uniform frame extraction, and then representing these images using deep learning embedding features. Deduplication is then performed based on these embedded feature representations. However, given the massive amount of videos added daily on the internet, uniformly extracting frames and representing them using deep learning models would be extremely resource-intensive in inference, retrieval, and storage. Many short videos often feature relatively static scenes, such as live-streamed homework, videos of people drawing eyebrows, or chess games. These videos may remain unchanged for several minutes. In such cases, uniform frame extraction can easily lead to feature redundancy across multiple frames, consuming excessive storage and computational resources. Conversely, increasing the frame extraction interval to handle static scenes can lead to missing images of fast-moving scenes, such as scenes of scenery captured while driving. Therefore, automatically extracting frames to avoid feature redundancy while simultaneously improving image representation performance in automatically extracted frames is a major challenge for high-performance video deduplication retrieval.

[0155] This application addresses the problem of efficient video representation and deduplication retrieval by proposing automatic frame extraction based on a deep learning frame extraction model and a video deduplication method based on non-uniform frame extraction. First, the automatic frame extraction method compares two images and outputs the image more conducive to subsequent feature compression retrieval, thus enabling this image to represent similar consecutive frames. Simultaneously, by sharing the underlying network structure of the frame extraction model and the representation model (i.e., the comparison model and the feature extraction model), the computational load of model inference parameters during application is reduced. Furthermore, by alternating the learning of the frame extraction model and the representation model, the frame extraction results generated by the frame extraction model are adapted to the representation model, thereby avoiding significant differences between the actual image representation effect and the model training result caused by inconsistent frame extraction methods. Since automatic frame extraction results in a non-uniform time interval for the final video frame extraction, this application also designs a video deduplication method based on non-uniform frame extraction. By recording the time period represented by each frame's features, the similarity of the embedded features only needs to be calculated once between that time period and the segment to be compared. Therefore, this reduces both storage space and computational load, leading to an overall performance improvement for the system.

[0156] The embodiments of this application address the shortcomings of previous research by designing an innovative and complete process that simultaneously solves these problems: 1) Variable length frame dropping: The frame dropping model is used to achieve non-uniform frame dropping effect for image comparison, which can avoid the problem of repeated features occupying too much storage space.

[0157] 2) A frame extraction model that is consistent with the feature extraction model: By sharing the underlying parameters and alternating learning, the results of the frame extraction are guaranteed to have better measurement performance in the representation model.

[0158] 3) Deduplication method based on variable-length frame extraction: Design a deduplication scheme based on the above variable-length frame extraction to achieve deduplication capability with image sequence consistency.

[0159] This application embodiment represents images through image representation and automatic frame extraction strategy to perform video retrieval. It is used for common video similarity retrieval and ranking, video deduplication retrieval, etc., and can compress the storage space of retrieval features and improve retrieval effect.

[0160] Figure 7 This is a schematic diagram illustrating the implementation process of the frame-skipping model provided in this application. Figure 8 This is a schematic diagram of the structure of the characterization model in the embodiments of this application. Figure 9 This is a schematic diagram of the frame-skipping model provided in an embodiment of this application. For example... Figure 7 As shown, the process of the frame extraction model in the final application of this application is as follows: When the video is added to the database, the comparison process for the input video 701 is as follows: First, the first two frames of the video image 702 are selected for extraction (where the first frame is the target frame and the second frame is the reference frame; subsequent operations require determining whether the target frame needs to be added to the database; the first comparison is between the first and second frames to determine whether the first frame needs to be added to the database, and the second comparison is between the second and third frames to determine whether the second frame needs to be added to the database). The two frames are then embedded with features (using...). Figure 8 The representation model in the model is used, and then the similarity between the two embedded features is used to determine whether a comparison is needed. When a comparison is needed, the model is used... Figure 9 The frame-skipping model (i.e.) Figure 7 The comparison model 703 in the database is used for comparison. After comparison, a better frame is determined. The better frame is then compared with the next frame. When no comparison is needed, the target frame is stored in the database. At this time, the next frame is the target frame. The next frame is then compared with the frame after that.

[0161] It should be noted that, Figure 7 The comparison model 703 in the middle is Figure 9 The frame-slicing model in the video compares two frames at a time for each input video. Figure 8 The representation model in the text is used to extract the image representation of each frame in video retrieval, and the similarity between two images can be calculated based on the image representation. Figure 9The frame-slicing model in the video extracts the embedding features (i.e., image representations) of two images using the same structure. Then, the comparison module 901 compares the embedding features of two images and outputs a comparison score. The higher the score, the better image is considered to be (image 1 or image 2). Here, "better image" refers to the representation model. For example, if two consecutive frames from scenes A and B in a video are selected, the selected image is considered better if its embedding features are closer in similar images (scenes A and B are similar) and farther apart in dissimilar images (scenes A and B are dissimilar). It should be noted that... Figure 9 The Convolutional Neural Network (CNN) module 900 in the frame-slicing model can be reused. Figure 8 The representation model in the CNN module 801 has already been trained.

[0162] In this embodiment of the application, the representation model is as follows: Figure 8 As shown, the CNN module 801 is the basic feature module, which can adopt the convolution module of ResNet-101 (a residual network) and use the parameters pre-trained by ImageNet (a large visualization database for research on visual object recognition software) as pre-training parameters to output embedded features to represent the image.

[0163] In this embodiment, the frame extraction model also needs to extract basic image features. For ease of application, the pre-trained CNN module 801 and embedding module 802 are directly reused. Then, the embedding features of the two images are input into the comparison module 901 to obtain the comparison score. The operation process of the comparison module 901 is as follows: First, the input embedding features pass through a fully connected layer (FC) 9011 (which can be considered a feature mapping module, i.e., a mapping layer, mapping from embedding features to scoring information). The information more suitable for comparison is extracted from the embedding features. Then, the difference between the two embedding features is calculated by the difference layer 9012. The difference is then processed by an FC layer 9013 (which can be considered a scoring module or prediction layer) to obtain the score, i.e., the output result. In the output result, 1 indicates that image 1 is better, and -1 indicates that image 2 is better. The model needs to satisfy the fact that when image 1 and image 2 are swapped, the better frame remains unchanged. The comparison module 901 is directly connected to the output of the embedding module 802, which allows the extraction and comparison of embedded features to be calculated separately. This facilitates the batch extraction of embedded features to speed up the extraction process. At the same time, it flexibly supports pairwise frame comparisons, such as inputting each batch of 32 images into the representation model to obtain 32 embedded features. Then, the preceding and following frames in the 32 embedded features are compared pairwise to determine the features to be added to the database.

[0164] The following describes the representation model for image similarity representation embedding feature extraction, the training process of the comparison model, and the video retrieval method after frame compression in application.

[0165] Table 1 shows the overall network structure of the video retrieval method in this application embodiment. The overall network structure can be based on the ResNet-101 feature embedding structure. Table 2 shows the network structure of the representation model in this application embodiment. The representation model can be based on the ResNet-101 embedding module, and the output 1x128 is used as the similarity embedding feature. Table 3 shows the network structure of the comparison model in this application embodiment. The comparison module is based on the embedded features. The input embedded features are passed through the mapping layer to obtain the intermediate output. The intermediate outputs of the two embedded features are passed through the difference layer to obtain the difference vector. The difference vector is passed through the prediction layer to obtain the frame selection comparison score.

[0166] Table 1

[0167] Table 2

[0168] Table 3

[0169] The network structures in this embodiment are shown in Tables 1, 2, and 3. Image feature extraction can use the feature extraction module of ResNet-101 pre-trained on ImageNet and the output of the pooling layer as the image embedding features, as shown in Table 1. The embedding features are then obtained through pooling layers and the embedding module, which are 1x128 in dimension. The comparison model is shown in Table 3.

[0170] Since the representation model and the alignment model share the same underlying modules, the overall model structure contains two functional modules: a representation module and an alignment module. During training, these modules are trained sequentially. For example, when training the representation model, the output of Table 2 is used for triplet metric learning, and only the network parameters in Tables 1 and 2 are updated; when training the alignment model, the output of Table 3 is used for regression learning, and only the network parameters in Table 3 are updated.

[0171] The training process of the representation model and the comparison model will be explained below.

[0172] In the data preparation phase: First, sample groups are prepared based on video scene segmentation. Scene segmentation here refers to dividing a long video into multiple consecutive shots, with each shot containing approximately identical images. For example, the open-source library pyScenedetect can be used for scene segmentation. After segmentation, four sample images are randomly selected from each scene to form a scene segmentation sample group (a scene segmentation sample group is a positive sample group, where each pair of images can be the target sample and positive sample in a triplet). Figure 10This is a rendering of four images extracted from each of the two scenes in the two videos provided in this application embodiment. The images within each scene sample group form pairs of similar samples.

[0173] In the learning process of the image similarity representation model, the goal of this stage is to pre-train Tables 1 and 2. This part both generates embedded features and serves as a pre-module for the comparison module. If the representation model (Tables 1 and 2) has already been trained, this step can be ignored. 1) Parameter initialization: Except for the parameters pre-trained by ResNet, the other parameters in Tables 1 and 2 above are initialized using a Gaussian distribution with a variance of 0.01 and a mean of 0.

[0174] 2) Data preparation: Prepare image similarity samples: image pairs (i.e., labeled whether two images are the same or similar).

[0175] Data sampling during the learning process includes similarity triplet mining: In this embodiment, triplet learning is used. Two images are randomly selected from the storyboard sample group as similar sample pairs. The similar sample pairs are used as input for triplet learning. In each batch of similar sample pairs (bs), triplets are mined as follows: For a certain sample pair, one image is randomly selected as the target sample. The distance between the remaining bs-1 sample pairs (each pair randomly selects one image) and x is determined. The samples are sorted in ascending order of distance. The top 10 samples are taken as negative samples and formed triplets with the positive sample pairs in x. Therefore, each sample generates 10 triplets, and the whole batch obtains 10*bs triplets.

[0176] 3) Overall training process: For a total of M storyboard sample groups, 2 samples are randomly selected from each group to obtain M sample pairs. Each batch consists of bs sample pairs, for a total of M / bs batches. For each batch, the model is forward computed and the parameters are updated. After all M / bs updates are completed, one training cycle is completed. A total of K training cycles are performed (or training is stopped when the average loss does not decrease for 10 consecutive training cycles).

[0177] 4) Forward pass: Set all parameters of the model to the state to be learned. During training, the neural network performs forward pass calculations on an input image to obtain the prediction result em. Calculate the triplet loss based on the prediction result em. The calculation of the triplet loss is described later.

[0178] 5) Model parameter update process: The triplet loss is used to perform gradient backpropagation to obtain the updated values ​​of all model parameters, and the model is updated accordingly. A learning rate of 0.005 (10 times smaller than FC) is used. After obtaining the gradient from the triplet loss in each learning round, the network weights are updated according to the learning rate.

[0179] During the learning process of the comparison model, the Table 2 module is fine-tuned overall, and the Table 3 comparison module is learned. The goal of this stage is to combine the embedding module and the comparison module so that the results of the comparison module meet the triplet metric requirements while also representing the scene sample to which it belongs.

[0180] 1) Parameter initialization: The parameters in Tables 1 and 2 above are initialized using the results of the first-stage training. The parameters in Table 3 to be learned are initialized using a Gaussian distribution with a variance of 0.01 and a mean of 0.

[0181] 2) Data Preparation. Obtaining the best sample from the sample group: Input each of the above-mentioned storyboard sample groups into the model to obtain 4 embedding features for each sample group. Calculate the embedding feature center and find the sample closest to the center as the best sample xb. The remaining three samples are denoted as x1, x2, and x3. At this point, the training samples and labels are in the form of: (xb, x1, 1), (xb, x2, 1), (xb, x3, 1), (x1, xb, -1), (x2, xb, -1), (x3, xb, -1), a total of 6 labeled samples, corresponding to... Figure 8 The data is represented as (image 1, image 2, label), where label = 1 indicates that image 1 is better, and -1 indicates that image 2 is better. Three similar sample pairs are also generated: (xb, x1), (xb, x2), and (xb, x3). In each similar sample pair, the first sample is the target sample, and the second is the positive sample.

[0182] 3) Overall Training Process: For a total of M / bs batches, each batch (denoted as bs) is trained once (for each batch, the model performs forward computation and updates the embedding layer parameters, then the best sample and label are input into the updated model, and the loss is calculated to update the frame extraction module). For a total of M sample groups, training is performed once for every bs sample groups. 3*bs of similar sample pairs are used to mine triples (10 triples are generated for each sample pair as described above, for a total of 30*bs triples). The average value of the embedding module's metric loss is calculated, and backward computation is performed on the model (the gradient is calculated layer by layer from the output loss layer towards the network input using the average loss). The parameters of the representation module are updated based on the gradient values ​​of the parameters obtained from the backward computation. Then, following the best sample acquisition method described above, a total of 6*bs of labeled samples are obtained, and the best sample and label are input into the updated model to update the frame extraction module.

[0183] Once the full M / bs learning is completed, one batch is finished; a total of Ne training cycles are performed (or training is stopped when the average regression loss does not decrease after 10 consecutive training cycles).

[0184] The loss module of this application embodiment will be described below.

[0185] For triplet loss, after finding triplet pairs (a, p, n) in the batch samples, the triplet loss is calculated based on the em features of these triplet samples. The calculation of triplet loss is as follows, where... Let the threshold be 10. Let be the embedding vector of sample a in the triple (a, p, n); Let be the embedding vector of sample p in the triple (a, p, n); Let n be the embedding vector of sample n in the triple (a, p, n); This represents the L2 distance between two embedding vectors. The goal of triplet loss is to ensure that the distance between the target sample and the negative sample is greater than 10 compared to the distance between the target sample and the positive sample. Triplet loss The calculation is as follows: (1) (1) In a certain batch of training, after calculating the above triplet loss for all triplets, the average of all losses is calculated to obtain the final triplet loss.

[0186] For the regression loss, as shown in the following formula (2), where For the label (-1 or 1). This is the output of the frame selection model. During a batch of training, the average of all regression losses is calculated for all best samples and labels to obtain the final regression loss for frame selection.

[0187] (2) The video retrieval method provided in this application embodiment has the following retrieval process: The first step is the data entry process. An initial database (db) can be established. For a given input video, the first two frames are selected (frame 1 is the target frame, and frame 2 is the reference frame; subsequent operations require determining whether the target frame needs to be entered into the database. The first comparison is between frames 1 and 2 to determine if frame 1 needs to be entered; the second comparison is between frames 2 and 3 to determine if frame 2 needs to be entered). Embedded features are extracted from these two frames, and the Euclidean distance between them is calculated. If the distance is less than a threshold thr1, frame extraction and comparison are performed; otherwise, frame 1 is directly entered into the database without comparison. When comparison is needed, the embedded features of these two frames are used as features for image 1 and image 2, respectively, and input into the comparison module (Table 3) to obtain a comparison score. If the comparison score is greater than 0, image 1 is better, and frame 1 is entered into the database; if the score is less than 0, frame 1 is discarded. The comparison of frames 2 and 3 continues. This process of comparing two frames each time is repeated until all frames are finally entered into the database.

[0188] The above method first parses all frames of the input video to provide data for the subsequent automatic frame extraction step. Alternatively, one frame can be extracted every three frames of the input video. For example, for a 24fps video, eight frames per second are retained, and all extracted frames are then used for automatic frame extraction.

[0189] When recording features in the database, each embedded feature records its corresponding video ID (vid, corresponding to the video identifier mentioned above), the video duration (dur1, i.e., video duration), the embedded feature, the start time of the video frame it represents (time1), and the duration it represents (e_dur1, corresponding to the video frame duration mentioned above). Since each frame's embedded feature represents a period of time, when using the original interval frame extraction scheme (e.g., extracting one frame every 5 frames for 25fps) or using all frames without automatic frame selection, then time1 is the time of the current frame, and e_dur1 is 1.

[0190] In this embodiment, the deduplication retrieval process includes the following steps: S1, query video frame extraction: For the query video (i.e., the video to be retrieved), following the above-mentioned storage steps, the video is first automatically extracted, and the frame information after automatic frame extraction is recorded, such as vid, dur, embedding, time, e_dur. S2, the query video and the inventory video are paired to form a video pair to be retrieved. S3, time period matching is performed on each video pair based on the automatically extracted frames. S4, the videos that match the time period are sorted from largest to smallest according to the matching duration (the duration of the query video matching the inventory).

[0191] The following example illustrates the video time segment matching process based on automatic frame extraction in S3.

[0192] For a retrieved video pair (i, r), where i represents the query video and r represents the reference video (in-stock video), the time period matching algorithm steps for query video i are as follows: 1) The preset distance threshold t0 for embedded features is 0.5 (when the Euclidean distance between two embedded features is less than 0.5, it means that the two embedded features come from similar frames). 2) Automatic frame extraction is performed on the two videos in the video pair to obtain frame information such as vid, dur, embedding features, time, and e_dur. Each subsequent frame refers to the frames selected by automatic frame extraction and their related frame information.

[0193] 3) For each frame j in video i (the frame selected by the automatic frame extraction method): calculate the Euclidean distance between it and the embedded features of each frame in video r, take the frames less than t0 as similar frames of j, obtain the list of similar frames (or matching frames) of j sim-id-list (record the frame number of j and the number of its similar frames, such as ([1], [4, 5, 6]) indicating that the first frame is similar to the fourth, fifth, and sixth frames), and at the same time record the corresponding similar frame time deviation list diff-time-list (at this time, the time deviation is determined according to the overall start and end time represented by these frames). The starting time is calculated, and its offset from the starting time of frame j is calculated as the time deviation. For example, for a frame j=1 representing the first 1 to 2 seconds of time, the similar frames in video r are frames 4, 5, and 6. The start and end times of these 3 frames are recorded as 6 to 7 seconds, 8 to 11 seconds, and 11 to 13 seconds, respectively. This means that the first 1 to 2 seconds of video i are similar to the 6 to 13 seconds of video r, and the time deviation is 6-1=5. The frame matching duration is recorded. For example, in this case, (2, 8) means that the 2 seconds represented by the first frame of i match the 8 seconds represented by the 4th to 6th frames of r. Therefore, the list of similar frames SL, the time deviation table TL, and the matching duration MTL of all frames of i are obtained.

[0194] 4) Iterate through all frames and count the number of matching frames between videos i and r. If the number of matching frames is less than 1, then videos i and r do not have the same video segment. Otherwise, proceed to the next step.

[0195] 5) Reordering dt to obtain the SL list: Sort all matching frames in SL by diff-time (i.e., dt) from smallest to largest (when dt is the same, sort by the video i index from smallest to largest), and reorganize the corresponding diff-time list in this order, that is, the time difference of 0 is placed at the first, the time difference of 1 is placed after it, and so on. For example, the new SL list is [([3,4,5,6,7],[2,3]),([1],[4,5,6]),([8,9],[7,8])] to record similar Frame matching segments: If there are 3 matching segments here, it means that frames 3-7 of i match frames 2-3 of r with time ranges of 4-8 seconds (each frame in [3, 4, 5, 6, 7] represents 1 second) and match frames 2-5 seconds (each frame in [2, 3] represents 2 seconds); frame 1 matches frames 4-6 of r with assumed time ranges of 1-2 seconds and 6-13 seconds; frames 8-9 match frames 7-8 with time ranges of 9-10 seconds (each frame represents 1 second) and match frames 14-15 seconds (each frame represents 1 second). The time difference list corresponding to SL is [2, 5, 5].

[0196] 6) Reorganize the data using dt to obtain match-dt-list: For all frames of i, reorganize the list in the similar frame list SL using the time deviation as the primary key to obtain a list of dt in ascending order, resulting in the match-dt-list of similar frames with time deviations of 0, 1, 2, ..., where each element is [dt:{frame-count, time-count, start-id, match-id-list, start-time, match-time-list}], such as [0:{xx}], [1:{xx}, 1:{xx}, 1:{xx}] For example, 【2:{5, 5, 3, ([3, 4, 5, 6, 7], [2, 3]), 4, ([4, 5, 6, 7, 8], [2, 3, 4, 5])}】, 【5:{1, 2, 1, ([1], [4, 5, 6]), 1, ([1, 2], [6, 7, 8, 9, 10, 11, 12, 13])}, 5:{2, 2, 8, ([8, 9], [7, 8]), 9, ([9, 10], [14, 15])}】, where in the first set of data, 2 refers to the time difference of 2, 5 refers to the query video having 5 frames that match video r, the first matching frame number is 3, and the first matching frame time is 4 seconds. Frame-count is the number of similar frames under this time difference. If frames 3 to 7 of i match, there are a total of 5 frames. time-count refers to the duration of the match. If frames 3 to 7 of i represent a time of 4 to 8 seconds, the duration is 5. start-id refers to the smallest frame ID of i under this time difference.

[0197] 7) Calculate the time period matching list: Merge multiple matching segments under the same time period (dt) according to a certain strategy. The time interval between two matching segments is preset to be greater than T2 (e.g., 10 seconds, meaning that if the difference between the start times of two matching segments is less than T2, the two segments are merged. This value is preset and can be changed according to different videos). Each dt in match-dt-list (e.g., dt=2) is processed as follows: when it is the first time interval of a certain dt, it is added to the temporary list tmplist. For example, when dt-2 in this case, tmplist=[[4,5,6,7,8],[2,3,4,5]].

[0198] When a segment is not the first matching segment of a certain dt, it is determined whether the difference between the start time srcT and the previous start time srcT is less than T2 (e.g., for the second matching segment with dt=5, compared to tmplist with srcT=1, the second matching segment has srcT=9, which is 8, so it is less than the interval threshold). If it is less, the similar frame matching segments in tmplist are merged with the current matching segment into one matching segment. If it is greater than T2, tmplist is stored in match-duration-list. The method for merging matching segments is as follows: For example, if dt=5, srcT=1 and srcT=9 are merged, that is, the current [[1,2], [6,7,8,9,10,11,12,13]] is merged with the previous tmplist=[9,10], [14,15] and tmplist is updated to get tmplist=[[1,2,9,10], [6,7,8,9,10,11,12,13,14,15]]. like Figure 11 The diagram shown is a schematic diagram of the matching segment information provided in an embodiment of this application.

[0199] When the current segment is the last matching segment of the current duration (e.g., srcT=4), the accumulated similar matching segments in tmplist are added to match-duration-list. If dt=2 has only one matching segment, it is stored in match-duration-list, such as [4, 8, 2, 5, 2, 4]. The values ​​represent [src-startTime, src-endTime, ref-startTime, ref-endTime, dt, duration], which means the matching segment stores the starting frame and ending frame of video i, the starting frame and ending frame of the matching video, the duration of the matching segment, and the duration of the matching segment. Simultaneously, the data in tmplist is cleared. When the difference between a matching segment's srcT and the previous srcT is greater than T2, a temporary list is added to match-duration-list.

[0200] Ultimately, the match-duration-list contains one matching segment with dt=2 and one matching segment with dt=5, and includes [[4, 5, 6, 7, 8], [2, 3, 4, 5]] and [[1, 2, 9, 10], [6, 7, 8, 9, 10, 11, 12, 13, 14, 15]].

[0201] 8) Handling overlapping time periods in the match-duration-list. Since similar frame calculation involves traversing all frames of two videos, calculating the distance, and selecting similar frames within a certain threshold range, a single frame may be similar to multiple frames. This can lead to overlapping time periods between two matched frames in the match-duration-list, which needs to be addressed.

[0202] In this embodiment of the application, a minimum matching segment duration T3 can be set (e.g., Figures 12A to 12C The diagram shown illustrates the processing of three intersection scenarios. Matches with a duration of 6 seconds or less can be merged if they meet the following requirements: (i.e., if they contain each other, delete them; if j is later, adjust the starting point of j to be non-intersecting; if j is earlier, adjust the ending point of j to be non-intersecting). For time interval i in the match-duration-list (the time interval consisting of the starting time src-startTime and the ending time src-endTime), for time interval j in the match-duration-list, = i + 1. If time interval i contains time interval j, delete j; if i and j intersect, and i's starting point is the earliest starting point, move j's starting point to the ending point of i, update j, and if the duration of time interval j is less than T3, delete j; otherwise, replace the old j with the new j; if i and j intersect, and j's starting point is the earliest starting point, move j's ending point to the starting point of i, update j, and if the duration of time interval j is less than T3, delete j; otherwise, replace the old j with the new j.

[0203] Finally, if an inclusion relationship exists and the threshold requirement is met (the included segment's matching duration is less than T3), the included segment is deleted according to the above steps. If an inclusion relationship exists but the threshold requirement is not met, the included segment is merged into the including segment, and the included segment is deleted. A timing scrambling marker is also added to indicate that the two videos are duplicated and that timing scrambling exists. This results in [[1, 2, 4, 5, 6, 7, 8, 9, 10], [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]].

[0204] 9) Return the matching time period information, such as match-duration-list=

[1,2,4,5,6,7,8,9,10],[2,3,4,5,6,7,8,9,10,11,12,13,14,15]

[0205] The video retrieval method provided in this application has at least the following technical effects: By using variable-length frame extraction, the problem of duplicate features occupying excessive storage space can be avoided. The frame extraction model, which is consistent with the feature extraction model, has better measurement performance in the representation model. Based on the variable-length frame extraction video temporal deduplication method, the above-mentioned automatic frame selection model can be applied in business operations, providing a more flexible retrieval method.

[0206] It is understood that in the embodiments of this application, if the content involving user information, such as the video to be retrieved, the matching results of video pairs, and the duplicate videos of the video to be retrieved, involves data related to user information or enterprise information, user permission or consent is required when the embodiments of this application are applied to specific products or technologies, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0207] The following continues to describe the exemplary structure of the video retrieval device 354 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the video retrieval device 354 includes: a feature extraction module 3541, used to extract embedding features from the video to be retrieved, obtaining the embedding features of each video frame in the video to be retrieved; a video frame extraction module 3542, used to extract video frames from all video frames in the video to be retrieved based on the embedding features of every two adjacent video frames, obtaining at least one extracted video frame; a first determination module 3543, used to determine the frame information of the extracted video frame based on the embedding features of each extracted video frame and the attribute parameters of the extracted video frame in the video to be retrieved; and a video pair structure. A construction module 3544 is used to construct a video pair between the video to be retrieved and each stock video in a preset video library; wherein, the stock video includes at least one stock video frame, and each stock video frame corresponds to a stock frame information; a matching module 3545 is used to perform video frame time segment matching on each video pair based on the frame information of the extracted video frame in the video to be retrieved and the stock frame information of the stock video frames in each stock video to obtain a matching result; a second determination module 3546 is used to determine the duplicate videos of the video to be retrieved from the preset video library based on the matching result.

[0208] In some embodiments, the feature extraction module 3541 is further configured to: input the video to be retrieved into a feature extraction model; sequentially extract features from different levels of each video frame through at least two convolutional layers of the feature extraction model to obtain convolutional features for each video frame; perform max pooling on the convolutional features through a pooling layer of the feature extraction model to obtain pooled convolutional features for each video frame; and perform feature combination on the pooled convolutional features through a fully connected layer of the feature extraction model to obtain the embedded features for each video frame.

[0209] In some embodiments, the feature extraction model is trained through the following steps: segmenting the sample video to obtain multiple segmentation sample groups; wherein each segmentation sample group includes at least two sample images, and every two sample images in each segmentation sample group form a similar sample pair; based on the similar sample pairs in each segmentation sample group, determining multiple triples from the multiple segmentation sample groups; randomly selecting a similar sample pair from each segmentation sample group as the input sample of the feature extraction model, and inputting the input sample of the feature extraction model into the feature extraction model; and sequentially performing feature extraction on different levels of features of each sample image in the input sample through at least two convolutional layers of the feature extraction model. Feature extraction is performed to obtain the convolutional features of each sample image. Max pooling is then applied to these convolutional features through the pooling layer of the feature extraction model to obtain pooled convolutional features for each sample image. Feature combination is then performed on the pooled convolutional features through the fully connected layer of the feature extraction model to obtain the embedding features for each sample image. These embedding features are then used as the prediction result of the feature extraction model and input into a triplet loss model. The triplet loss model calculates the loss based on the multiple triplets used in the prediction result of the feature extraction model to obtain the triplet loss. Based on the triplet loss, the parameters in the feature extraction model are corrected to obtain the trained feature extraction model.

[0210] In some embodiments, the feature extraction model is trained through the following steps: randomly selecting a similar sample pair from each of the multiple storyboard sample groups to obtain multiple similar sample pairs with the same number of groups as the multiple storyboard sample groups; dividing the multiple similar sample pairs into equal batches to obtain multiple batches of similar sample pairs; wherein each batch of similar sample pairs has a first number of similar sample pairs; from each batch of similar sample pairs, determining any one of the similar sample pairs as the target sample pair, and determining the other similar sample pairs in the batch of similar sample pairs besides the target sample pair as the remaining sample pairs; determining the image distance between any sample image in the target sample pair and any sample image in each of the remaining sample pairs; sorting the remaining sample pairs in descending order of the image distance to form a sequence of remaining sample pairs; sequentially selecting a second number of sample images in the order of the remaining sample pairs in the sequence of remaining sample pairs to form a set of negative sample images; constructing a triplet by combining two sample images from the target sample pair and one sample image from the set of negative sample images.

[0211] In some embodiments, the video frame extraction module 3542 is further configured to: determine the similarity between the embedding features of two adjacent video frames; when the similarity is greater than a similarity threshold, determine the comparison score of the two video frames based on the embedding features of the two video frames; and determine the video frame with the higher comparison score as the extracted video frame among the two video frames.

[0212] In some embodiments, each video frame corresponds to a timestamp in the video to be retrieved; the video frame extraction module is further configured to: for two adjacent video frames, when the similarity is less than or equal to the similarity threshold, determine the video frame with the smaller timestamp among the two video frames as the extracted video frame; when any video frame is determined as the extracted video frame, continue to extract video frames from two adjacent video frames that have timestamps greater than the timestamp of the extracted video frame to obtain the next extracted video frame.

[0213] In some embodiments, the video frame extraction module 3542 is further configured to: input the embedded features of the two video frames into a comparison model; perform feature mapping on the embedded features through the mapping layer of the comparison model to obtain key features in the embedded features of each video frame in the two video frames; determine the difference vector of the key features corresponding to the two video frames through the difference layer of the comparison model; and perform difference scoring on the two video frames based on the difference vector through the prediction layer of the comparison model to obtain the comparison score of the two video frames.

[0214] In some embodiments, the comparison model is trained through the following steps: segmenting the sample video to obtain multiple segmentation sample groups; wherein each segmentation sample group includes at least two sample images, and each pair of sample images in each segmentation sample group forms a similar sample pair; inputting each sample image in each segmentation sample group into a feature extraction model to obtain at least two image embedding features; determining the optimal sample corresponding to the segmentation sample group based on the at least two image embedding features; and generating a label sample based on the optimal sample; using the image embedding features of each similar sample pair in each segmentation sample group as the input sample of the comparison model, and inputting the input sample of the comparison model into the comparison model; and through the mapping of the comparison model... The model performs feature mapping on the input samples to obtain the key features of the samples in the image embedding features of each sample image in the similar sample pair; through the difference layer of the comparison model, the sample difference vector of the key features of the samples corresponding to the two sample images is determined; through the prediction layer of the comparison model, the difference between the two sample images is scored based on the sample difference vector to obtain the sample comparison score of the two sample images; the sample comparison score is used as the prediction result of the comparison model and input into the regression loss model; through the regression loss model, based on the best sample and the label sample, the loss of the prediction result of the comparison model is calculated to obtain the regression loss; based on the regression loss, the parameters in the comparison model are corrected to obtain the trained comparison model.

[0215] In some embodiments, the comparison model is trained by the following steps: determining the feature centers of the at least two image embedding features; determining the distance between each image embedding feature and the feature center; and determining the sample image corresponding to the image embedding feature with the smallest distance as the best sample.

[0216] In some embodiments, the first determining module 3543 is further configured to: obtain the video identifier and video duration of the video to be retrieved from the attribute parameters of the video to be retrieved; determine the start time corresponding to the extracted video frame in the video to be retrieved, and the video frame duration represented by the extracted video frame in the video to be retrieved; wherein, the video frame duration refers to the duration of a video segment from the extracted video frame to the next adjacent extracted video frame; and determine the video identifier, the video duration, the embedding feature, the start time, and the video frame duration as the frame information of the extracted video frame.

[0217] In some embodiments, the matching module 3545 is further configured to: acquire the embedding features of each extracted video frame in the video to be retrieved and the embedding features of each stock video frame in the stock video; based on the embedding features of the extracted video frames and the stock video frames, determine similar video frames of each extracted video frame in the video to be retrieved from at least one stock video frame in the stock video; determine a first start time and a first video frame duration corresponding to each extracted video frame in the video to be retrieved, and a second start time and a second video frame duration corresponding to each similar video frame in the stock video; when the first start time of any extracted video frame in the video to be retrieved is similar to the stock video frame, the matching module 3545 is configured to: acquire the embedding features of each extracted video frame in the video to be retrieved and the embedding features of each ... If the second start time of any similar video frame in the video is matched, and the duration of the first video frame of the extracted video frame matches the duration of the second video frame of the similar video frame, the similar video frame is determined as the matching video frame of the extracted video frame; based on the duration of the second video frame of the matching video frame corresponding to each extracted video frame in the video to be retrieved, the matching duration between the video to be retrieved and the inventory video is determined; the matching duration is determined as the matching result; the second determining module is further configured to: determine the inventory video with the maximum matching duration, or the inventory video with the matching duration greater than the matching duration threshold, from the preset video library as the duplicate video of the video to be retrieved.

[0218] In some embodiments, the feature extraction module 3541 is further configured to perform embedding feature extraction on the inventory videos in each video pair to obtain the inventory embedding feature of each video frame in the inventory videos; the video frame extraction module 3542 is further configured to sequentially extract video frames from all video frames in the inventory videos based on the inventory embedding features of every two adjacent video frames in the inventory videos to obtain at least one inventory video frame; the first determining module 3543 is further configured to determine the inventory frame information of the inventory video frame based on the inventory embedding feature of each inventory video frame and the attribute parameters of the inventory video frame in the inventory videos.

[0219] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0220] This application provides a computer program product comprising a computer program or executable instructions, which are computer instructions; the computer program or executable instructions are stored in a computer-readable storage medium. When the processor of a video retrieval device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, the video retrieval device performs the method described in this application embodiment.

[0221] This application provides a storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example... Figure 3 The method shown.

[0222] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0223] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0224] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file containing other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single video retrieval device, or on multiple video retrieval devices located in one location, or on multiple video retrieval devices distributed across multiple locations and interconnected via a communication network.

[0225] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A video retrieval method, characterized in that, The method includes: Embedding features are extracted from the video to be retrieved to obtain the embedding features of each video frame in the video to be retrieved; For two adjacent video frames, determine the similarity between the embedded features of the two video frames; When the similarity is greater than the similarity threshold, the comparison score of the two video frames is determined based on the embedding features of the two video frames. Of the two video frames, the one with the higher comparison score is selected as the extracted video frame. Based on the embedding features of each extracted video frame and the attribute parameters of the extracted video frame in the video to be retrieved, the frame information of the extracted video frame is determined. The video to be retrieved is constructed into a video pair with each of the stored videos in the preset video library; wherein, the stored video includes at least one stored video frame, and each stored video frame corresponds to a stored frame information. Based on the frame information of the extracted video frames in the video to be retrieved and the inventory frame information of the inventory video frames in each inventory video, video frame time period matching is performed on each video pair to obtain the matching result. Based on the matching results, duplicate videos of the video to be retrieved are determined from the preset video library.

2. The method according to claim 1, characterized in that, The step of extracting embedding features from the video to be retrieved, to obtain the embedding features of each video frame in the video to be retrieved, includes: The video to be retrieved is input into the feature extraction model. The feature extraction model extracts features from different levels of each video frame sequentially through at least two convolutional layers, thereby obtaining the convolutional features of each video frame. The convolutional features are subjected to max pooling through the pooling layer of the feature extraction model to obtain the pooled convolutional features for each video frame. The pooled convolutional features are combined using the fully connected layer of the feature extraction model to obtain the embedded features for each video frame.

3. The method according to claim 2, characterized in that, The feature extraction model is trained through the following steps: The sample video is processed into multiple segmentation sample groups; each segmentation sample group includes at least two sample images, and every two sample images in each segmentation sample group form a similar sample pair. Based on similar sample pairs in each of the multiple shot sample groups, multiple triplet groups are determined from the multiple shot sample groups; Randomly select a similar sample pair from each of the storyboard sample groups as the input sample of the feature extraction model, and input the input sample of the feature extraction model into the feature extraction model; Through at least two convolutional layers of the feature extraction model, features at different levels of each sample image in the input sample are extracted sequentially to obtain the sample convolutional features of each sample image. The pooling layer of the feature extraction model performs max pooling on the sample convolutional features to obtain the pooled sample convolutional features for each sample image. The pooled sample convolution features are combined using the fully connected layer of the feature extraction model to obtain the sample embedding features for each sample image. The sample embedding features are used as the prediction result of the feature extraction model and input into the triplet loss model. The triplet loss model calculates the loss based on the prediction result of the feature extraction model using the multiple triplets to obtain the triplet loss. Based on the triplet loss, the parameters in the feature extraction model are corrected to obtain the trained feature extraction model.

4. The method according to claim 3, characterized in that, The step of determining multiple triplets from the multiple segmentation sample groups based on similar sample pairs in each of the segmentation sample groups includes: Randomly select a similar sample pair from each of the storyboard sample groups to obtain multiple similar sample pairs with the same number of groups as the multiple storyboard sample groups; The multiple similar sample pairs are divided into equal batches to obtain multiple batches of similar sample pairs; wherein each batch of similar sample pairs contains a first number of similar sample pairs. From each batch of similar sample pairs, any one of the similar sample pairs is determined as the target sample pair, and the other similar sample pairs in the batch other than the target sample pair are determined as the remaining sample pairs; Determine the image distance between any sample image in the target sample pair and any sample image in each of the remaining sample pairs; The remaining sample pairs are sorted in descending order of image distance to form a sequence of remaining sample pairs; According to the order of the remaining sample pairs in the remaining sample pair sequence, the second number of sample images are selected in sequence to form a negative sample image set; The two sample images from the target sample pair and one sample image from the negative sample image set are used to construct a triplet.

5. The method according to claim 1, characterized in that, Each video frame corresponds to a timestamp in the video to be retrieved, and the method further includes: For two adjacent video frames, when the similarity is less than or equal to the similarity threshold, the video frame with the smaller timestamp among the two video frames is determined as the extracted video frame. When any video frame is identified as the extracted video frame, video frame extraction continues for the two video frames adjacent to the extracted video frame and whose timestamps are greater than the timestamps of the extracted video frame, to obtain the next extracted video frame.

6. The method according to claim 1, characterized in that, The step of determining the comparison score of the two video frames based on their embedding features includes: The embedding features of the two video frames are input into the comparison model. The mapping layer of the comparison model performs feature mapping on the embedding features to obtain the key features in the embedding features of each video frame in the two video frames. The difference vector of the key features corresponding to the two video frames is determined through the difference layer of the comparison model. The comparison model's prediction layer scores the two video frames based on the difference vector to obtain their comparison scores.

7. The method according to claim 6, characterized in that, The comparison model is trained through the following steps: The sample video is processed into multiple segmentation sample groups; each segmentation sample group includes at least two sample images, and every two sample images in each segmentation sample group form a similar sample pair. Each sample image in each of the aforementioned storyboard sample groups is input into the feature extraction model to obtain at least two image embedding features. Based on the at least two image embedding features, determine the optimal sample corresponding to the storyboard sample group; and generate label samples based on the optimal sample. The image embedding features of each similar sample pair in each of the segmentation sample groups are used as the input samples of the comparison model, and the input samples of the comparison model are input into the comparison model. The input samples are feature-mapped by the mapping layer of the comparison model to obtain the key sample features in the image embedding features of each sample image in the similar sample pair; The difference vector of the key features of the two sample images is determined by the difference layer of the comparison model. The comparison model uses a prediction layer to score the difference between the two sample images based on the sample difference vector, thus obtaining the sample comparison score of the two sample images. The sample comparison score is used as the comparison model prediction result and input into the regression loss model. The regression loss model calculates the loss based on the best sample and the label sample, and obtains the regression loss. Based on the regression loss, the parameters in the comparison model are corrected to obtain the trained comparison model.

8. The method according to claim 7, characterized in that, The step of determining the optimal sample corresponding to the segmentation sample group based on the at least two image embedding features includes: Determine the feature centers of the at least two image embedding features; Determine the distance between each of the image embedding features and the feature center; The sample image corresponding to the embedding feature of the image with the smallest distance is determined as the best sample.

9. The method according to claim 1, characterized in that, The step of determining the frame information of the extracted video frame based on the embedding features of each extracted video frame and the attribute parameters of the extracted video frame in the video to be retrieved includes: From the attribute parameters of the video to be retrieved, obtain the video identifier and the video duration of the video to be retrieved; Determine the start time corresponding to the extracted video frame in the video to be retrieved, and the duration of the video frame represented by the extracted video frame in the video to be retrieved; wherein, the duration of the video frame refers to the duration of the video segment from the extracted video frame to the next adjacent extracted video frame; The video identifier, the video duration, the embedding feature, the start time, and the video frame duration are determined as the frame information of the extracted video frame.

10. The method according to claim 9, characterized in that, The process involves matching video frames across time periods for each video pair based on the frame information of the extracted video frames from the video to be retrieved and the inventory frame information of the inventory video frames in each inventory video, to obtain matching results, including: Obtain the embedding features of each extracted video frame in the video to be retrieved and the embedding features of each stock video frame in the stock videos; Based on the embedding features of the extracted video frames and the embedding features of the stock video frames, similar video frames are determined from at least one stock video frame in the stock video for each extracted video frame in the video to be retrieved. Determine the first start time and first video frame duration for each extracted video frame in the video to be retrieved, and the second start time and second video frame duration for each similar video frame in the inventory videos; When the first start time of any extracted video frame in the video to be retrieved matches the second start time of any similar video frame in the inventory video, and the duration of the first video frame of the extracted video frame matches the duration of the second video frame of the similar video frame, the similar video frame is determined as the matching video frame of the extracted video frame. Based on the duration of the second video frame corresponding to the matching video frame of each extracted video frame in the video to be retrieved, the matching duration between the video to be retrieved and the inventory video is determined. The matching duration is determined as the matching result; The step of determining duplicate videos of the video to be retrieved from the preset video library based on the matching result includes: From the preset video library, the inventory videos with the longest matching duration, or the inventory videos with a matching duration greater than the matching duration threshold, are identified as duplicate videos of the video to be retrieved.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: Embedding features are extracted from the stock videos in each video pair to obtain the stock embedding features of each video frame in the stock videos; Based on the inventory embedding features of each two adjacent video frames in the inventory video, video frame extraction is performed on all video frames in the inventory video to obtain at least one inventory video frame. Based on the inventory embedding features of each inventory video frame and the attribute parameters of the inventory video frame in the inventory video, the inventory frame information of the inventory video frame is determined.

12. A video retrieval device, characterized in that, The device includes: The feature extraction module is used to extract embedding features from the video to be retrieved, and obtain the embedding features of each video frame in the video to be retrieved. The video frame extraction module is used to determine the similarity between the embedding features of two adjacent video frames; when the similarity is greater than a similarity threshold, the module determines the comparison score of the two video frames based on the embedding features; and in the two video frames, the video frame with the higher comparison score is determined as the extracted video frame. The first determining module is used to determine the frame information of the extracted video frame based on the embedding features of each extracted video frame and the attribute parameters of the extracted video frame in the video to be retrieved. The video pair construction module is used to construct a video pair by combining the video to be retrieved with each stock video in the preset video library; wherein, the stock video includes at least one stock video frame, and each stock video frame corresponds to a stock frame information. The matching module is used to perform video frame time segment matching on each video pair based on the frame information of the extracted video frames in the video to be retrieved and the inventory frame information of the inventory video frames in each inventory video, so as to obtain the matching result. The second determining module is used to determine, based on the matching result, duplicate videos of the video to be retrieved from the preset video library.

13. The apparatus according to claim 12, characterized in that, The feature extraction module is further configured to input the video to be retrieved into the feature extraction model, extract features from different levels of each video frame sequentially through at least two convolutional layers of the feature extraction model to obtain convolutional features for each video frame; perform max pooling on the convolutional features through the pooling layer of the feature extraction model to obtain pooled convolutional features for each video frame; and combine the pooled convolutional features through the fully connected layer of the feature extraction model to obtain the embedded features for each video frame.

14. The apparatus according to claim 13, characterized in that, The feature extraction model is trained through the following steps: The sample video is segmented to obtain multiple segmentation sample groups; each segmentation sample group includes at least two sample images, and every two sample images in each segmentation sample group form a similar sample pair; based on the similar sample pairs in each segmentation sample group, multiple triples are determined from the multiple segmentation sample groups; a similar sample pair is randomly selected from each segmentation sample group as the input sample of the feature extraction model, and the input sample of the feature extraction model is input into the feature extraction model; through at least two convolutional layers of the feature extraction model, features at different levels of each sample image in the input sample are extracted sequentially. The convolutional features of each sample image are obtained. Max pooling is performed on these convolutional features through the pooling layer of the feature extraction model to obtain pooled convolutional features for each sample image. Feature combination is performed on the pooled convolutional features through the fully connected layer of the feature extraction model to obtain sample embedding features for each sample image. These sample embedding features are then input as the prediction result of the feature extraction model into a triplet loss model. The triplet loss model calculates the loss based on the multiple triplets used to predict the feature extraction model's prediction results, yielding a triplet loss. Based on the triplet loss, the parameters in the feature extraction model are corrected to obtain the trained feature extraction model.

15. The apparatus according to claim 14, characterized in that, The feature extraction model is trained by the following steps: randomly selecting a similar sample pair from each of the multiple storyboard sample groups to obtain multiple similar sample pairs with the same number of groups as the multiple storyboard sample groups; The multiple similar sample pairs are divided into equal batches to obtain multiple batches of similar sample pairs; wherein, each batch of similar sample pairs has a first number of similar sample pairs; from each batch of similar sample pairs, any one of the similar sample pairs is determined as the target sample pair, and the other similar sample pairs in the batch other than the target sample pair are determined as the remaining sample pairs; the image distance between any sample image in the target sample pair and any sample image in each of the remaining sample pairs is determined; the remaining sample pairs are sorted in descending order of the image distance to form a sequence of remaining sample pairs; according to the arrangement order of the remaining sample pairs in the sequence of remaining sample pairs, a second number of sample images are selected in sequence to form a set of negative sample images; two sample images in the target sample pair and one sample image in the set of negative sample images are used to construct a triplet.

16. The apparatus according to claim 12, characterized in that, Each video frame corresponds to a timestamp in the video to be retrieved; The video frame extraction module is further configured to, for two adjacent video frames, when the similarity is less than or equal to the similarity threshold, determine the video frame with the smaller timestamp among the two video frames as the extracted video frame; when any video frame is determined as the extracted video frame, continue to extract video frames from the two video frames adjacent to the extracted video frame and whose timestamps are greater than the timestamps of the extracted video frame to obtain the next extracted video frame.

17. The apparatus according to claim 12, characterized in that, The video frame extraction module is further configured to input the embedded features of the two video frames into the comparison model, perform feature mapping on the embedded features through the mapping layer of the comparison model to obtain the key features in the embedded features of each video frame in the two video frames; determine the difference vector of the key features corresponding to the two video frames through the difference layer of the comparison model; and perform difference scoring on the two video frames based on the difference vector through the prediction layer of the comparison model to obtain the comparison score of the two video frames.

18. The apparatus according to claim 17, characterized in that, The comparison model is trained through the following steps: The sample video is segmented to obtain multiple segmentation sample groups; each segmentation sample group includes at least two sample images, and every two sample images in each segmentation sample group form a similar sample pair; each sample image in each segmentation sample group is input into a feature extraction model to obtain at least two image embedding features; based on the at least two image embedding features, the optimal sample corresponding to the segmentation sample group is determined; and a label sample is generated based on the optimal sample; the image embedding features of each similar sample pair in each segmentation sample group are used as input samples for the comparison model, and the input samples of the comparison model are input into the comparison model; the comparison model is then processed through a mapping layer. Input samples are feature-mapped to obtain the key features of each sample image in the image embedding features of the similar sample pair; through the difference layer of the comparison model, the sample difference vector of the key features of the two sample images is determined; through the prediction layer of the comparison model, the two sample images are scored based on the sample difference vector to obtain the sample comparison score of the two sample images; the sample comparison score is used as the prediction result of the comparison model and input into the regression loss model; through the regression loss model, based on the best sample and the label sample, the loss of the prediction result of the comparison model is calculated to obtain the regression loss; based on the regression loss, the parameters in the comparison model are corrected to obtain the trained comparison model.

19. The apparatus according to claim 18, characterized in that, The comparison model is trained through the following steps: determining the feature centers of the at least two image embedding features; determining the distance between each image embedding feature and the feature center; and determining the sample image corresponding to the image embedding feature with the smallest distance as the best sample.

20. The apparatus according to claim 12, characterized in that, The first determining module is further configured to obtain the video identifier and video duration of the video to be retrieved from the attribute parameters of the video to be retrieved; determine the start time corresponding to the extracted video frame in the video to be retrieved, and the video frame duration represented by the extracted video frame in the video to be retrieved; wherein, the video frame duration refers to the duration of the video segment from the extracted video frame to the next adjacent extracted video frame; and determine the video identifier, the video duration, the embedded feature, the start time, and the video frame duration as the frame information of the extracted video frame.

21. The apparatus according to claim 20, characterized in that, The matching module is also used to obtain the embedding features of each extracted video frame in the video to be retrieved and the embedding features of each stock video frame in the stock videos. Based on the embedding features of the extracted video frames and the embedding features of the stock video frames, similar video frames are determined from at least one stock video frame in the stock video for each extracted video frame in the video to be retrieved. The process involves determining a first start time and a first video frame duration for each extracted video frame in the video to be retrieved, and a second start time and a second video frame duration for each similar video frame in the inventory videos. When the first start time of any extracted video frame in the video to be retrieved matches the second start time of any similar video frame in the inventory videos, and the first video frame duration of the extracted video frame matches the second video frame duration of the similar video frame, the similar video frame is identified as the matching video frame of the extracted video frame. Based on the second video frame duration of the matching video frame corresponding to each extracted video frame in the video to be retrieved, the matching duration between the video to be retrieved and the inventory videos is determined. The matching duration is determined as the matching result; The second determining module is further configured to: determine, from the preset video library, the inventory video with the longest matching duration, or the inventory video with a matching duration greater than the matching duration threshold, as a duplicate video of the video to be retrieved.

22. The apparatus according to any one of claims 12 to 21, characterized in that, The feature extraction module is also used to extract the embedding features of the inventory videos in each video pair to obtain the inventory embedding features of each video frame in the inventory videos. The video frame extraction module is further configured to extract video frames from all video frames in the inventory video based on the inventory embedding features of each two adjacent video frames in the inventory video, thereby obtaining at least one inventory video frame. The first determining module is further configured to determine the inventory frame information of the inventory video frame based on the inventory embedding features of each of the inventory video frames and the attribute parameters of the inventory video frame in the inventory video.

23. A video retrieval device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the video retrieval method according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that, The device stores executable instructions for causing a processor to execute the executable instructions to implement the video retrieval method according to any one of claims 1 to 11.

25. A computer program product, characterized in that, It includes a computer program or executable instructions, which, when executed by a processor, implement the video retrieval method according to any one of claims 1 to 11.

Images