Video matching processing method and device, computer device, storage medium and product

Abstract

The application provides a video matching processing method and device, computer equipment, a storage medium and a product. The method comprises: performing global feature extraction processing on each first video frame obtained from a to-be-matched video to obtain global video image features of the to-be-matched video, and performing local feature extraction processing on each second video frame in the to-be-matched video to obtain local video image features of the to-be-matched video; determining an initial recall video set associated with the to-be-matched video based on the global video image features of the to-be-matched video; and determining a matching recall video matched with the to-be-matched video from the initial recall video set according to the local video image features of the to-be-matched video. The application can perform video matching processing based on multi-fine-grained image features, and improves the accuracy of the video matching process.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video matching processing method, a video matching processing apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the continuous development of internet technology, all kinds of videos are emerging in various fields. Therefore, video plagiarism detection technology has become an important research topic in various fields.

[0003] Currently, the most common method for video plagiarism detection is manual matching, where the video to be matched is manually compared with various videos in a video library. This manual matching method is cumbersome and inefficient. While there are solutions that use artificial intelligence models to intelligently compare the similarity of two or more videos, current AI models suffer from feature extraction biases, making them unsuitable for handling the diverse range of online video content and deduplication. Summary of the Invention

[0004] This application provides a video matching processing method, apparatus, computer equipment, storage medium, and product, which can improve the accuracy of video matching processing.

[0005] On one hand, embodiments of this application provide a video matching processing method, the method comprising:

[0006] Global feature extraction is performed on each first video frame obtained from the video to be matched to obtain the global video image features of the video to be matched.

[0007] Based on the global video image features of the video to be matched, an initial recall video set associated with the video to be matched is determined;

[0008] Local feature extraction is performed on each second video frame in the video to be matched to obtain the local video image features of the video to be matched.

[0009] Based on the local video image features of the video to be matched, the matching recall videos that match the video to be matched are determined from the initial recall video set.

[0010] On one hand, embodiments of this application provide a video matching processing apparatus, the apparatus comprising:

[0011] The processing unit is used to perform global feature extraction processing on each first video frame obtained from the video to be matched, so as to obtain the global video image features of the video to be matched.

[0012] The determining unit is used to determine an initial recall video set associated with the video to be matched based on the global video image features of the video to be matched.

[0013] The processing unit is also used to perform local feature extraction processing on each second video frame in the video to be matched, so as to obtain local video image features of the video to be matched.

[0014] The determining unit is also used to determine the matching recall video that matches the video to be matched from the initial recall video set based on the local video image features of the video to be matched.

[0015] In one possible implementation, the processing unit performs global feature extraction processing on each first video frame obtained from the video to be matched to obtain global video image features of the video to be matched, which are then used to perform the following operations:

[0016] The video to be matched is subjected to frame extraction to obtain multiple initial video frames;

[0017] Multiple initial video frames are downsampled to obtain multiple first video frames;

[0018] Global feature extraction is performed on each first video frame to obtain the global video image features of each first video frame, wherein the global video image features of each first video frame constitute the global video image features of the video to be matched.

[0019] In one possible implementation, the determining unit determines an initial recall video set associated with the video to be matched based on the global video image features of the video to be matched, for performing the following operations:

[0020] In the video frame feature library, feature lookup processing is performed on the global video image features of each first video frame to determine the recall list corresponding to each first video frame. The recall list includes the frame identifiers of n video frames to be recalled, and the video identifier of the video to be recalled to which each video frame to be recalled belongs, where n is a positive integer.

[0021] Based on the video identifier of the video to be recalled to which each video frame to be recalled belongs, determine the initial set of recall videos associated with the videos to be matched.

[0022] In one possible implementation, the determining unit determines an initial set of recall videos associated with the videos to be matched, based on the video identifier of the video to be recalled to which each video frame to be recalled belongs, for performing the following operations:

[0023] Iterate through the video identifiers in the recall list corresponding to each first video frame to determine the videos to be recalled. The videos to be recalled are represented by a set of multiple video frames to be recalled that have the same video identifier.

[0024] Based on the interval search rule, the video frames to be recalled in each recall list are searched and processed to obtain the equally spaced video sequence corresponding to each video to be recalled.

[0025] Based on the equally spaced video sequence corresponding to each video to be recalled, determine the initial set of recalled videos associated with the videos to be matched.

[0026] In one possible implementation, the initial recall video set includes one or more initial recall videos; the determining unit determines the initial recall video set associated with the video to be matched based on the equally spaced video sequence corresponding to each video to be recalled, for the following operations:

[0027] The repetition rate between the video to be matched and the equally spaced video sequences corresponding to each video to be recalled is calculated to obtain the repetition rate of each video to be recalled.

[0028] The videos to be recalled that meet the repetition threshold are identified as the initial recall videos associated with the videos to be matched, thus obtaining the initial recall video set.

[0029] In one possible implementation, the determining unit determines matching recall videos from the initial recall video set based on local video image features of the video to be matched, and performs the following operations:

[0030] Obtain the associated video features for each initially recalled video in the initial recalled video set;

[0031] Based on the image matching algorithm, feature matching processing is performed on the local video image features of the video to be matched and the associated video features of each initial recall video to determine multiple feature matching points between the target video frame in the video to be matched and the corresponding recall video frame in each initial recall video.

[0032] Based on multiple feature matching points between the target video frame and each recalled video frame, the matching recalled videos that match the video to be matched are determined from the initial recalled video set.

[0033] In one possible implementation, the determining unit determines the matching recalled video that matches the video to be matched from the initial recalled video set based on multiple feature matching points between the target video frame and each recalled video frame, and performs the following operations:

[0034] The similarity between the video to be matched and any initial recalled video is determined based on the number of feature matching points between the target video frame and any recalled video frame, and the minimum number of local features between the target video frame and any recalled video frame.

[0035] The initial recall videos that reach the similarity threshold in the initial recall video set are identified as matching recall videos that match the video to be matched.

[0036] In one possible implementation, the number of matched and recalled videos is multiple; the processing unit is also used to perform the following operations:

[0037] Based on the first audio information of the video to be matched, the first audio fingerprint of the video to be matched is obtained; and...

[0038] Based on the second audio information of any matched recalled video, the second audio fingerprint of any matched recalled video is obtained.

[0039] Based on the fingerprint similarity between the first audio fingerprint and the second audio fingerprint, the target recall video that matches the video to be matched is determined from multiple matching recall videos.

[0040] On one hand, embodiments of this application provide a computer device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the video matching processing method described above.

[0041] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when read and executed by a processor of a computer device, causes the computer device to perform the aforementioned video matching processing method.

[0042] On one hand, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned video matching processing method.

[0043] In this embodiment, a video to be matched can be acquired, and global feature extraction processing can be performed on each first video frame obtained from the video to be matched to obtain global video image features of the video to be matched; and local feature extraction processing can be performed on each second video frame in the video to be matched to obtain local video image features of the video to be matched; then, based on the global video image features of the video to be matched, an initial recall video set associated with the video to be matched is determined; finally, based on the local video image features of the video to be matched, matching recall videos that match the video to be matched are determined from the initial recall video set. Therefore, this application, when performing video matching and deduplication, can perform preliminary screening based on coarse-grained global video image features to obtain an initial recall video set, and then perform further screening based on fine-grained local video image features to obtain the final matching recall videos. Because multiple screenings are performed considering both global and local features, the multi-fine-grained image feature matching method can make the video matching process more accurate. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the architecture of a video matching and processing system provided in an embodiment of this application;

[0046] Figure 2 This is a schematic flowchart of a video matching processing method provided in an embodiment of this application;

[0047] Figure 3 This is a schematic diagram of a global feature extraction process provided in an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of a global feature matching process provided in an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of a model structure for extracting local features provided in an embodiment of this application;

[0050] Figure 6 This is a flowchart illustrating another video matching processing method provided in an embodiment of this application;

[0051] Figure 7 This is a schematic diagram of a local feature matching process provided in an embodiment of this application;

[0052] Figure 8This is a schematic diagram of an audio matching process provided in an embodiment of this application;

[0053] Figure 9 This is a schematic diagram of the structure of a video matching processing device provided in an embodiment of this application;

[0054] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0055] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0056] This application proposes a video matching processing scheme that can be applied to video matching scenarios, specifically video deduplication scenarios, to improve the accuracy of video deduplication. The principle of this video matching processing scheme mainly includes: in video matching scenarios such as those requiring video deduplication, a video to be matched is obtained, and global feature extraction processing is performed on each first video frame obtained from the video to be matched to obtain global video image features of the video to be matched; then, based on the global video image features of the video to be matched, an initial recall video set associated with the video to be matched is determined; next, local feature extraction processing is performed on each second video frame in the video to be matched to obtain local video image features of the video to be matched; finally, based on the local video image features of the video to be matched, matching recall videos that match the video to be matched are determined from the initial recall video set.

[0057] Therefore, when performing video matching and deduplication, this application can perform preliminary screening based on coarse-grained global video image features to obtain an initial recall video set, and then perform further screening based on fine-grained local video image features to obtain the final matched recall videos. Since multiple screenings are performed by comprehensively considering both global and local features, the multi-fine-grained image feature matching method can make the video matching process more accurate.

[0058] Next, based on the principles of the video matching processing solution provided in this application, we will introduce the relevant technical terms and main application scenarios in detail:

[0059] I. Artificial Intelligence:

[0060] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence.

[0061] In one possible implementation, this application can be combined with machine learning techniques from the field of artificial intelligence. Specifically, machine learning techniques (such as image recognition techniques) can be used to train a neural network model (e.g., an image recognition model), and based on the image recognition model, global feature extraction processing can be performed on each first video frame obtained from the video to be matched to obtain the global video image features of the video to be matched; and the image recognition model can also be used to perform local feature extraction processing on each second video frame in the video to be matched to obtain the local video image features of the video to be matched. Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to enable computers to possess intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0062] II. Cloud Technology:

[0063] Cloud computing refers to the delivery and usage model of IT infrastructure, meaning obtaining necessary resources in an on-demand and easily scalable manner through a network. In a broader sense, cloud computing also refers to the delivery and usage model of services, meaning obtaining necessary services in an on-demand and easily scalable manner through a network. These services can be IT and software related, internet-related, or other services. Cloud computing is a product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing.

[0064] In this application, "global feature extraction processing is performed on each first video frame obtained from the video to be matched to obtain global video image features of the video to be matched; and local feature extraction processing is performed on each second video frame in the video to be matched to obtain local video image features of the video to be matched." The above process involves large-scale computation and requires a large amount of computing power and storage space. Therefore, in one possible implementation of this application, a computer device can obtain sufficient computing power and storage space through cloud computing technology to execute the process of determining the matching recall video that matches the video to be matched involved in this application.

[0065] III. Blockchain:

[0066] In one possible implementation, the data involved in the video matching process provided in this application, including the video to be matched, the global video image features of the video to be matched, and the local video image features of the video to be matched, can be uploaded to a blockchain network for storage. This prevents the internal data of the blockchain nodes (computer devices) from being tampered with, thereby improving the security and reliability of the video matching process. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block.

[0067] It should be noted that in the subsequent specific embodiments of this application, when object information and other related data are involved, the above embodiments of this application are applied to specific products or technologies, and the object's permission or consent must be obtained. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0068] Next, the architecture diagram of the video matching processing system involved in this application will be described accordingly. Please refer to... Figure 1 , Figure 1 This is a schematic diagram of the architecture of a video matching processing system provided in an embodiment of this application. Figure 1 As shown in the diagram, the system architecture diagram may include at least: a server 104 and a cluster of terminal devices, wherein the cluster of terminal devices may include at least: terminal device 101, terminal device 102, terminal device 103, etc. Any terminal device in the cluster can be directly or indirectly connected to the server 104 via wired or wireless communication, which is not limited herein.

[0069] in, Figure 1The server 104 shown can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0070] Figure 1 Any of the terminal devices shown can be mobile phones, tablets, laptops, PDAs, mobile internet devices (MIDs), vehicles, in-vehicle devices, roadside devices, aircraft, wearable devices such as smartwatches, smart bracelets, pedometers, and other smart devices with video matching and processing capabilities.

[0071] In one possible implementation, taking terminal device 101 as an example, the video matching processing scheme provided in this application embodiment will be further described. Specifically, when video deduplication needs to be performed on the video to be matched, terminal device 101 can obtain the video to be matched. Then, terminal device 101 can send the video to be matched to server 104. Server 104 can perform global feature extraction processing on each first video frame obtained from the video to be matched to obtain the global video image features of the video to be matched. Then, server 104 can determine an initial recall video set associated with the video to be matched based on the global video image features of the video to be matched. Next, server 104 can perform local feature extraction processing on each second video frame in the video to be matched to obtain the local video image features of the video to be matched. Finally, server 104 can determine the matching recall video that matches the video to be matched from the initial recall video set based on the local video image features of the video to be matched. Subsequently, server 104 can send one or more matching recall videos to terminal device 101. Subsequently, terminal device 101 can also determine the target recall video from one or more matched recall videos; or terminal device 101 can also use all the matched recall videos returned by server 104 as the target recall video.

[0072] It should be understood that the above is merely an illustrative description of the specific operations performed by terminal device 101 and server 104. In another possible implementation, the global feature extraction processing of each first video frame in the video to be matched to obtain the global video image features of the video to be matched is not necessarily performed by server 104, but can also be performed by terminal device 101 (or any other terminal device in the terminal device cluster). In yet another possible implementation, the video matching processing scheme mentioned above can also be executed by the server in the video matching processing system or by any terminal device in the terminal device cluster alone, and this application embodiment does not specifically limit this.

[0073] In one possible implementation, the video matching processing system provided in this application embodiment can be deployed on a blockchain. For example, terminal device 101, terminal device 102, and server 103 can all be used as nodes in the blockchain, jointly forming a blockchain network. Therefore, the video matching processing process in this application can be executed on the blockchain, which can ensure the fairness and impartiality of the video matching processing process, while also making the video matching processing process traceable, thereby improving the security of the video matching processing process.

[0074] It is understood that the system architecture diagrams described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0075] Based on the above description of the video matching processing scheme and system, this application proposes a video matching processing method. See also... Figure 2 As shown, Figure 2 This is a flowchart illustrating a video matching processing method provided in an embodiment of this application. The video matching processing method can be derived from the above... Figure 1 The video matching processing system mentioned above is executed by a terminal device or server. For ease of explanation, this application embodiment uses a computer device as an example for illustration. The video matching processing method may include the following steps S201 to S204:

[0076] S201. Perform global feature extraction processing on each first video frame obtained from the video to be matched to obtain the global video image features of the video to be matched.

[0077] It should be understood that the global video image features of the video to be matched specifically include the global image features of each first video frame in the video to be matched. That is, the global video image features are specific to the image features of each first video frame. The first video frames obtained from the video to be matched may include: video frames obtained after frame extraction processing; or video frames obtained after frame extraction processing and downsampling processing.

[0078] In one possible implementation, the computer device performs global feature extraction processing on each first video frame obtained from the video to be matched to obtain global video image features of the video to be matched. Specifically, this may include: first, performing frame extraction processing on the video to be matched to obtain multiple initial video frames; then, performing downsampling processing on the multiple initial video frames to obtain multiple first video frames; and finally, performing global feature extraction processing on each first video frame to obtain global video image features of each first video frame, wherein the global video image features of each first video frame constitute the global video image features of the video to be matched.

[0079] Specifically, please see Figure 3 , Figure 3 This is a schematic diagram of a global feature extraction process provided in an embodiment of this application. For example... Figure 3 As shown, global features refer to the image features extracted from each first video frame of the video to be matched. The following section details the specific process of extracting global features (global video image features) from the video to be matched:

[0080] 1) The input video to be matched is processed by frame extraction at a frequency of 2fps / s (2 frames per second). That is, 2 video frames are extracted from the video to be matched every second, thus obtaining multiple initial video frames;

[0081] 2) Downsample the multiple initial video frames obtained after frame extraction to obtain multiple first video frames. The frame number is downsampled to 1 / 6 after frame extraction. For example, if the number of initial video frames is 600, then the number of first video frames obtained after downsampling is 100.

[0082] 3) A global feature extraction network (such as ResNet-101 network) can be used to extract global features from each downsampled first video frame, obtaining 2048-dimensional features for each first video frame. Then, the 2048-dimensional features for each first video frame are quantized to obtain 128-dimensional features (image features of each first video frame), and finally, the global video image features of the video to be matched are obtained.

[0083] S202. Based on the global video image features of the video to be matched, determine the initial recall video set associated with the video to be matched.

[0084] The initial recall video set may include one or more initial recall videos, which are obtained by performing video matching based on the global video image features of the videos to be matched. See also... Figure 4 , Figure 4 This is a schematic diagram of a global feature matching process provided in an embodiment of this application. Next, in conjunction with... Figure 4 The global feature matching process shown below details the specific steps involved in determining the initial recall videos:

[0085] In one possible implementation, the computer device determines an initial set of recall videos associated with the video to be matched based on global video image features of the video to be matched, which may include:

[0086] (1) Perform feature lookup processing on global video image features in the video frame feature library to determine the recall list corresponding to each first video frame in the video to be matched. The recall list includes the frame identifiers of n video frames to be recalled, and the video identifier of the video to be recalled to which each video frame belongs, where n is a positive integer. Specifically, the feature lookup processing mentioned above may include processing using a feature lookup algorithm.

[0087] Specifically, a video library can be obtained, containing multiple videos. Global feature extraction can then be performed on each video in the library (the specific steps for global feature extraction are described in step S201 above and will not be repeated here), resulting in global video image features for each video in the library. Next, the global video image features of each video in the library can be packaged and indexed for subsequent queries. For example, if the video library includes Video 1, Video 2, and Video 3, the global video image features of each video frame can be extracted, indexed, and stored in a video frame feature library. For instance, the video frame feature library could store: Video frame feature 1 -> index1; Video frame feature 2 -> index2; Video frame feature 3 -> index3.

[0088] Further, after the indexing and packaging are completed, a nearest neighbor search is performed in the video frame feature library based on the global video image features of each first video frame in the video to be matched. That is, for any first video frame, a search is performed in the video frame feature library using a feature search algorithm (e.g., a nearest neighbor search algorithm) to check the global image features of each first video frame against the video frame feature library, resulting in a recall list corresponding to each first video frame. It should be understood that each video frame feature in the video frame feature library refers to global video image features, and each video frame feature in the video frame feature library is also obtained based on the specific execution steps corresponding to step S201. The recall list includes the frame identifiers (frame ids) of the top-N(n) video frames closest to the currently queried first video frame, and the video identifier (video id) of the video to which each video frame belongs. Figure 4 As shown, assume the first video frames are represented as: x1, x2, x3, x4 ( Figure 4 If the data is arranged from top to bottom, then the recall list corresponding to the first video frame x2 can include: the frame id = 011 and video id = 01 of the video frame 1 associated with the first video frame x2; the frame id = 011 and video id = 05 of the video frame 2 associated with the first video frame x2; and the frame id = 005 and video id = 06 of the video frame 3 associated with the first video frame x2.

[0089] (2) Determine the initial set of recall videos associated with the videos to be matched based on the video identifier of the video to be recalled to which each video frame to be recalled belongs.

[0090] In one possible implementation, firstly, the computer device iterates through the video identifiers in the recall list corresponding to each first video frame to determine the videos to be recalled. The videos to be recalled are represented by a set of multiple video frames with the same video identifier. Then, the computer device performs a search process on the video frames to be recalled in each recall list based on an interval lookup rule, obtaining an equally spaced video sequence corresponding to each video to be recalled. Finally, the computer device determines an initial set of recalled videos associated with the videos to be matched based on the equally spaced video sequences corresponding to each video to be recalled.

[0091] Specifically, the process begins by iterating through the retrieved lists to identify the retrieved video IDs (videos to be retrieved) and the set of retrieved frames belonging to those videos (the set of videos to be retrieved). Then, the longest equal-interval sequence algorithm is used to find the equally spaced video sequences within the set of videos to be retrieved, and these sequences are considered as repeating segments between two retrieved videos (the video to be matched and any video to be retrieved).

[0092] In one possible implementation, the initial recall video set includes one or more initial recall videos. The process by which a computer device determines the initial recall videos associated with a video to be matched based on the equally spaced video sequences corresponding to each video to be recalled may include: calculating the repetition between the video to be matched and the equally spaced video sequences corresponding to each video to be recalled, obtaining the repetition of each video to be recalled; and determining the videos to be recalled that meet the repetition threshold as the initial recall videos associated with the videos to be matched, thus obtaining the initial recall video set.

[0093] Specifically, the similarity score between any initial recall video and the video to be matched can be calculated based on a similarity algorithm. Such algorithms include, but are not limited to, cosine similarity, Jaccard similarity, and Euclidean distance. If the calculated similarity score reaches a similarity threshold, the video to be recalled corresponding to that score is used as the initial recall video. Following this method, the calculation can be performed for each video to be recalled, thus forming an initial recall video set from all determined initial recall videos.

[0094] In this way, one or more initial recall videos that are associated with the video to be matched can be initially screened from the video feature library based on the global video image features of the video to be matched.

[0095] S203. Perform local feature extraction processing on each second video frame in the video to be matched to obtain the local video image features of the video to be matched.

[0096] In one possible implementation, the computer device performs local feature extraction processing on each second video frame in the video to be matched to obtain local video image features of the video to be matched. This may include: first, performing frame extraction processing on the video to be matched to obtain multiple second video frames; then, calling a local feature model to perform local feature extraction processing on each second video frame to obtain local video image features corresponding to each second video frame in the video to be matched.

[0097] Specifically, the process of the computer device performing frame extraction on the video to be matched can be consistent with the relevant frame extraction steps involved in step S201 above. That is, the computer device can perform frame extraction on the input video to be matched at a frequency of 2fps / s (2 frames / second). That is, two initial video frames are obtained from the video to be matched per second, thereby obtaining multiple initial video frames; then, the multiple initial video frames obtained after frame extraction are downsampled to obtain multiple second video frames. In one possible implementation, in this embodiment, the second video frames and the first video frames can be the same or different. Specifically, the second video frames can be: the various video frames obtained after performing frame extraction on the video to be matched; or, the various video frames obtained after performing frame extraction and downsampling on the video to be matched. For ease of explanation, the following embodiments of this application will be described with the example that the second video frames and the first video frames are the same. It should be noted that when the first video frames and the second video frames are the same, the first video frames and the second video frames in the video to be matched can be simply referred to as video frames, that is, they can be referred to as video frames of the video to be matched in the following description.

[0098] Furthermore, after obtaining multiple second video frames, these extracted second video frames can be grouped into batches. That is, assuming 100 second video frames are extracted, then every 10 second video frames can be grouped into a batch, resulting in batch 1, batch 2, batch 3...batch 10. By packaging multiple second video frames into a single batch and inputting it into the model for processing in this way, the model's processing efficiency can be improved.

[0099] In this embodiment, the local feature models mainly include two types: those based on deep learning methods and those based on traditional methods. The models based on traditional methods mainly include SIFT (Scale-invariant feature transform) models and Harris (corner detection) models. It should be understood that traditional methods primarily rely on CPU (Central Processing Unit) resources. However, in practical deployments, it is desirable to incorporate local features into existing extraction modules. Therefore, in this embodiment, deep learning-based methods can be used to extract image features using GPU (Graphics Processing Unit) resources. Furthermore, in this embodiment, the local feature extraction model can be a neural network model, which can be used to extract local video image features. This embodiment does not specifically limit the model structure of the local feature extraction model.

[0100] Research has found that the DELF (Deep Local Features) model can be used to extract local features from video frames. Please refer to... Figure 5 , Figure 5 This is a schematic diagram of a model structure for extracting local features according to an embodiment of this application. Specifically, the extracted second video frames from the video to be matched can be input into the DELF model in batches 1, 2, 3...10. Figure 5 The first branch in the DELF model shown (e.g.) Figure 5 The features obtained within the dashed box shown are used as local video image features for each second video frame. Specifically, after recognizing any batch of video frames (e.g., batch 1) based on the DELF model, further self-attention extraction processing can be performed based on the autoencoder module to obtain the local video image features for each video frame. Optionally, any batch of video frames (e.g., batch 1) mentioned above can be input into... Figure 5 After the local feature extraction model shown performs recognition processing, the results can be further pooled. Based on the global video image features obtained after pooling, and the local video image features of each video frame, local video image features for each video frame are retrieved. In this way, local video image features can be further fused based on global video image features to make the obtained local video image features more accurate. Furthermore, the extracted local video image features of each video frame can be stored in the cloud, and the cloud address can be recorded in a database for subsequent deduplication. That is, when there is a query requirement, the corresponding database can be found by accessing the cloud address, and the stored local video image features of each video frame can be retrieved from the database.

[0101] In this way, the local feature model can be invoked to extract local video image features from each second video frame in the video to be matched. The extracted local video image features can be used for subsequent accurate matching.

[0102] S204. Based on the local video image features of the video to be matched, determine the target recall video that matches the video to be matched from the initial recall video set.

[0103] The initial recall video set may include at least one initial recall video. The number of target recall videos that match the video to be matched can be one or more, and this application embodiment does not make a specific limitation.

[0104] In one possible implementation, the computer device determines a target recall video that matches the video to be matched from an initial recall video set based on the local video image features of the video to be matched. This may include: first, acquiring the associated video features of each initial recall video in the initial recall video set; then, performing feature matching processing on the local video image features of the video to be matched and the associated video features of each initial recall video based on an image matching algorithm to determine multiple feature matching points between the target video frame in the video to be matched and the corresponding recall video frame in each initial recall video; finally, determining the matching recall video that matches the video to be matched from the initial recall video set based on the multiple feature matching points between the target video frame and each recall video frame.

[0105] Specifically, the computer device determines matching recalled videos from an initial recalled video set based on multiple feature matching points between the target video frame and each recalled video frame. This can include: first, determining the similarity between the video to be matched and any initial recalled video based on the number of feature matching points between the target video frame and any recalled video frame, and the minimum number of local features between the target video frame and any recalled video frame; then, determining reference recalled videos from the initial recalled video set that reach a similarity threshold; and finally, determining matching recalled videos that match the video to be matched based on the reference recalled videos.

[0106] In this way, based on the local video image features of the video to be matched, the initial recall video set obtained from the preliminary screening can be further filtered to obtain a matching recall video with a higher degree of matching with the video to be matched. Since multiple screenings are performed by comprehensively considering local and global features, the accuracy of the video matching process can be improved.

[0107] In this embodiment, a video to be matched can be acquired, and global feature extraction processing can be performed on each first video frame obtained from the video to be matched to obtain global video image features of the video to be matched. Then, based on the global video image features of the video to be matched, an initial recall video set associated with the video to be matched is determined. Next, local feature extraction processing can be performed on each second video frame in the video to be matched to obtain local video image features of the video to be matched. Finally, based on the local video image features of the video to be matched, matching recall videos that match the video to be matched are determined from the initial recall video set. Therefore, this application, when performing video matching and deduplication, can perform preliminary screening based on coarse-grained global video image features to obtain an initial recall video set, and then perform further screening based on fine-grained local video image features to obtain the final matching recall videos. Because multiple screenings are performed considering both global and local features, the multi-fine-grained image feature matching method can make the video matching process more accurate.

[0108] Next, please see Figure 6 , Figure 6 This is a flowchart illustrating another video matching processing method provided in an embodiment of this application. This video matching processing method can be derived from the above... Figure 1 The video matching processing system mentioned herein is executed by a terminal device or server. For ease of explanation, this application embodiment uses computer equipment as an example for illustration. The video matching processing method may include the following steps S601 to S606:

[0109] S601: Perform global feature extraction processing on each first video frame obtained from the video to be matched to obtain the global video image features of the video to be matched.

[0110] S602: Based on the global video image features of the video to be matched, determine the initial recall video set associated with the video to be matched.

[0111] S603: Perform local feature extraction processing on each second video frame in the video to be matched to obtain the local video image features of the video to be matched.

[0112] It should be noted that the specific steps performed by the computer device in steps S601-S603 can be found in the execution methods of the relevant steps in steps S201-S203, and will not be repeated here in the embodiments of this application.

[0113] S604: Obtain the associated video features for each initial recalled video.

[0114] It should be noted that the associated video features of each initial recalled video refer to the local video image features of each initial recalled video. Specifically, the initial recalled video includes multiple recalled video frames, and the associated video features can include the local video image features of each recalled video frame. The detailed steps for the computer device to obtain the associated video features of each initial recalled video can be found in the process of the computer device performing local feature extraction processing on each second video frame in the video to be matched to obtain the local video image features of the video to be matched. These steps will not be repeated here in the embodiments of this application.

[0115] S605: Based on the image matching algorithm, perform feature matching processing on the local video image features of the video to be matched and the associated video features of each initial recall video to determine multiple feature matching points between the target video frame in the video to be matched and the corresponding recall video frame in each initial recall video.

[0116] For example, please see Figure 7 , Figure 7 This is a schematic diagram of a local feature matching process provided in an embodiment of this application. For example... Figure 7 As shown, for two videos requiring local feature matching: the video to be matched (video A) and the initial recall video (video B), the following steps are taken: Figure 2 The global feature search and matching described in step S202 of the embodiment can obtain the starting positions of possible repeating segments between two videos, such as... Figure 4 The initial recalled video shows the shaded area (as shown in Figure S401). Then, based on the Local Feature Extraction (DELF) model, local feature extraction is performed on the repeated segments in video A and video B to obtain the local video image features of each frame in video A and video B. For details on how to extract local video image features of video frames, please refer to [link to relevant documentation]. Figure 2 The relevant processes described in step S203 of the embodiments will not be repeated here.

[0117] Image matching algorithms may include, but are not limited to: RANSAC (RANdom Sampling Consensus) algorithm, MAD (Mean Absolute Differences) algorithm, SAD (Sum of Absolute Differences) algorithm, and so on.

[0118] S606: Based on multiple feature matching points between the target video frame and each recalled video frame, determine the matching recalled video that matches the video to be matched from the initial recalled video set.

[0119] In one possible implementation, the computer device determines matching recalled videos from an initial recalled video set based on multiple feature matching points between the target video frame and each recalled video frame. This may include: determining the similarity between the target video frame and the recalled video frames based on the number of feature matching points between the target video frame and any recalled video frame, and the minimum number of local features between the target video frame and any recalled video frame; determining the similarity between the video to be matched and any initial recalled video based on the similarity between the target video frame and the recalled video frames; and identifying the initial recalled videos in the initial recalled video set that reach a similarity threshold as matching recalled videos that match the video to be matched.

[0120] In specific implementation, when performing local feature matching, assuming that the local video image features corresponding to video frames in video A are represented as local feature 1, and the local video image features corresponding to video frames in video B are represented as local feature 2, then an image matching algorithm can be used to perform feature matching processing on the local video image features of the video to be matched and the associated video features of each initially recalled video. For ease of explanation, this application's embodiment uses the RANSAC algorithm as an example for detailed description:

[0121] First, when performing local feature matching on two sets of local features (local feature 1 corresponding to video A and local feature 2 corresponding to video B) based on the RANSAC algorithm, local feature matching is performed on any video frame in video A and the recalled video frame in video B. In other words, local feature matching is performed on a set of video frames. Specifically, for the target video frame in video A, feature matching can be performed on local feature 1 of the target video frame x1 in video A and local feature 2 of the recalled video frame y1 in video B based on the RANSAC algorithm, thereby obtaining multiple feature matching points between the target video frame x1 and the recalled video frame y1.

[0122] Then, the similarity between the target video frame and any recalled video frame is calculated based on the number of feature matching points between the target video frame and any recalled video frame, and the minimum number of local features between them. Specifically, the ratio of the number of feature matching points to the minimum number of local features between the target video frame and any recalled video frame can be used as the similarity between them. In this way, the similarity between each video frame in the video to be matched and the corresponding recalled video frame in any initial recalled video can be calculated. For example, the video frames included in the video to be matched are denoted as x1, x2, ..., xn; the recalled video frames included in any initial recalled video are denoted as y1, y2, ..., yn. Using the above method, the similarity s11 between x1 and y1; the similarity s12 between x2 and y2; ..., the similarity s1n between xn and yn can be obtained.

[0123] For example, the video to be matched is denoted as V0, and the initial recall videos are denoted as V1, V2, ..., Vn. Following this method, the similarity between each video frame in the video to be matched and the corresponding recall video frames in each of the initial recall videos can be obtained, and the matching results can be recorded in a table, as shown in Table 1 below:

[0124] Table 1. Similarity between each video frame of the video to be matched and the recalled video frames in each initial recall video.

[0125]

[0126] As shown in Table 1 above, s11 is used to represent the similarity between video frame x1 in the video to be matched and the corresponding video frame to be recalled in the initial recall video V1; s21 is used to represent the similarity between video frame x1 in the video to be matched and the corresponding video frame to be recalled in the initial recall video V2; s12 is used to represent the similarity between video frame x2 in the video to be matched and the corresponding video frame to be recalled in the initial recall video V1, and so on, s1n is used to represent the similarity between video frame xn in the video to be matched and the corresponding video frame to be recalled in the initial recall video V1.

[0127] Next, based on the similarity between each video frame in the video to be matched and the corresponding recalled video frame in any initial recalled video, the similarity between the video to be matched and any initial recalled video is determined. In one possible implementation, the obtained similarities s11, s12...s1n can be weighted and averaged to obtain the similarity between the video to be matched and any initial recalled video. In another possible implementation, the maximum (or minimum) value can be selected from the obtained similarities s11, s12...s1n as the similarity between the video to be matched and any initial recalled video.

[0128] Finally, the similarity between the video to be matched and each initial recalled video can be obtained. Then, the initial recalled videos with a similarity greater than or equal to the similarity threshold are used as the matched recalled videos that match the video to be matched. The number of matched recalled videos can be one or more.

[0129] In one possible implementation, further matching processing can be performed on the multiple matched recall videos obtained based on the audio information of the video to be matched. Specifically, the computer device can obtain a first audio fingerprint of the video to be matched based on the first audio information of the video to be matched; and obtain a second audio fingerprint of any matched recall video based on the second audio information of any matched recall video. Then, the computer device can determine the target recall video that matches the video to be matched from the matched recall videos based on the fingerprint similarity between the first audio fingerprint and the second audio fingerprint.

[0130] Please see Figure 8 , Figure 8 This is a schematic diagram of an audio matching process provided in an embodiment of this application. For example... Figure 8 As shown, audio comparison can be performed between the video to be matched and the video to be recalled. The specific audio comparison process is as follows: First, the audio information of each video is obtained; specifically, the audio information can be stored in WAV file format. Then, the audio fingerprints of each video are extracted based on the WAV files. Finally, the repetition rate between the two sets of audio fingerprints is calculated as the similarity between the two audio samples (the video to be matched and the video to be recalled). Next, combined with... Figure 8 The above process will be explained in detail:

[0131] 1) Obtain audio information of each video: The computer device can obtain the first audio information of the video to be matched (e.g., the wav1 file), and then extract the first audio fingerprint corresponding to the wav1 file based on the audio fingerprint extraction algorithm; similarly, the computer device can obtain the second audio information of the video to be matched and recalled (e.g., the wav2 file), and then extract the second audio fingerprint corresponding to the wav2 file based on the audio fingerprint extraction algorithm.

[0132] 2) The process of extracting audio fingerprints (taking the first audio fingerprint as an example). For example... Figure 8 As shown, firstly, a Fast Fourier Transform is performed on the audio to be matched to obtain spectral information. Then, peak points are searched in the spectral information. It can be understood that if a peak point is not found in the spectrum, it and its surrounding peak points can be grouped together. Next, the peak positions and magnitudes in each group are mapped to corresponding hash codes. Specifically, this can be done based on hash operations. Thus, an audio file can obtain a set of hash codes, which can serve as the first audio fingerprint of the audio to be matched. It should be understood that the method for extracting the second audio fingerprint of any matching recall video can also refer to the process of extracting the first audio fingerprint of the audio to be matched, and will not be elaborated further in this embodiment.

[0133] 3) The repetition between the first and second audio fingerprints can be calculated and used as the fingerprint similarity between the video to be matched and the recalled videos. Finally, the recalled videos with fingerprint similarity greater than or equal to the fingerprint similarity threshold are used as the target recalled videos that match the video to be matched.

[0134] In this embodiment, video matching can be performed by combining image and audio information from the video. Specifically, firstly, preliminary screening and matching are performed based on the global video image features of the video to be matched, resulting in an initial recall video set; then, further screening and matching are performed based on the local video image features of the video to be matched, resulting in matched recall videos. Furthermore, matching can also be performed again based on the audio features (audio fingerprint) of the video to be matched, thereby obtaining target recall videos with a higher degree of matching. Therefore, by comprehensively considering both audio and image information, the video matching method based on audio features and multi-fine-grained image features in this embodiment can further improve the accuracy of the video matching process.

[0135] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above-described solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below. Next, the relevant apparatus of the embodiments of this application will be introduced in conjunction with the video matching processing scheme provided in the above-described embodiments of this application.

[0136] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a video matching processing device provided in an embodiment of this application. Figure 9As shown, the video matching processing device 900 can be applied to the computer equipment (terminal device or server) mentioned in the foregoing embodiments. Specifically, the video matching processing device 900 can be a computer program (including program code) running on the computer equipment, for example, the video matching processing device 900 is an application software; the video matching processing device 900 can be used to execute the corresponding steps in the video matching processing method provided in the embodiments of this application. The video matching processing device 900 includes:

[0137] The processing unit 901 is used to perform global feature extraction processing on each first video frame obtained from the video to be matched, so as to obtain the global video image features of the video to be matched.

[0138] The determining unit 902 is used to determine an initial recall video set associated with the video to be matched based on the global video image features of the video to be matched;

[0139] The processing unit 901 is also used to perform local feature extraction processing on each second video frame in the video to be matched, so as to obtain local video image features of the video to be matched.

[0140] The determining unit 902 is further configured to determine, based on the local video image features of the video to be matched, a matching recall video that matches the video to be matched from the initial recall video set.

[0141] In one possible implementation, the processing unit 901 performs global feature extraction processing on each first video frame obtained from the video to be matched to obtain global video image features of the video to be matched, which are used to perform the following operations:

[0142] The video to be matched is subjected to frame extraction to obtain multiple initial video frames;

[0143] Multiple initial video frames are downsampled to obtain multiple first video frames;

[0144] Global feature extraction is performed on each first video frame to obtain the global video image features of each first video frame, wherein the global video image features of each first video frame constitute the global video image features of the video to be matched.

[0145] In one possible implementation, the determining unit 902 determines an initial recall video set associated with the video to be matched based on the global video image features of the video to be matched, for performing the following operations:

[0146] In the video frame feature library, feature lookup processing is performed on the global video image features of each first video frame to determine the recall list corresponding to each first video frame. The recall list includes the frame identifiers of n video frames to be recalled, and the video identifier of the video to be recalled to which each video frame to be recalled belongs, where n is a positive integer.

[0147] Based on the video identifier of the video to be recalled to which each video frame to be recalled belongs, determine the initial set of recall videos associated with the videos to be matched.

[0148] In one possible implementation, the determining unit 902 determines an initial set of recall videos associated with the videos to be matched based on the video identifier of the video to be recalled to which each video frame to be recalled belongs, for performing the following operations:

[0149] Iterate through the video identifiers in the recall list corresponding to each first video frame to determine the videos to be recalled. The videos to be recalled are represented by a set of multiple video frames to be recalled that have the same video identifier.

[0150] Based on the interval search rule, the video frames to be recalled in each recall list are searched and processed to obtain the equally spaced video sequence corresponding to each video to be recalled.

[0151] Based on the equally spaced video sequence corresponding to each video to be recalled, determine the initial set of recalled videos associated with the videos to be matched.

[0152] In one possible implementation, the initial recall video set includes one or more initial recall videos; the determining unit 902 determines the initial recall video set associated with the video to be matched based on the equally spaced video sequence corresponding to each video to be recalled, and performs the following operations:

[0153] The repetition rate between the video to be matched and the equally spaced video sequences corresponding to each video to be recalled is calculated to obtain the repetition rate of each video to be recalled.

[0154] The videos to be recalled that meet the repetition threshold are identified as the initial recall videos associated with the videos to be matched, thus obtaining the initial recall video set.

[0155] In one possible implementation, the determining unit 902 determines matching recall videos from the initial recall video set based on the local video image features of the video to be matched, and performs the following operations:

[0156] Obtain the associated video features for each initially recalled video in the initial recalled video set;

[0157] Based on the image matching algorithm, feature matching processing is performed on the local video image features of the video to be matched and the associated video features of each initial recall video to determine multiple feature matching points between the target video frame in the video to be matched and the corresponding recall video frame in each initial recall video.

[0158] Based on multiple feature matching points between the target video frame and each recalled video frame, the matching recalled videos that match the video to be matched are determined from the initial recalled video set.

[0159] In one possible implementation, the determining unit 902 determines the matching recalled video that matches the video to be matched from the initial recalled video set based on multiple feature matching points between the target video frame and each recalled video frame, and performs the following operations:

[0160] The similarity between the video to be matched and any initial recalled video is determined based on the number of feature matching points between the target video frame and any recalled video frame, and the minimum number of local features between the target video frame and any recalled video frame.

[0161] The initial recall videos that reach the similarity threshold in the initial recall video set are identified as matching recall videos that match the video to be matched.

[0162] In one possible implementation, the number of matched and recalled videos is multiple; the processing unit 901 is also used to perform the following operations:

[0163] Based on the first audio information of the video to be matched, the first audio fingerprint of the video to be matched is obtained; and...

[0164] Based on the second audio information of any matched recalled video, the second audio fingerprint of any matched recalled video is obtained.

[0165] Based on the fingerprint similarity between the first audio fingerprint and the second audio fingerprint, the target recall video that matches the video to be matched is determined from multiple matching recall videos.

[0166] In this embodiment, a video to be matched can be acquired, and global feature extraction processing can be performed on each first video frame obtained from the video to be matched to obtain global video image features of the video to be matched. Then, based on the global video image features of the video to be matched, an initial recall video set associated with the video to be matched is determined. Next, local feature extraction processing can be performed on each second video frame in the video to be matched to obtain local video image features of the video to be matched. Finally, based on the local video image features of the video to be matched, matching recall videos that match the video to be matched are determined from the initial recall video set. Therefore, this application, when performing video matching and deduplication, can perform preliminary screening based on coarse-grained global video image features to obtain an initial recall video set, and then perform further screening based on fine-grained local video image features to obtain the final matching recall videos. Because multiple screenings are performed considering both global and local features, the multi-fine-grained image feature matching method can make the video matching process more accurate.

[0167] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device 1000 is used to execute the steps performed by the computer device in the aforementioned method embodiments. The computer device 1000 includes: one or more processors 1010; one or more input devices 1020; one or more output devices 1030; and a memory 1040. The processors 1010, input devices 1020, output devices 1030, and memory 1040 are connected via a bus 1050. Specifically, the memory 1040 is used to store a computer program, which includes program instructions. The processor 1010 is used to call the program instructions stored in the memory 1040 to perform the following operations:

[0168] Global feature extraction is performed on each first video frame obtained from the video to be matched to obtain the global video image features of the video to be matched.

[0169] Based on the global video image features of the video to be matched, an initial recall video set associated with the video to be matched is determined;

[0170] Local feature extraction is performed on each second video frame in the video to be matched to obtain the local video image features of the video to be matched.

[0171] Based on the local video image features of the video to be matched, the matching recall videos that match the video to be matched are determined from the initial recall video set.

[0172] In one possible implementation, the processor 1010 performs global feature extraction processing on each first video frame obtained from the video to be matched to obtain global video image features of the video to be matched, which are used to perform the following operations:

[0173] The video to be matched is subjected to frame extraction to obtain multiple initial video frames;

[0174] Multiple initial video frames are downsampled to obtain multiple first video frames;

[0175] Global feature extraction is performed on each first video frame to obtain the global video image features of each first video frame, wherein the global video image features of each first video frame constitute the global video image features of the video to be matched.

[0176] In one possible implementation, the processor 1010 determines an initial recall video set associated with the video to be matched based on the global video image features of the video to be matched, for performing the following operations:

[0177] In the video frame feature library, feature lookup processing is performed on the global video image features of each first video frame to determine the recall list corresponding to each first video frame. The recall list includes the frame identifiers of n video frames to be recalled, and the video identifier of the video to be recalled to which each video frame to be recalled belongs, where n is a positive integer.

[0178] Based on the video identifier of the video to be recalled to which each video frame to be recalled belongs, determine the initial set of recall videos associated with the videos to be matched.

[0179] In one possible implementation, the processor 1010 determines an initial set of recall videos associated with the videos to be matched based on the video identifier of the video to be recalled to which each video frame to be recalled belongs, for performing the following operations:

[0180] Iterate through the video identifiers in the recall list corresponding to each first video frame to determine the videos to be recalled. The videos to be recalled are represented by a set of multiple video frames to be recalled that have the same video identifier.

[0181] Based on the interval search rule, the video frames to be recalled in each recall list are searched and processed to obtain the equally spaced video sequence corresponding to each video to be recalled.

[0182] Based on the equally spaced video sequence corresponding to each video to be recalled, determine the initial set of recalled videos associated with the videos to be matched.

[0183] In one possible implementation, the initial recall video set includes one or more initial recall videos; the processor 1010 determines the initial recall video set associated with the video to be matched based on the equally spaced video sequence corresponding to each video to be recalled, for the following operations:

[0184] The repetition rate between the video to be matched and the equally spaced video sequences corresponding to each video to be recalled is calculated to obtain the repetition rate of each video to be recalled.

[0185] The videos to be recalled that meet the repetition threshold are identified as the initial recall videos associated with the videos to be matched, thus obtaining the initial recall video set.

[0186] In one possible implementation, the processor 1010 determines matching recall videos from the initial recall video set based on local video image features of the video to be matched, for the following operations:

[0187] Obtain the associated video features for each initially recalled video in the initial recalled video set;

[0188] Based on the image matching algorithm, feature matching processing is performed on the local video image features of the video to be matched and the associated video features of each initial recall video to determine multiple feature matching points between the target video frame in the video to be matched and the corresponding recall video frame in each initial recall video.

[0189] Based on multiple feature matching points between the target video frame and each recalled video frame, the matching recalled videos that match the video to be matched are determined from the initial recalled video set.

[0190] In one possible implementation, the processor 1010 determines the matching recall video that matches the video to be matched from the initial recall video set based on multiple feature matching points between the target video frame and each recalled video frame, for the following operations:

[0191] The similarity between the video to be matched and any initial recalled video is determined based on the number of feature matching points between the target video frame and any recalled video frame, and the minimum number of local features between the target video frame and any recalled video frame.

[0192] The initial recall videos that reach the similarity threshold in the initial recall video set are identified as matching recall videos that match the video to be matched.

[0193] In one possible implementation, the number of matched and recalled videos is multiple; the processor 1010 is also used to perform the following operations:

[0194] Based on the first audio information of the video to be matched, the first audio fingerprint of the video to be matched is obtained; and...

[0195] Based on the second audio information of any matched recalled video, the second audio fingerprint of any matched recalled video is obtained.

[0196] Based on the fingerprint similarity between the first audio fingerprint and the second audio fingerprint, the target recall video that matches the video to be matched is determined from multiple matching recall videos.

[0197] In this embodiment, a video to be matched can be acquired, and global feature extraction processing can be performed on each first video frame obtained from the video to be matched to obtain global video image features of the video to be matched. Then, based on the global video image features of the video to be matched, an initial recall video set associated with the video to be matched is determined. Next, local feature extraction processing can be performed on each second video frame in the video to be matched to obtain local video image features of the video to be matched. Finally, based on the local video image features of the video to be matched, matching recall videos that match the video to be matched are determined from the initial recall video set. Therefore, this application, when performing video matching and deduplication, can perform preliminary screening based on coarse-grained global video image features to obtain an initial recall video set, and then perform further screening based on fine-grained local video image features to obtain the final matching recall videos. Because multiple screenings are performed considering both global and local features, the multi-fine-grained image feature matching method can make the video matching process more accurate.

[0198] Furthermore, it should be noted that this application also provides a computer storage medium storing a computer program, which includes program instructions. When the processor executes these program instructions, it can perform the methods described in the preceding embodiments. Therefore, further details will not be provided here. For technical details not disclosed in the embodiments of the computer storage medium involved in this application, please refer to the description of the method embodiments of this application. As an example, the program instructions can be deployed on a computer device, executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network.

[0199] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described in the preceding embodiments; therefore, further details will not be provided here.

[0200] Those skilled in the art will understand that implementing all or part of the processes in the above embodiments can be accomplished by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0201] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A video matching processing method, characterized in that, include: Global feature extraction processing is performed on each first video frame obtained from the video to be matched to obtain global video image features of the video to be matched; and local feature extraction processing is performed on each second video frame in the video to be matched to obtain local video image features of the video to be matched; wherein, each first video frame is a video frame obtained after frame extraction and downsampling processing of the video to be matched, and each second video frame is a video frame obtained after frame extraction processing of the video to be matched; In a video frame feature library that pre-stores global video image features of reference video frames and corresponding frame and video identifiers, a nearest neighbor search is performed on the global video image features of each first video frame to determine the recall list corresponding to each first video frame. The recall list includes the frame identifiers of n video frames to be recalled and the video identifier of the video to be recalled to which each video frame belongs, where n is a positive integer. Iterate through the video identifiers in the recall list corresponding to each of the first video frames to determine the set of videos to be recalled, which consists of multiple video frames to be recalled with the same video identifier. Based on the interval search rule, the video frames to be recalled in each of the recall lists are searched and processed to obtain the equally spaced video sequence corresponding to each video to be recalled. Based on the equally spaced video sequence corresponding to each video to be recalled, an initial set of recalled videos associated with the videos to be matched is determined; Obtain the associated video features for each initially recalled video in the initial recalled video set; Based on the image matching algorithm, feature matching processing is performed on the local video image features of the video to be matched and the associated video features of each initial recall video to determine multiple feature matching points between the target video frame in the video to be matched and the corresponding recall video frame in each initial recall video. Determine the number of feature matching points between the target video frame and any of the recalled video frames, and the minimum number of local features between the target video frame and any of the recalled video frames; The similarity between the video to be matched and any of the initial recalled videos is determined based on the ratio between the number of feature matching points and the minimum number of local features. The initial recall videos that reach the similarity threshold in the initial recall video set are identified as matching recall videos that match the video to be matched.

2. The method as described in claim 1, characterized in that, The step of performing global feature extraction processing on each first video frame obtained from the video to be matched to obtain the global video image features of the video to be matched includes: The video to be matched is subjected to frame extraction to obtain multiple initial video frames; The multiple initial video frames are downsampled to obtain multiple first video frames; Global feature extraction is performed on each of the first video frames to obtain global video image features of each of the first video frames, wherein the global video image features of each of the first video frames constitute the global video image features of the video to be matched.

3. The method as described in claim 1, characterized in that, The step of determining the initial recall video set associated with the video to be matched based on the equally spaced video sequence corresponding to each video to be recalled includes: The repetition degree between the video to be matched and the equally spaced video sequences corresponding to each video to be recalled is calculated to obtain the repetition degree of each video to be recalled. The videos to be recalled that meet the repetition threshold are identified as the initial recall videos associated with the videos to be matched, thus obtaining the initial recall video set.

4. The method according to any one of claims 1-3, characterized in that, The number of videos to be matched and recalled is multiple; the method also includes: Based on the first audio information of the video to be matched, a first audio fingerprint of the video to be matched is obtained; and, Based on the second audio information of any matched recalled video, the second audio fingerprint of the any matched recalled video is obtained; Based on the fingerprint similarity between the first audio fingerprint and the second audio fingerprint, a target recall video that matches the video to be matched is determined from multiple matching recall videos.

5. A computer device, characterized in that, include: Storage devices and processors; A memory, wherein one or more computer programs are stored; A processor for loading one or more computer programs to implement the video matching processing method as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-4.

7. A computer program product, characterized in that, The computer program product includes a computer program adapted to be loaded by a processor and execute the video matching processing method as described in any one of claims 1-4.

Images