Video retrieval methods, devices, equipment and storage media

Abstract

This application discloses a video retrieval method, apparatus, device, and storage medium, belonging to the field of artificial intelligence technology. This application obtains a target similarity parameter by comprehensively considering both video-level and video frame-level spatiotemporal information. This target similarity parameter can more accurately represent the degree of similarity between a first video and a second video. Therefore, based on this target similarity parameter, the accuracy of retrieving duplicate segments between the first and second videos can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video retrieval method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of computer technology and mobile internet, the number of videos on the internet has exploded, including a large amount of duplicate content, leading to a waste of network resources. Therefore, retrieving duplicate segments between two videos has become an important research topic. Related technologies, for a given set of two videos, use one as the query video (i.e., Query) and the other as the target video. A convolutional model is used to extract features from each frame of both the query and target videos. Based on these features, similarity parameters between each frame are calculated. Algorithms such as dynamic programming and Huffman voting are then used to process these similarity parameters, resulting in the final similarity parameters between the query and target videos. Finally, duplicate segments between the target and query videos are retrieved based on these similarity parameters.

[0003] In the aforementioned techniques, the video frame features obtained based on the convolution model cannot accurately represent the similarity between the query video and the target video, resulting in low accuracy in retrieving duplicate segments between the target video and the query video. Summary of the Invention

[0004] This application provides a video retrieval method, apparatus, device, and storage medium. The method can improve the accuracy of retrieving duplicate segments between a target video and a query video. The technical solution is as follows:

[0005] On the one hand, a video retrieval method is provided, which includes:

[0006] Obtain the first global spatiotemporal features of the first video, the first spatiotemporal features of multiple first video frames in the first video, the second global spatiotemporal features of the second video, and the second spatiotemporal features of multiple second video frames in the second video;

[0007] Based on the first global spatiotemporal feature and the second global spatiotemporal feature, a first similarity parameter is obtained, which is used to represent the video-level similarity between the first video and the second video.

[0008] Based on multiple first spatiotemporal features and multiple second spatiotemporal features, a second similarity parameter is obtained, which is used to represent the degree of similarity between the first video and the second video at the video frame level.

[0009] The first similarity parameter and the second similarity parameter are fused to obtain a target similarity parameter. In response to the target similarity parameter being greater than or equal to a first threshold, it is determined that there are repeated segments between the first video and the second video. The target similarity parameter is used to represent the overall similarity between the first video and the second video.

[0010] On the one hand, a video retrieval device is provided, the device comprising:

[0011] The acquisition module is used to acquire the first global spatiotemporal features of the first video, the first spatiotemporal features of multiple first video frames in the first video, the second global spatiotemporal features of the second video, and the second spatiotemporal features of multiple second video frames in the second video;

[0012] The acquisition module is used to acquire a first similarity parameter based on the first global spatiotemporal feature and the second global spatiotemporal feature. The first similarity parameter is used to represent the video-level similarity between the first video and the second video.

[0013] The acquisition module is used to acquire a second similarity parameter based on multiple first spatiotemporal features and multiple second spatiotemporal features. The second similarity parameter is used to represent the degree of similarity between the first video and the second video at the video frame level.

[0014] The determination module is used to fuse the first similarity parameter and the second similarity parameter to obtain a target similarity parameter. In response to the target similarity parameter being greater than a first threshold, it is determined that there are repeated segments between the first video and the second video. The target similarity parameter is used to represent the overall similarity between the first video and the second video.

[0015] In some embodiments, for any one of the first video and the second video, the acquisition module is configured to divide each video frame of the video into multiple sub-video frames, acquire the spatial features of the multiple video frames based on the spatial information between the multiple sub-video frames in each video frame, and acquire the global spatiotemporal features of the video and the spatiotemporal features of the multiple video frames based on the spatial features of the multiple video frames and the temporal information between the multiple video frames.

[0016] In some embodiments, the acquisition module is configured to acquire a similarity matrix based on a plurality of first spatiotemporal features and a plurality of second spatiotemporal features, wherein each element in the similarity matrix is ​​used to represent the degree of similarity between each first video frame and each second video frame; and to acquire the second similarity parameter based on the maximum value of each row in the similarity matrix and the number of first video frames.

[0017] In some embodiments, the determining module is configured to perform a weighted summation of the first similarity parameter and the second similarity parameter based on a first weight and a second weight to obtain the target similarity parameter.

[0018] In some embodiments, the first weight and the second weight are trained based on multiple sample videos and corresponding sample labels;

[0019] The sample videos include multiple original videos and videos obtained by transforming the original videos. Any original video and any video obtained by transforming the original video belong to the same category. The sample label is used to indicate the category of the corresponding sample video.

[0020] In some embodiments, the acquisition module is further configured to acquire a similarity vector, wherein each element in the similarity vector is the maximum value of each row in the similarity matrix, and each element in the similarity vector is used to represent the degree of similarity between the second video and each first video frame in the first video;

[0021] The determining module is also used to determine the first video frame corresponding to the element in the similarity vector that is greater than or equal to the second threshold as a duplicate video frame.

[0022] In some embodiments, the acquisition module is used to acquire the first global spatiotemporal feature, multiple first spatiotemporal features, the second global spatiotemporal feature, and multiple second spatiotemporal features based on the video feature extraction model, the first video, and the second video; wherein the video feature extraction model includes a spatial sub-model and a temporal sub-model, and the spatial sub-model and the temporal sub-model are composed of multiple Transformers.

[0023] In some embodiments, the video feature extraction model is trained based on the plurality of sample videos and corresponding sample labels;

[0024] The acquisition module is used to acquire a target number of sample videos and corresponding sample labels from the multiple sample videos; randomly combine the target number of sample videos to obtain multiple sample video pairs; acquire the first sample global spatiotemporal features, multiple first sample spatiotemporal features, second sample global spatiotemporal features, and multiple second sample spatiotemporal features corresponding to the multiple sample video pairs; acquire multiple first sample similarity parameters based on the first sample global spatiotemporal features and the second sample global spatiotemporal features corresponding to the multiple sample video pairs; acquire multiple second sample similarity parameters based on the multiple first sample spatiotemporal features and the multiple second sample spatiotemporal features corresponding to the multiple sample video pairs; acquire multiple sample target similarity parameters based on the multiple first sample similarity parameters and the multiple second sample similarity parameters; and train the video feature extraction model based on the multiple sample target similarity parameters.

[0025] On one hand, a computer device is provided, the computer device including one or more processors and one or more memories, the one or more memories storing at least one computer program, the at least one computer program being loaded and executed by the one or more processors to implement the operations performed by the video retrieval method.

[0026] On the one hand, a computer-readable storage medium is provided, which stores at least one computer program, which is loaded and executed by a processor to implement the operations performed by the video retrieval method.

[0027] On one hand, a computer program product is provided, comprising at least one computer program stored in a computer-readable storage medium. A processor of a computer device reads the at least one computer program from the computer-readable storage medium and executes the at least one computer program, causing the computer device to perform the operations performed by the video retrieval method.

[0028] The technical solution provided in this application comprehensively considers both video-level and video frame-level spatiotemporal information to obtain a target similarity parameter. This target similarity parameter can more accurately represent the degree of similarity between the first video and the second video. Therefore, based on this target similarity parameter, the accuracy of retrieving duplicate segments between the first video and the second video can be improved. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of a repeating video segment provided in an embodiment of this application;

[0031] Figure 2 This is a schematic diagram of a video editing transformation provided in an embodiment of this application;

[0032] Figure 3 This is a schematic diagram of the implementation environment of a video retrieval method provided in an embodiment of this application;

[0033] Figure 4 This is a flowchart of a video retrieval method provided in an embodiment of this application;

[0034] Figure 5 This is a flowchart of a video retrieval method provided in an embodiment of this application;

[0035] Figure 6 This is a flowchart of a video retrieval method provided in an embodiment of this application;

[0036] Figure 7 This is a schematic diagram of a video feature extraction model provided in an embodiment of this application;

[0037] Figure 8 This is a schematic diagram illustrating the location of repeated segments between a first video and a second video, provided in an embodiment of this application.

[0038] Figure 9 This is a flowchart illustrating the training process of a video feature extraction model provided in an embodiment of this application.

[0039] Figure 10 This is a schematic diagram of a video transformation provided in an embodiment of this application;

[0040] Figure 11 This is a flowchart of a video retrieval method provided in an embodiment of this application;

[0041] Figure 12 This is a flowchart of a video retrieval method provided in an embodiment of this application;

[0042] Figure 13 This is a schematic diagram of the structure of a video retrieval device provided in an embodiment of this application;

[0043] Figure 14 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0046] To facilitate understanding of the technical processes involved in the embodiments of this application, some terms used in the embodiments of this application are explained below:

[0047] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0048] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0049] The technical solutions provided in this application can also be combined with cloud technology. For example, the trained video feature extraction model can be deployed on a cloud server. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0050] Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and cryptographic algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying platform, a platform product service layer, and an application service layer.

[0051] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0052] Convolutional Neural Networks (CNNs) are a class of feedforward neural networks that include convolutional computations and have a deep structure. Their artificial neurons can respond to a portion of the surrounding units within their coverage area. They consist of one or more convolutional layers and a fully connected layer at the top (corresponding to classic neural networks), and also include associated weights and pooling layers. They are one of the representative algorithms of deep learning.

[0053] The Rectified Linear Unit (ReLU), also known as the Modified Linear Unit, is a commonly used activation function in artificial neural networks. It usually refers to nonlinear functions represented by the ramp function and its variants.

[0054] The following describes the application scenarios of the video retrieval method proposed in the embodiments of this application.

[0055] The video retrieval method provided in this application embodiment can retrieve segments in a first video that are repeated in a second video. These repeated segments include not only segments in the first video that are completely identical to those in the second video, but also segments in the first video that are identical to those in the transformed second video, for example... Figure 1 As shown, Figure 1 The image shows segments in the first video that are repeated in the second video. It can be seen that these segments are not exactly identical to those in the second video, but rather are the same as segments in the second video that have undergone flipping, resizing, and other transformations. Currently, methods for transforming videos mainly include photometric transformation, geometric transformation, and editing transformation, such as... Figure 2 As shown, Figure 2 An example of video editing transformation is shown. Specifically, this video retrieval method can be used in the following three scenarios:

[0056] (1) Search for videos of interest.

[0057] If a user wants to retrieve videos that have duplicate segments with a specific video, the user can upload the video to their terminal, which then sends the video to the server. The server, based on the video retrieval method provided in this application, retrieves videos with duplicate segments from its database. Furthermore, the method provided in this application can also locate the duplicate segments within the retrieved videos, improving the efficiency of the user's video search.

[0058] (2) Repeated video attacks.

[0059] For video applications, especially short video applications, the presence of too many duplicate videos on the platform not only negatively impacts user experience but also leads to copyright disputes. The method provided in this application can reduce duplicate videos on the platform in the following two ways:

[0060] On the one hand, for videos uploaded by users in short video applications, the server detects the video before publishing it. Based on the method provided in this application, it can determine whether there are duplicate segments between the video uploaded by the user and any video in the database. Furthermore, the method provided in this application can also locate duplicate segments in the video uploaded by the user and prompt the user to make modifications, thereby reducing duplicate videos published on the platform.

[0061] On the other hand, for the videos in the platform's database, the server can detect the videos in the database. During the detection process, multiple pairs of videos are retrieved from the database. For any pair of videos, if it is determined, based on the method proposed in this application, that there are duplicate segments between one video and another video, then one of the videos is deleted from the database, thereby reducing the number of videos with duplicate content on the platform.

[0062] (3) Advertising placement.

[0063] The method provided in this application can be applied to advertising scenarios in the following two ways:

[0064] On the one hand, the server can detect the videos in the database. During the detection process, multiple pairs of advertising videos are retrieved from the database. For any pair of advertising videos, if it is determined that there is a duplicate segment between one advertising video and another advertising video based on the method proposed in this application, one of the advertising videos is deleted from the database, thereby deduplicating the advertising videos in the database.

[0065] On the other hand, during the advertising process, advertising videos are often recommended to users based on advertising recommendation models. Adding the features of the advertising videos obtained based on the method provided in this application to the advertising recommendation model can improve the performance of the advertising recommendation model and enhance the advertising delivery effect.

[0066] It is understood that the video retrieval method provided in this application is not limited to the above-mentioned application scenarios. In some embodiments, the video retrieval method can also be applied to other computer vision scenarios, such as image retrieval, image matching, and multimodal media resource retrieval.

[0067] The implementation environment of the video retrieval method provided in this application is described below.

[0068] Figure 3This is a schematic diagram illustrating the implementation environment of a video retrieval method provided in an embodiment of this application. See also... Figure 3 The implementation environment includes: terminal 301 and server 302, which are directly or indirectly connected via wired or wireless network.

[0069] Terminal 301 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., and this application embodiment does not limit this. Terminal 301 can run various applications that support video functions, such as short video applications, social applications, etc. Users can upload videos through the applications running on terminal 301, and terminal 301 can send the videos to server 302.

[0070] Server 302 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Server 302 may be associated with a database used to store multiple videos for video retrieval.

[0071] In some embodiments, the terminal 301 and server 302 described above can function as nodes in a blockchain system.

[0072] Optionally, terminal 301 generally refers to one of multiple terminals; this embodiment only uses terminal 301 as an example. Those skilled in the art will understand that the number of terminals 301 can be greater. For example, there may be dozens or hundreds, or even more, terminals 301. In this case, the implementation environment of the video retrieval method may also include other terminals. This application embodiment does not limit the number of terminals or the type of device.

[0073] Based on the above implementation environment, Figure 4 This is a flowchart of a video retrieval method provided in an embodiment of this application, such as... Figure 4 As shown, the method includes the following steps.

[0074] 401. The server obtains the first global spatiotemporal features of the first video, the first spatiotemporal features of multiple first video frames in the first video, the second global spatiotemporal features of the second video, and the second spatiotemporal features of multiple second video frames in the second video.

[0075] In some embodiments, the server first obtains a first video and a second video, and then obtains features of the first video and features of the second video. The first video is the video to be compared with the second video, and the second video is a query video. The purpose of comparing the first video and the second video is to determine whether there are duplicate segments between them.

[0076] In this context, for any one of the first and second videos, the global spatiotemporal features of the video are video-level features used to represent the overall spatiotemporal information of the video, while the spatiotemporal features of a video frame are video-frame-level features used to represent the spatiotemporal information of the video frame. Specifically, the overall spatiotemporal information of the video refers to the semantic dependencies between multiple video frames and the spatial structure information of each video frame. The spatiotemporal information of a video frame refers to the semantic dependencies between that video frame and other video frames, and the spatial structure relationships between multiple sub-video frames of that video frame.

[0077] 402. Based on the first global spatiotemporal feature and the second global spatiotemporal feature, the server obtains a first similarity parameter, which is used to represent the video-level similarity between the first video and the second video.

[0078] The larger the value of the first similarity parameter, the higher the video-level similarity between the first video and the second video; the smaller the value of the first similarity parameter, the lower the video-level similarity between the first video and the second video.

[0079] 403. The server obtains a second similarity parameter based on multiple first spatiotemporal features and multiple second spatiotemporal features. The second similarity parameter is used to represent the degree of similarity between the first video and the second video at the video frame level.

[0080] The larger the value of the first similarity parameter, the higher the similarity between the first video frame and the second video frame; the smaller the value of the first similarity parameter, the lower the similarity between the first video frame and the second video frame.

[0081] 404. The server fuses the first similarity parameter and the second similarity parameter to obtain a target similarity parameter. In response to the target similarity parameter being greater than a first threshold, it is determined that there are duplicate segments between the first video and the second video. The target similarity parameter is used to represent the overall similarity between the first video and the second video.

[0082] Because this target similarity parameter takes into account both video-level and video frame-level spatiotemporal information, it can more accurately represent the degree of similarity between the first video and the second video.

[0083] The technical solution provided in this application comprehensively considers both video-level and video frame-level spatiotemporal information to obtain a target similarity parameter. This target similarity parameter can more accurately represent the degree of similarity between the first video and the second video. Therefore, based on this target similarity parameter, the accuracy of retrieving duplicate segments between the first video and the second video can be improved.

[0084] The video retrieval method provided in this application will be described below through several embodiments based on the three application scenarios mentioned above. This application's embodiments take the scenario of combating repeated videos as an example, combined with... Figure 5 and Figure 6 This video retrieval method will be explained. Figure 5 and Figure 6 This is a flowchart of a video retrieval method provided in an embodiment of this application, such as... Figure 5 As shown, the method includes the following steps.

[0085] 501. In response to the user's video upload operation, the terminal sends a deduplication request to the server, which carries the first video.

[0086] In some embodiments, if it is necessary to detect whether there are duplicate segments between the video uploaded by the user and the video in the database, the terminal responds to the user uploading a video on the platform, obtains the video uploaded by the user as the first video, and sends a deduplication request to the server. The deduplication request carries the first video and is used to instruct the server to perform a deduplication on the first video.

[0087] Optionally, the plagiarism check request carries a video identifier, which is used to indicate any video that has been published on the platform, and the plagiarism check request is used to instruct the server to perform a plagiarism check on the video indicated by the video identifier.

[0088] Video platform administrators can use the methods described above to detect user-uploaded videos or videos already published on the platform, thereby determining whether there are duplicate segments between the user-uploaded videos and the videos in the database.

[0089] 502. The server responds to the terminal's plagiarism check request and retrieves the second video from the database.

[0090] In some embodiments, in response to the plagiarism check request, the server retrieves the first video from the plagiarism check request and randomly selects a second video from a plurality of videos in the database.

[0091] In some embodiments, the plagiarism check request also carries video information of a first video, including the type and duration of the first video. In response to the plagiarism check request, the server retrieves the first video and its video information from the request, and based on this video information, retrieves a second video from the database. For example, the server may, based on the video information, retrieve videos in the database that are of the same type as the first video as the second video; or, the server may, based on the video information, determine videos in the database whose duration differs from the duration of the first video from a target threshold, and retrieve those videos as the second video.

[0092] Optionally, if the plagiarism check request carries a video identifier, the server will retrieve the video indicated by the video identifier as the first video and retrieve the second video from the database.

[0093] It should be noted that the server can retrieve multiple second videos from the database and search for duplicate segments between the first video and each second video in a serial or parallel manner.

[0094] 503. The server obtains the first global spatiotemporal features of the first video, the first spatiotemporal features of multiple first video frames in the first video, the second global spatiotemporal features of the second video, and the second spatiotemporal features of multiple second video frames in the second video.

[0095] In some embodiments, the server first obtains first spatial features of multiple first video frames and second spatial features of multiple second video frames. Then, based on the multiple first spatial features, multiple second spatial features, temporal information of the multiple first video frames, and temporal information of the multiple second video frames, it obtains first global spatiotemporal features, second global spatiotemporal features, multiple first spatiotemporal features, and multiple second spatiotemporal features. The above process is described below based on steps 503A to 503D:

[0096] 503A. The server divides each first video frame of the first video into multiple sub-video frames, and obtains the first spatial features of the multiple first video frames based on the spatial information between the multiple sub-video frames in each first video frame.

[0097] In some embodiments, for any first video frame, the server uniformly divides the first video frame into multiple n*n sub-video frames, each sub-video frame including a partial view of the first video frame, where n is an integer greater than 0. These multiple sub-video frames are then input into the spatial sub-model of the video feature extraction model. For example... Figure 7As shown, the spatial sub-model consists of multiple Transformers, and each Transformer contains a multi-head self-attention unit. Therefore, by inputting the multiple sub-video frames into the spatial sub-model, the spatial structural relationship between each sub-video frame and other sub-videos can be obtained based on the multiple multi-head self-attention units. That is, the first spatial feature of the first video frame can be obtained, which is used to represent the spatial information of the first video frame.

[0098] 503B. The server obtains the first global spatiotemporal features and the first spatiotemporal features of the multiple first video frames based on the first spatial features of the multiple first video frames and the temporal information between the multiple first video frames.

[0099] In some embodiments, the server concatenates a target vector with multiple first spatial features based on the target order to obtain concatenated features. Concatenation based on the target order means that the target vector is used as the first feature in the concatenated features, and the multiple first spatial features are concatenated with the target vector sequentially according to the order of the corresponding first video frames in the first video. The server inputs the concatenated features into the temporal sub-model of the video feature extraction model. Based on multiple Transformers in the temporal sub-model, the concatenated features are processed. The first feature in the output data, i.e., the output feature corresponding to the target vector, is used as the first global spatiotemporal feature. The other features in the output data of the temporal sub-model, i.e., the output features corresponding to the multiple first spatial features, are used as multiple first spatiotemporal features. For example, if the concatenated feature is an S×T dimensional vector, then the first row of the concatenated feature is the target vector, and the second to S rows are multiple first spatial features. This concatenated vector is input into the temporal sub-model. If the output data is an S×J dimensional vector, then the first row of the output data is the first global spatiotemporal feature, and the second to S rows are multiple first spatiotemporal features. Where S, T, and J are integers greater than 0.

[0100] The target vector is a randomly generated vector by the server, used to fuse the spatiotemporal information of multiple video frames. The target vector has the same dimension as the multiple spatial features. Optionally, the target vector is a [CLS] embedding vector.

[0101] For example, the process by which the server processes the stitched features based on multiple Transformers in the temporal sub-model is described. In the temporal sub-model, each Transformer contains a multi-head self-attention unit. For the first Transformer, the server uses the stitched features as input data. Based on the multi-head attention unit in the first Transformer, it extracts the relationships between the features in the stitched vector. By obtaining the relationships between each first spatial feature and other first spatial features, the output data of the first Transformer contains the spatial and temporal information of the corresponding first video frame for each first spatial feature. By obtaining the relationship between the target vector and multiple first spatial features, the multiple first spatial features are fused, so that the output data of the first Transformer contains the spatial information of multiple first video frames for the output feature corresponding to the target vector.

[0102] Furthermore, for any Transformer other than the first Transformer, the server uses the output data of the previous Transformer as the input data of this Transformer. Since the output feature corresponding to each first spatial feature in the output data of the previous Transformer contains spatial and temporal information of the corresponding first video frame, the server extracts the relationship between the features in the output data of the previous Transformer based on the multi-head attention unit in this Transformer. This makes the output feature corresponding to the target vector in the output data of this Transformer contain spatial and temporal information of multiple first video frames, and the output feature corresponding to each first spatial feature contains richer spatial and temporal information. The server uses the output feature corresponding to the target vector in the output data of the last Transformer as the first global spatiotemporal feature, and the output features of multiple first spatial features as multiple first spatiotemporal features.

[0103] This temporal sub-model incorporates features from other video frames into the first spatial features of each first video frame, thereby acquiring the temporal information of the first video frame. This enriches the features of each first video frame, improving video retrieval accuracy. Furthermore, this temporal sub-model can fuse the spatial and temporal information of multiple first video frames through multiple Transformers to obtain the first global spatiotemporal features of the first video, further enhancing video retrieval accuracy.

[0104] In some embodiments, for step 503B, the server inputs the location information of multiple video frames, the spatial features of multiple video frames, and the target vector into the temporal sub-model to obtain global spatiotemporal features and spatiotemporal features of multiple video frames. The location information is used to represent the chronological order of the corresponding video frames in the video. For example, the server first concatenates each first spatial feature with the location information of the corresponding first video frame, and concatenates the target vector with an invalid location information to obtain multiple first concatenated features. Then, according to the target order, the multiple first concatenated features are further concatenated to obtain second concatenated features. The second concatenated features are input into the temporal sub-model, and the output features corresponding to the first concatenated features of the target vector are used as global spatiotemporal features. The output features corresponding to the first concatenated features of the multiple spatial features are used as spatiotemporal features of the multiple video frames. By inputting the location information of the first video frame into the temporal sub-model, the obtained features contain richer video information, thereby further improving the accuracy of video retrieval. Optionally, the location information is the frame number or timestamp of the video frame, etc.

[0105] 503C. The server divides each second video frame into multiple sub-video frames and obtains the spatial features of the multiple sub-video frames based on the spatial information between them.

[0106] In some embodiments, step 503C is the same as step 503A described above, and will not be repeated here.

[0107] Based on the spatial features of the multiple second video frames and the temporal information between the multiple video frames, the 503D server obtains the second global spatiotemporal features and the spatiotemporal features of the multiple second video frames.

[0108] In some embodiments, step 503D is the same as step 503B described above, and will not be repeated here.

[0109] It should be noted that the above processing of the first video and the second video can be performed simultaneously or in a certain order, and this application embodiment does not limit this.

[0110] 504. Based on the first global spatiotemporal feature and the second global spatiotemporal feature, the server obtains a first similarity parameter, which is used to represent the video-level similarity between the first video and the second video.

[0111] In some embodiments, the server obtains the first similarity parameter based on the first global spatiotemporal feature and the second global spatiotemporal feature using the method shown in formula (1).

[0112]

[0113] Among them, Sv2v V represents the first similarity parameter. A V represents the first global spatiotemporal feature. B This represents the second global spatiotemporal feature. Formula (1) is also known as cosine similarity.

[0114] 505. The server obtains a second similarity parameter based on multiple first spatiotemporal features and multiple second spatiotemporal features. The second similarity parameter is used to represent the degree of similarity between the first video and the second video at the video frame level.

[0115] In some embodiments, step 505 is implemented based on steps 505A to 505B:

[0116] 505A. The server obtains a similarity matrix based on multiple first spatiotemporal features and multiple second spatiotemporal features. Each element in the similarity matrix is ​​used to represent the degree of similarity between each first video frame and each second video frame.

[0117] In some embodiments, the server obtains a similarity matrix, where each element is a first spatiotemporal feature and a second spatiotemporal feature, obtained based on the above formula (1). That is, the similarity between a first video frame and a second video frame. The dimension of the similarity matrix is ​​the product of the number of first video frames and the number of second video frames. For example, if the first video includes N first video frames and the second video includes M second video frames, that is, the first video has N first spatiotemporal features and the second video has M second spatiotemporal features, where M and N are integers greater than 0, then the server obtains an N×M dimension similarity matrix P based on these M first spatiotemporal features and N second spatiotemporal features. A2B The element P in the matrix ij This indicates the degree of similarity between the i-th first video frame and the j-th second video frame.

[0118] 505B. The server obtains the second similarity parameter based on the maximum value of each row in the similarity matrix and the number of the first video frames.

[0119] The maximum value in each row refers to the element with the largest value in each row of the similarity matrix.

[0120] In some embodiments, the server obtains the second similarity parameter based on the similarity matrix as shown in formula (2).

[0121]

[0122] Among them, S f2f The second similarity parameter is represented by N, which represents the number of the first video frames, and P represents the number of the second similarity parameter. A2B Let P represent the similarity matrix.A2B (i,:) represents the i-th row in the similarity matrix.

[0123] 506. The server fuses the first similarity parameter with the second similarity parameter to obtain the target similarity parameter.

[0124] In some embodiments, the server performs a weighted summation of the first similarity parameter and the second similarity parameter based on a first weight and a second weight to obtain the target similarity parameter. Optionally, the target similarity parameter is denoted by S. The first weight and the second weight are preset weight coefficients in the server, and their values ​​can be set based on actual needs; this embodiment does not limit this.

[0125] In some embodiments, the first weight and the second weight are trained based on multiple sample videos and corresponding sample labels, and the training process is detailed in [link to documentation]. Figure 9 A corresponding implementation example. By training to obtain the first weight and the second weight, the accuracy of the target similarity coefficient obtained based on the first weight and the second weight can be improved, thereby increasing the accuracy of retrieving duplicate segments between the first video and the second video.

[0126] 507. The server responds to the target similarity parameter being greater than or equal to a first threshold and determines that there are duplicate segments between the first video and the second video.

[0127] The first threshold is a parameter preset in the server and can be set based on actual needs. This application embodiment does not limit this. Optionally, the first threshold is represented by T1.

[0128] In some embodiments, the server can locate duplicate segments between the first video and the second video, such as... Figure 8 As shown, step 507 further includes: in response to determining that there are segments in the first video that overlap with the second video, the server obtains a similarity vector, where each element of the similarity vector is the maximum value of each row in the similarity matrix, and each element of the similarity vector represents the degree of similarity between the second video and each first video frame in the first video; the first video frames corresponding to elements in the similarity vector that are greater than or equal to a second threshold are identified as overlapping video frames, optionally denoted by T2. Connecting consecutive overlapping video frames yields overlapping segments between the first and second videos, and the frame number or timestamp corresponding to the overlapping video frame in the first video is the position of the overlapping segment in the first video. Optionally, the aforementioned first threshold and second threshold can also be called similarity thresholds.

[0129] 508. The server sends a duplicate message to the terminal, which indicates that there are duplicate segments between the video uploaded by the user and the video in the database.

[0130] In some embodiments, if the server locates duplicate segments between the first and second videos, the deduplication message carries the location of the duplicate segments. The terminal receives the deduplication message from the server and displays a prompt message on the video application interface, which prompts the user to modify the video. If the server also sends the location of the duplicate segments to the terminal, the terminal displays an instruction message and the location of the duplicate segments on the video application interface.

[0131] Optionally, if the first video is a video already published on the platform, the server can delete the video. Alternatively, the server can send the identifiers of the first video and the second video to the terminal. The terminal displays the identifiers of the two videos on the platform management interface. Platform administrators can retrieve the corresponding videos from the database based on the identifiers, review the two videos, and decide whether to delete the first video.

[0132] The training process of the above video feature extraction model is described below, such as... Figure 9 As shown, the training process includes the following two parts:

[0133] (1) Sample video preparation.

[0134] The video feature extraction model is trained based on multiple sample videos and corresponding sample labels. The multiple sample videos include multiple original videos and videos obtained by transforming the original videos. Any original video and a video obtained by transforming the original video are called family videos. The sample label is used to represent the family of the corresponding sample videos.

[0135] Methods for transforming the original video include: photometric transformation, set transformation, and editing transformation. For example... Figure 10 As shown, photometric transformations include brightness transformation, contrast transformation, hue transformation, saturation transformation, and gamma transformation; geometric transformations include horizontal flipping, rotation, cropping, resizing, and translation; and editing transformations include adding a blurred background, adding an icon (logo), and picture-in-picture transformations.

[0136] (2) Model training.

[0137] The training process of this video feature extraction model is achieved through multiple iterations. Each iteration of the training process includes the following steps (a) to (g):

[0138] (a) The server obtains a target number of sample videos and corresponding sample tags from multiple sample videos, and randomly combines the target number of sample videos to obtain multiple sample video pairs.

[0139] Specifically, if the two sample videos in a sample video pair are from the same family, then the sample video pair is a positive sample pair; if the two sample videos in a sample video pair are not from the same family, then the sample video pair is a negative sample pair.

[0140] (b) The server obtains the first sample global spatiotemporal features, multiple first sample spatiotemporal features, second sample global spatiotemporal features and multiple second sample spatiotemporal features corresponding to the multiple sample video pairs using a method similar to step 503 above.

[0141] (c) Based on the global spatiotemporal features of the first sample and the global spatiotemporal features of the second sample corresponding to the multiple sample video pairs, the server obtains multiple first sample similarity parameters by means of the same method as in step 504 above.

[0142] (d) Based on the spatiotemporal features of multiple first samples and multiple spatiotemporal features of multiple sample videos, the server obtains multiple similarity parameters of the second samples using a method similar to step 505 above.

[0143] (e) The server performs a weighted summation of multiple first sample similarity parameters and multiple second sample similarity parameters based on the first weight and the second weight to obtain multiple sample target similarity parameters.

[0144] (f) The server obtains the depth metric loss based on the multiple sample target similarity parameters, the corresponding sample labels, and the depth metric learning (DML) function. Optionally, the depth metric loss function is a triplet loss function, a contrastive loss function, or other DML functions; this embodiment of the application does not limit this.

[0145] (g) The server trains the video feature extraction model based on this depth metric loss. Since the DML loss accurately represents the similarity between sample video pairs, and the DML loss function includes a margin constraint—the minimum similarity between negative sample pairs—if the similarity between negative sample pairs is less than this margin constraint, the depth metric loss will increase. Therefore, training the video feature extraction model based on this depth metric loss increases the similarity between positive sample pairs and decreases the similarity between negative sample pairs. This allows the video features obtained from the model to accurately represent the similarity between videos, thereby improving video retrieval accuracy.

[0146] In some embodiments, if the first weight and the second weight are obtained based on the sample video and the corresponding sample label in step 505 above, then step (g) in the training process above further includes: the server training the first weight and the second weight based on the depth metric loss. The first weight and the second weight obtained based on the depth metric loss can more accurately represent the proportion of the first similarity parameter and the second similarity parameter, thereby improving the accuracy of video retrieval.

[0147] The technical solution provided in this application comprehensively considers both video-level and video frame-level spatiotemporal information to obtain a target similarity parameter. This target similarity parameter can more accurately represent the degree of similarity between the first video and the second video. Therefore, based on this target similarity parameter, the accuracy of retrieving duplicate segments between the first video and the second video can be improved.

[0148] This application's embodiments take the scenario of searching for videos of interest as an example, combined with... Figure 11 This video retrieval method will be explained. Figure 11 This is a flowchart of a video retrieval method provided in an embodiment of this application, such as... Figure 11 As shown, the method includes the following steps.

[0149] 1101. In response to the user's video upload operation, the terminal sends a retrieval request to the server, which carries the second video.

[0150] In some embodiments, if a user wants to retrieve videos that have duplicate segments with a certain video, the user can upload a video on the terminal. In response to the user's video upload operation, the terminal obtains the uploaded video as a second video and sends a retrieval request to the server. The retrieval request carries the second video and is used to instruct the server to return videos that have duplicate segments with the second video.

[0151] Optionally, users can select videos of interest through the interface of a video application. For example, the terminal displays links to multiple videos and corresponding selection controls on the interface of the video application. In response to the user's click on the selection control of any video, the terminal sends a search request to the server, which carries the identifier of the video.

[0152] 1102. The server responds to the terminal's retrieval request and retrieves the first video from the database.

[0153] In some embodiments, the server responds to the retrieval request by retrieving the second video from the retrieval request and retrieves the first video from the database using a method similar to that in step 1002.

[0154] Optionally, if the retrieval request carries a video identifier, the server obtains the video indicated by the video identifier as the second video, and retrieves the first video from the database using a method similar to step 1002.

[0155] It should be noted that the server can retrieve multiple first videos from the database and retrieve duplicate segments between each first video and a second video in a serial or parallel manner.

[0156] 1103. The server obtains the first global spatiotemporal features of the first video, the first spatiotemporal features of multiple first video frames in the first video, the second global spatiotemporal features of the second video, and the second spatiotemporal features of multiple second video frames in the second video.

[0157] 1104. Based on the first global spatiotemporal feature and the second global spatiotemporal feature, the server obtains a first similarity parameter, which is used to represent the video-level similarity between the first video and the second video.

[0158] 1105. The server obtains a second similarity parameter based on multiple first spatiotemporal features and multiple second spatiotemporal features. The second similarity parameter is used to represent the degree of similarity between the first video and the second video at the video frame level.

[0159] 1106. The server fuses the first similarity parameter with the second similarity parameter to obtain the target similarity parameter.

[0160] 1107. The server responds to the target similarity parameter being greater than or equal to a first threshold and determines that there are duplicate segments between the first video and the second video.

[0161] In some embodiments, steps 1103 to 1107 are the same as steps 503 to 507, and will not be repeated here.

[0162] 1108. The server sends a link to a video that has a duplicate segment with the second video to the terminal. This video link is used to play the video based on a click operation.

[0163] In some embodiments, if the server has located the duplicate segment, it also sends the location of the duplicate segment to the terminal. The terminal receives a link to the video from the server and displays the video link in the video application's interface. If the server has also sent the location of the duplicate segment, the terminal displays both the video link and the location of the duplicate segment in the video application's interface.

[0164] Optionally, the server simultaneously sends links to multiple videos that have overlapping segments with the second video, along with the locations of the overlapping segments, to the terminal. The terminal then displays the links to these multiple videos and the locations of the overlapping segments on the interface of the video application.

[0165] The technical solution provided in this application comprehensively considers both video-level and video frame-level spatiotemporal information to obtain a target similarity parameter. This target similarity parameter can more accurately represent the degree of similarity between the first video and the second video. Therefore, based on this target similarity parameter, the accuracy of retrieving duplicate segments between the first video and the second video can be improved.

[0166] This application's embodiments take an advertising placement scenario as an example, combined with... Figure 12 This video retrieval method will be explained. Figure 12 This is a flowchart of a video retrieval method provided in an embodiment of this application, such as... Figure 12 As shown, the method includes the following steps.

[0167] 1201. The terminal sends a deduplication request to the server, which instructs the server to delete duplicate advertising videos from the database.

[0168] In some embodiments, the administrator of the advertising delivery system can trigger the process of deleting duplicate advertising videos from the database through the management interface of the terminal. In response to the administrator's triggering operation for deduplication of advertising videos on the management interface, the terminal sends a deduplication request to the server, which instructs the server to delete duplicate advertising videos from the database.

[0169] 1202. In response to the terminal's deduplication request, the server retrieves the first and second videos from the database.

[0170] In some embodiments, in response to a terminal's deduplication request, the server randomly retrieves multiple pairs of ad videos from the database; alternatively, the server retrieves multiple pairs of ad videos of the same type based on the ad video tags; or, the server retrieves multiple pairs of ad videos whose duration difference is less than a target threshold based on the ad video duration. For any pair of ad videos, one ad video is designated as the first video, and the other ad video is designated as the second video.

[0171] Optionally, the server can periodically trigger a process to deduplicate the ad videos in the database. That is, at target time intervals, the server retrieves multiple pairs of ad videos from the database. For any pair of ad videos, one ad video is designated as the first video and the other ad video as the second video, so as to retrieve duplicate segments between the first video and the second video.

[0172] It should be noted that for the multiple pairs of advertising videos obtained, the server can retrieve the duplicate segments between each pair of advertising videos in a serial or parallel manner.

[0173] 1203. The server obtains the first global spatiotemporal features of the first video, the first spatiotemporal features of multiple first video frames in the first video, the second global spatiotemporal features of the second video, and the second spatiotemporal features of multiple second video frames in the second video.

[0174] 1204. Based on the first global spatiotemporal feature and the second global spatiotemporal feature, the server obtains a first similarity parameter, which is used to represent the video-level similarity between the first video and the second video.

[0175] 1205. The server obtains a second similarity parameter based on multiple first spatiotemporal features and multiple second spatiotemporal features. The second similarity parameter is used to represent the degree of similarity between the first video and the second video at the video frame level.

[0176] 1206. The server fuses the first similarity parameter with the second similarity parameter to obtain the target similarity parameter.

[0177] 1207. The server responds to the target similarity parameter being greater than or equal to a first threshold and determines that there are duplicate segments between the first video and the second video.

[0178] In some embodiments, steps 1203 to 1207 are the same as steps 503 to 507, and will not be described again here.

[0179] 1208. The server will delete either the first or second video from the database.

[0180] Optionally, the server sends the identifiers of the first video and the second video to the terminal. The terminal displays the identifiers of the two videos on the management interface of the advertising delivery system. The administrators of the advertising delivery system can retrieve the corresponding videos from the database based on the identifiers, review the two videos, and decide whether to delete one of them.

[0181] The technical solution provided in this application comprehensively considers both video-level and video frame-level spatiotemporal information to obtain a target similarity parameter. This target similarity parameter can more accurately represent the degree of similarity between the first video and the second video. Therefore, based on this target similarity parameter, the accuracy of retrieving duplicate segments between the first video and the second video can be improved.

[0182] The above Figures 5 to 12 The various embodiments illustrate the video retrieval method based on different application scenarios. The main difference between the methods in different application scenarios lies in the process by which the server obtains the first video and the second video. It should be understood that in other application scenarios, the server can also obtain the first video and the second video in other ways, and this application embodiment does not limit this.

[0183] Figure 13 This is a schematic diagram of the structure of a video retrieval device provided in an embodiment of this application. See also... Figure 13 The device includes an acquisition module 1301 and a determination module 1302.

[0184] The acquisition module 1301 is used to acquire the first global spatiotemporal features of the first video, the first spatiotemporal features of multiple first video frames in the first video, the second global spatiotemporal features of the second video, and the second spatiotemporal features of multiple second video frames in the second video;

[0185] The acquisition module 1301 is used to acquire a first similarity parameter based on the first global spatiotemporal feature and the second global spatiotemporal feature. The first similarity parameter is used to represent the video-level similarity between the first video and the second video.

[0186] The acquisition module 1301 is used to acquire a second similarity parameter based on multiple first spatiotemporal features and multiple second spatiotemporal features. The second similarity parameter is used to represent the degree of similarity between the first video and the second video at the video frame level.

[0187] The determining module 1302 is used to fuse the first similarity parameter and the second similarity parameter to obtain a target similarity parameter. In response to the target similarity parameter being greater than or equal to a first threshold, it is determined that there is a segment in the first video that is repeated with the second video. The target similarity parameter is used to represent the overall similarity between the first video and the second video.

[0188] In some embodiments, for any one of the first video and the second video, the acquisition module 1301 is configured to divide each video frame of the video into multiple sub-video frames, acquire the spatial features of the multiple video frames based on the spatial information between the multiple sub-video frames in each video frame, and acquire the global spatiotemporal features of the video and the spatiotemporal features of the multiple video frames based on the spatial features of the multiple video frames and the temporal information between the multiple video frames.

[0189] In some embodiments, the acquisition module 1301 is configured to acquire a similarity matrix based on a plurality of first spatiotemporal features and a plurality of second spatiotemporal features, wherein each element in the similarity matrix is ​​used to represent the degree of similarity between each first video frame and each second video frame; and to acquire the second similarity parameter based on the maximum value of each row in the similarity matrix and the number of first video frames.

[0190] In some embodiments, the determining module 1302 is used to perform a weighted summation of the first similarity parameter and the second similarity parameter based on the first weight and the second weight to obtain the target similarity parameter.

[0191] In some embodiments, the first weight and the second weight are trained based on multiple sample videos and corresponding sample labels;

[0192] The sample videos include multiple original videos and videos obtained by transforming the original videos. Any original video and any video obtained by transforming the original video belong to the same category. The sample label is used to indicate the category of the corresponding sample video.

[0193] In some embodiments, the acquisition module 1301 is further configured to acquire a similarity vector, wherein each element in the similarity vector is the maximum value of each row in the similarity matrix, and each element in the similarity vector is used to represent the degree of similarity between the second video and each first video frame in the first video.

[0194] The determining module 1302 is further configured to determine the first video frame corresponding to the element in the similarity vector that is greater than or equal to the second threshold as a duplicate video frame.

[0195] In some embodiments, the acquisition module 1301 is used to acquire the first global spatiotemporal feature, multiple first spatiotemporal features, the second global spatiotemporal feature, and multiple second spatiotemporal features based on the video feature extraction model, the first video, and the second video; wherein the video feature extraction model includes a spatial sub-model and a temporal sub-model, and the spatial sub-model and the temporal sub-model are composed of multiple Transformers.

[0196] In some embodiments, the video feature extraction model is trained based on the plurality of sample videos and corresponding sample labels;

[0197] The acquisition module 1301 is used to acquire a target number of sample videos and corresponding sample labels from the multiple sample videos; randomly combine the target number of sample videos to obtain multiple sample video pairs; acquire the first sample global spatiotemporal feature, multiple first sample spatiotemporal features, second sample global spatiotemporal features, and multiple second sample spatiotemporal features corresponding to the multiple sample video pairs; acquire multiple first sample similarity parameters based on the first sample global spatiotemporal feature and the second sample global spatiotemporal feature corresponding to the multiple sample video pairs; acquire multiple second sample similarity parameters based on the multiple first sample spatiotemporal feature and the multiple second sample spatiotemporal feature corresponding to the multiple sample video pairs; acquire multiple sample target similarity parameters based on the multiple first sample similarity parameters and the multiple second sample similarity parameters; and train the video feature extraction model based on the multiple sample target similarity parameters.

[0198] It should be noted that the video retrieval device provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video retrieval device and the video retrieval method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0199] This disclosure provides a computer device for performing the video retrieval method described above. In some embodiments, the computer device is provided as a server. Figure 14 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1400 can vary considerably due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1401 and one or more memories 1402. The one or more memories 1402 store at least one line of program code, which is loaded and executed by the one or more processors 1401 to implement the methods provided in the various method embodiments described above. Of course, the server 1400 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1400 may also include other components for implementing device functions, which will not be elaborated here.

[0200] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one line of program code, which can be executed by a processor to perform the video retrieval method in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0201] In an exemplary embodiment, a computer program product is also provided, comprising at least one computer program stored in a computer-readable storage medium. A processor of a computer device reads the at least one computer program from the computer-readable storage medium and executes the at least one computer program, causing the computer device to perform the operations performed by the video retrieval method described above.

[0202] In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.

[0203] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0204] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A video retrieval method, characterized in that, The method includes: Based on the video feature extraction model, the first video, and the second video, the first global spatiotemporal features of the first video, the first spatiotemporal features of multiple first video frames in the first video, the second global spatiotemporal features of the second video, and the second spatiotemporal features of multiple second video frames in the second video are obtained. Based on the first global spatiotemporal features and the second global spatiotemporal features, a first similarity parameter is obtained, which is used to represent the video-level similarity between the first video and the second video. Based on multiple first spatiotemporal features and multiple second spatiotemporal features, a second similarity parameter is obtained, which is used to represent the degree of similarity between the first video and the second video at the video frame level. The first similarity parameter and the second similarity parameter are fused to obtain a target similarity parameter. In response to the target similarity parameter being greater than or equal to a first threshold, it is determined that there are repeated segments between the first video and the second video. The target similarity parameter is used to represent the overall similarity between the first video and the second video. The video feature extraction model is trained based on multiple sample videos and corresponding sample labels. The training process of the video feature extraction model includes: From the plurality of sample videos, obtain a target number of sample videos and corresponding sample tags; The target number of sample videos are randomly combined to obtain multiple sample video pairs; Obtain the first sample global spatiotemporal features, multiple first sample spatiotemporal features, second sample global spatiotemporal features, and multiple second sample spatiotemporal features corresponding to the multiple sample video pairs; Based on the global spatiotemporal features of the first sample and the global spatiotemporal features of the second sample corresponding to the multiple sample video pairs, multiple first sample similarity parameters are obtained; Based on the spatiotemporal features of the multiple first samples and the multiple spatiotemporal features of the multiple second samples corresponding to the multiple sample video pairs, multiple similarity parameters of the second samples are obtained; Based on the plurality of first sample similarity parameters and the plurality of second sample similarity parameters, a plurality of sample target similarity parameters are obtained; The video feature extraction model is trained based on the similarity parameters of the multiple sample targets.

2. The method according to claim 1, characterized in that, For any one of the first video and the second video, obtaining the global spatiotemporal features of the video and the spatiotemporal features of multiple video frames in the video includes: Each video frame of the video is divided into multiple sub-video frames, and the spatial features of the multiple video frames are obtained based on the spatial information between the multiple sub-video frames in each video frame. Based on the spatial features of the multiple video frames and the temporal information between the multiple video frames, the global spatiotemporal features of the video and the spatiotemporal features of the multiple video frames are obtained.

3. The method according to claim 1, characterized in that, The step of obtaining the second similarity parameter based on multiple first spatiotemporal features and multiple second spatiotemporal features includes: Based on multiple first spatiotemporal features and multiple second spatiotemporal features, a similarity matrix is ​​obtained, wherein each element in the similarity matrix is ​​used to represent the degree of similarity between each first video frame and each second video frame; The second similarity parameter is obtained based on the maximum value of each row in the similarity matrix and the number of the first video frames.

4. The method according to claim 1, characterized in that, The step of fusing the first similarity parameter and the second similarity parameter to obtain the target similarity parameter includes: Based on the first weight and the second weight, the first similarity parameter and the second similarity parameter are weighted and summed to obtain the target similarity parameter.

5. The method according to claim 4, characterized in that, The first weight and the second weight are obtained by training based on multiple sample videos and corresponding sample labels; The plurality of sample videos includes a plurality of original videos and videos obtained by transforming the plurality of original videos. Any original video and any video obtained by transforming the original video belong to the same category. The sample label is used to indicate the category of the corresponding sample video.

6. The method according to any one of claims 1 to 5, characterized in that, After determining that there are duplicate video segments between the first video and the second video in response to the target similarity parameter being greater than a first threshold, the method further includes: Obtain a similarity vector, where each element in the similarity vector is the maximum value of each row in the similarity matrix, and each element in the similarity vector is used to represent the degree of similarity between the second video and each first video frame in the first video; The first video frames corresponding to elements in the similarity vector that are greater than or equal to the second threshold are identified as duplicate video frames.

7. The method according to claim 1, characterized in that, The video feature extraction model includes a spatial sub-model and a temporal sub-model, and the spatial sub-model and the temporal sub-model are composed of multiple Transformers.

8. A video retrieval device, characterized in that, The device includes: The acquisition module is used to acquire, based on a video feature extraction model, a first video, and a second video, a first global spatiotemporal feature of the first video, a first spatiotemporal feature of multiple first video frames in the first video, a second global spatiotemporal feature of the second video, and a second spatiotemporal feature of multiple second video frames in the second video; The acquisition module is used to acquire a first similarity parameter based on the first global spatiotemporal feature and the second global spatiotemporal feature. The first similarity parameter is used to represent the video-level similarity between the first video and the second video. The acquisition module is used to acquire a second similarity parameter based on multiple first spatiotemporal features and multiple second spatiotemporal features. The second similarity parameter is used to represent the degree of similarity between the first video and the second video at the video frame level. A determining module is used to fuse the first similarity parameter and the second similarity parameter to obtain a target similarity parameter. In response to the target similarity parameter being greater than a first threshold, it is determined that there are repeated segments between the first video and the second video. The target similarity parameter is used to represent the overall similarity between the first video and the second video. The video feature extraction model is trained based on multiple sample videos and corresponding sample labels. The acquisition module is further configured to: acquire a target number of sample videos and corresponding sample labels from the plurality of sample videos; randomly combine the target number of sample videos to obtain multiple sample video pairs; acquire a first sample global spatiotemporal feature, multiple first sample spatiotemporal features, a second sample global spatiotemporal feature, and multiple second sample spatiotemporal features corresponding to the multiple sample video pairs; acquire multiple first sample similarity parameters based on the first sample global spatiotemporal features and the second sample global spatiotemporal features corresponding to the multiple sample video pairs; acquire multiple second sample similarity parameters based on the multiple first sample spatiotemporal features and the multiple second sample spatiotemporal features corresponding to the multiple sample video pairs; acquire multiple sample target similarity parameters based on the multiple first sample similarity parameters and the multiple second sample similarity parameters; and train the video feature extraction model based on the multiple sample target similarity parameters.

9. A computer device, characterized in that, The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to perform the operations performed by the video retrieval method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to perform the operations of the video retrieval method as described in any one of claims 1 to 7.

Images