Video processing method, device and computer equipment

Abstract

The present application relates to a video processing method, device and computer equipment. The method comprises: extracting the first, second and third sample video segments from the training video clip, the first and second sample video segments are positive samples of each other, and the first and third sample video segments are negative samples of each other; extracting the first, second and third video features of the first, second and third sample video segments through a video feature extraction model; optimizing the video feature extraction model with the goal of making the first and second video features close, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library until the training end condition is reached; if the training end condition is reached, obtaining a video fingerprint feature extractor based on the optimized video feature extraction model. The use of this method can make the process cost of obtaining a video fingerprint feature extractor low and the obtained video fingerprint feature extractor have a strong representation ability of the video fingerprint features extracted.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video processing method, a video processing device, a computer equipment, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of computer and image processing technology, video has gradually become an important carrier of daily information. Extracting features from videos to obtain effective information has become an important part of video processing technology. In the related technologies of video processing, it is usually necessary to extract the video fingerprint features (also called video fingerprint features) of the related video content. The video fingerprint features refer to the features extracted based on the video content that can uniquely identify the video content. The obtained video fingerprint features can be used in various related technical scenarios, such as copyright detection, video duplication detection, etc.

[0003] In order to realize the extraction of video fingerprint features, in the traditional way, when obtaining a video fingerprint feature extractor for extracting video fingerprint features of video content, a large amount of multi-category manually annotated data is usually required for training, which is time-consuming and costly. Summary of the invention

[0004] Based on this, it is necessary to provide a video processing method, a video processing device, a computer equipment, a computer-readable storage medium and a computer program product with low cost and strong characterization capability to address the above technical problems.

[0005] A video processing method, the method comprising:

[0006] Get training video clips;

[0007] Extracting a first sample video segment, a second sample video segment, and a third sample video segment from the training video segment, wherein the first sample video segment and the second sample video segment are positive samples of each other, and the first sample video segment and the third sample video segment are negative samples of each other;

[0008] Extracting a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model;

[0009] The video feature extraction model is optimized with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library until the training end condition is reached; the video features in the memory library include video features extracted during the historical iterative training process;

[0010] If the training end condition is met, a video fingerprint feature extractor is obtained based on the optimized video feature extraction model.

[0011] In some embodiments, extracting the first video feature of the first sample video segment through the first network includes:

[0012] The features of the first sample video segment are extracted through the first network to obtain the first sample segment features of the first sample video segment; and feature mapping is performed on the first sample segment features to obtain the first video features of the first sample video segment.

[0013] In some embodiments, based on the first network parameter in the first network and the second network parameter in the second network, momentum updating the second network parameter of the second network includes:

[0014] A first network parameter of the first network and a second network parameter of the second network are weightedly summed in combination with a momentum factor to perform momentum update on the network parameters of the second network.

[0015] In some embodiments, weighted summing of the first network parameter of the first network and the second network parameter of the second network in combination with a momentum factor to perform momentum update on the second network parameter of the second network includes:

[0016] The momentum factor is used as the weighted coefficient of the second network parameter, and the difference between 1 and the momentum factor is used as the weighted coefficient of the first network parameter. The second network parameter and the first network parameter are weightedly summed, and the network parameter after the weighted sum is used as the second network parameter after momentum update.

[0017] In some embodiments, extracting the first sample segment feature of the first sample video segment by the feature extraction subnetwork includes:

[0018] The first sample video segment is sequentially subjected to a first feature graph convolution process, a first maximum pooling layer process, a convolution process, a second feature graph convolution process, a first maximum pooling layer process, a first predetermined number of multi-dimensional feature connection processes, a second maximum pooling layer process, a second predetermined number of multi-dimensional feature connection processes, a third maximum pooling layer process, a third predetermined number of multi-dimensional feature connection processes, and an average pooling layer process to obtain a first sample segment feature of the first sample video segment; the convolution kernels involved in the first feature graph convolution process, the convolution process, and the second feature graph convolution process are three-dimensional convolution kernels, and the sampling windows in the first maximum pooling layer process, the second maximum pooling layer process, the third maximum pooling layer process, and the average pooling layer process are three-dimensional sampling windows;

[0019] The multi-dimensional feature connection processing includes: processing the input features in at least two different processing methods to obtain sub-features corresponding to each processing method, performing feature connection on each of the sub-features, and obtaining features of the multi-dimensional feature connection processing.

[0020] In some embodiments, the input feature is processed in at least two different processing modes to obtain sub-features corresponding to each processing mode, and each of the sub-features is feature-connected to obtain a feature processed by multi-dimensional feature connection, including:

[0021] After performing maximum pooling processing on the input features, performing convolution processing on the features after the maximum pooling processing to obtain a first sub-feature;

[0022] Performing convolution processing on the input feature through a first convolution layer, a second convolution layer, and a third convolution layer to obtain a second sub-feature;

[0023] Performing convolution processing on the input feature through the first convolution layer, the second convolution layer, and the third convolution layer to obtain a third sub-feature;

[0024] Performing convolution processing on the input feature to obtain a fourth sub-feature;

[0025] Perform feature connection on the first sub-feature, the second sub-feature, the third sub-feature and the fourth sub-feature to obtain a feature obtained by multi-dimensional feature connection processing.

[0026] In some embodiments, determining a method for obtaining the training video segment includes:

[0027] Get the target video source;

[0028] Cutting the target video source into video clips of a predetermined length;

[0029] The edge change rate detection is performed on the video segment, and the video segment whose edge change rate detection result meets the predetermined requirement is determined as the training video segment.

[0030] A video processing method, the method comprising:

[0031] Get the target video clip;

[0032] A preset video fingerprint feature extractor is used to extract the video fingerprint feature of the target video segment to obtain the video fingerprint feature of the target video segment. The video fingerprint feature extractor is obtained in the following manner:

[0033] Get training video clips;

[0034] Extracting a first sample video segment, a second sample video segment, and a third sample video segment from the training video segment, wherein the first sample video segment and the second sample video segment are positive samples of each other, and the first sample video segment and the third sample video segment are negative samples of each other;

[0035] Extracting a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model;

[0036] The video feature extraction model is optimized with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library until the training end condition is reached; the video features in the memory library include video features extracted during the historical iterative training process;

[0037] If the training end condition is met, a video fingerprint feature extractor is obtained based on the optimized video feature extraction model.

[0038] A video processing device, comprising:

[0039] A training video segment acquisition module is used to acquire training video segments;

[0040] a data augmentation module, configured to extract a first sample video segment, a second sample video segment, and a third sample video segment from the training video segment, wherein the first sample video segment and the second sample video segment are positive samples of each other, and the first sample video segment and the third sample video segment are negative samples of each other;

[0041] a feature extraction module, configured to extract a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model;

[0042] A training iterative output module is used to optimize the video feature extraction model with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library until the training end condition is reached; the video features in the memory library include video features extracted during historical iterative training; if the training end condition is reached, a video fingerprint feature extractor is obtained based on the optimized video feature extraction model.

[0043] A video processing device, comprising:

[0044] A target video segment acquisition module is used to acquire a target video segment;

[0045] The video fingerprint feature extraction module is used to use a preset video fingerprint feature extractor to extract the video fingerprint feature of the target video segment to obtain the video fingerprint feature of the target video segment, wherein the video fingerprint feature extractor is obtained in the following manner: obtaining a training video segment; extracting a first sample video segment, a second sample video segment and a third sample video segment from the training video segment, wherein the first sample video segment and the second sample video segment are positive samples of each other, and the first sample video segment and the third sample video segment are negative samples of each other; extracting a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model; optimizing the video feature extraction model with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library until a training end condition is reached; the video features in the memory library include video features extracted during historical iterative training; if the training end condition is reached, obtaining a video fingerprint feature extractor based on the optimized video feature extraction model.

[0046] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0047] Get training video clips;

[0048] Extracting a first sample video segment, a second sample video segment, and a third sample video segment from the training video segment, wherein the first sample video segment and the second sample video segment are positive samples of each other, and the first sample video segment and the third sample video segment are negative samples of each other;

[0049] Extracting a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model;

[0050] The video feature extraction model is optimized with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library until the training end condition is reached; the video features in the memory library include video features extracted during the historical iterative training process;

[0051] If the training end condition is met, a video fingerprint feature extractor is obtained based on the optimized video feature extraction model.

[0052] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0053] Get the target video clip;

[0054] A preset video fingerprint feature extractor is used to extract the video fingerprint feature of the target video segment to obtain the video fingerprint feature of the target video segment. The video fingerprint feature extractor is obtained in the following manner:

[0055] Get training video clips;

[0056] Extracting a first sample video segment, a second sample video segment, and a third sample video segment from the training video segment, wherein the first sample video segment and the second sample video segment are positive samples of each other, and the first sample video segment and the third sample video segment are negative samples of each other;

[0057] Extracting a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model;

[0058] The video feature extraction model is optimized with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library until the training end condition is reached; the video features in the memory library include video features extracted during the historical iterative training process;

[0059] If the training end condition is met, a video fingerprint feature extractor is obtained based on the optimized video feature extraction model.

[0060] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0061] Get training video clips;

[0062] Extracting a first sample video segment, a second sample video segment, and a third sample video segment from the training video segment, wherein the first sample video segment and the second sample video segment are positive samples of each other, and the first sample video segment and the third sample video segment are negative samples of each other;

[0063] Extracting a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model;

[0064] The video feature extraction model is optimized with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library until the training end condition is reached; the video features in the memory library include video features extracted during the historical iterative training process;

[0065] If the training end condition is met, a video fingerprint feature extractor is obtained based on the optimized video feature extraction model.

[0066] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0067] Get the target video clip;

[0068] A preset video fingerprint feature extractor is used to extract the video fingerprint feature of the target video segment to obtain the video fingerprint feature of the target video segment. The video fingerprint feature extractor is obtained in the following manner:

[0069] Get training video clips;

[0070] Extracting a first sample video segment, a second sample video segment, and a third sample video segment from the training video segment, wherein the first sample video segment and the second sample video segment are positive samples of each other, and the first sample video segment and the third sample video segment are negative samples of each other;

[0071] Extracting a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model;

[0072] The video feature extraction model is optimized with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library until the training end condition is reached; the video features in the memory library include video features extracted during the historical iterative training process;

[0073] If the training end condition is met, a video fingerprint feature extractor is obtained based on the optimized video feature extraction model.

[0074] A computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0075] Get training video clips;

[0076] Extracting a first sample video segment, a second sample video segment, and a third sample video segment from the training video segment, wherein the first sample video segment and the second sample video segment are positive samples of each other, and the first sample video segment and the third sample video segment are negative samples of each other;

[0077] Extracting a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model;

[0078] The video feature extraction model is optimized with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library until the training end condition is reached; the video features in the memory library include video features extracted during the historical iterative training process;

[0079] If the training end condition is met, a video fingerprint feature extractor is obtained based on the optimized video feature extraction model.

[0080] A computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0081] Get the target video clip;

[0082] A preset video fingerprint feature extractor is used to extract the video fingerprint feature of the target video segment to obtain the video fingerprint feature of the target video segment. The video fingerprint feature extractor is obtained in the following manner:

[0083] Get training video clips;

[0084] Extracting a first sample video segment, a second sample video segment, and a third sample video segment from the training video segment, wherein the first sample video segment and the second sample video segment are positive samples of each other, and the first sample video segment and the third sample video segment are negative samples of each other;

[0085] Extracting a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model;

[0086] The video feature extraction model is optimized with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library until the training end condition is reached; the video features in the memory library include video features extracted during the historical iterative training process;

[0087] If the training end condition is met, a video fingerprint feature extractor is obtained based on the optimized video feature extraction model.

[0088] The above-mentioned video processing method, video processing device, computer equipment, storage medium and computer program product obtain three sample video segments (a first sample video segment, a second sample video segment and a third sample video segment) from the same training video clip to form a pair of positive samples and a pair of negative samples. It is no longer necessary to perform a manual labeling process, and the video feature extraction model is optimized with the goal that the first video feature of the first sample video segment is similar to the second video feature of the second sample video segment, the first video feature is far away from the third video feature of the third sample video segment, and the first video feature is far away from the video features in the memory library. While ensuring that the three video features come from the same training video clip, the first video feature is far away from the third video feature, and the first video feature is far away from the video features in the memory library. On this basis, a video fingerprint feature extractor with strong video fingerprint feature representation capability can be obtained without manual labeling, and the process cost of obtaining the video fingerprint feature extractor is low and the video fingerprint feature extracted by the obtained video fingerprint feature extractor has strong representation capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 A diagram showing an application environment of a video processing method in an embodiment;

[0090] Figure 2 is a schematic flow chart of a video processing method in one embodiment;

[0091] Figure 3 A schematic diagram of a process for determining and obtaining a training video segment in one embodiment;

[0092] Figure 4 A schematic diagram of a process for obtaining each sample video segment in an embodiment;

[0093] Figure 5 A schematic diagram of a process of extracting video features in one embodiment;

[0094] Figure 6 A schematic diagram of a process of extracting a first video feature of a first sample video segment in one embodiment;

[0095] Figure 7 is a flow chart of a video processing method in another embodiment;

[0096] Figure 8 A schematic diagram of a model architecture for training a video fingerprint feature extractor in one embodiment;

[0097] Fig. 9is a structural block diagram of a video processing device in one embodiment;

[0098] Fig.10 is a structural block diagram of a video processing device in another embodiment;

[0099] Fig.11 is an internal structure diagram of a computer device in one embodiment;

[0100] Fig.12 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0101] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0102] The video processing method provided in this application can be applied to Figure 1 In the application environment shown. Figure 1 As shown, the application environment may only involve the first device 101, which obtains a target video source, determines a training video segment based on the target video source, obtains a video fingerprint feature extractor based on the training video segment, and uses the video fingerprint feature extractor to extract video fingerprint features.

[0103] In some embodiments, the application environment may involve both the first device 101 and the second device 102, wherein the first device 101 and the second device 102 communicate via a network, the second device 102 obtains a target video source, and determines a training video segment based on the target video source, and the obtained training video segment can be provided to the first device 101, so that the first device 101 obtains a video fingerprint feature extractor based on the training video segment training, and uses the video fingerprint feature extractor to extract video fingerprint features. In this embodiment, a data storage system 104 may also be involved, and the training video segment obtained by the second device 102 may be stored in the data storage system 104, the first device 101 may obtain each training video segment from the data storage system 104, and obtain a video fingerprint feature extractor based on the training video segment training, and use the video fingerprint feature extractor to extract video fingerprint features.

[0104] In some embodiments, the application environment may involve the first device 101, the second device 102, and the third device 103 at the same time, wherein the first device 101 and the second device 102 communicate through a network, the first device 101 and the third device 103 communicate through a network, the second device 102 obtains a target video source, and determines a training video segment based on the target video source, and the obtained training video segment can be provided to the first device 101, so that the first device 101 obtains a video fingerprint feature extractor based on the training video segment training, and the video fingerprint feature extractor obtained by the first device 101 training can be provided to the third device 103, and the third device 103 uses the video fingerprint feature extractor to extract video fingerprint features. In this embodiment, a data storage system 104 may also be involved, and the training video segments obtained by the second device 102 can be stored in the data storage system 104, and the first device 101 can obtain each training video segment from the data storage system 104, and obtain a video fingerprint feature extractor based on the training video segment training. The data storage system 104 may also be used to store the video fingerprint feature extractor trained by the first device 101. The third device 103 may obtain the video fingerprint feature extractor from the data storage system 104 and use the video fingerprint feature extractor to extract video fingerprint features.

[0105] Among them, the first device 101, the second device 102, and the third device 103 can be terminals or servers. The terminals can be but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The embodiments involved in this application can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. For example, the data storage system 104 can be a cloud storage system to store the training video clips obtained by the second device 102 and the video fingerprint feature extractor obtained by the training of the first device 101. For another example, videos obtained in various scenarios such as artificial intelligence, smart transportation, and assisted driving can be processed based on the video processing method or device of the embodiments of this application.

[0106] In one embodiment, Figure 2 As shown, a video processing method is provided, which is applied to Figure 1 Taking the first device in the example as an example, the method includes the following steps S201 to S205.

[0107] Step S201: Obtain training video clips.

[0108] The training video segment is a video segment used for training to obtain the video fingerprint feature extractor, and the length of the training video segment can be a video segment of a predetermined length.

[0109] In some embodiments, as described above, a third-party device (e.g. Figure 1 The second device shown in FIG. 1 directly obtains the training video segment that has been segmented and processed by the third-party device. Alternatively, the first device itself processes the target video source to obtain the training video segment.

[0110] Taking the first device itself processing the target video source to obtain the training video segment as an example, in one embodiment, determining a method for obtaining the training video segment includes the following steps S2011 to S2013:

[0111] Step S2011: Acquire the target video source.

[0112] Wherein, the target video source refers to the video source for obtaining the training video clip. The video content obtained from any database storing the video source can be used as the target video source. In some embodiments, the target video source can be determined in combination with the actual technical scene needs. The obtained video fingerprint feature extractor is used to extract the video fingerprint features of the game video content. Taking the processing of each game video content such as checking for duplicates as an example, each game content video can be obtained, and each game content video can be used as the target video source. The obtained video fingerprint feature extractor is used to extract the video fingerprint features of the sports video content. Taking the processing of each sports video content such as checking for duplicates as an example, each sports video content can be obtained, and each sports video content can be used as the target video source. It should be understood that when processing different types of video content, the corresponding type of video content can be obtained as the target video source. In other embodiments, it can also be to distinguish the types of video content, and directly use the obtained related video source as the target video source.

[0113] Step S2012: cutting the target video source into video clips of a predetermined length.

[0114] When cutting the target video source, the target video source may be cut according to a predetermined length, that is, the target video source may be cut at intervals of a predetermined length, thereby obtaining a video segment of a predetermined length. It should be understood that after cutting, a certain number of video segments of a predetermined length may be obtained, and the number is related to the ratio of the length of the target video source to the predetermined length. In some embodiments, the number may be a rounded-down value or a rounded-up value of the ratio of the length of the target video source to the predetermined length.

[0115] The specific length of the predetermined length can be set in combination with actual technical needs. In some embodiments of the present application, the value of the predetermined length can be set to a value between 3 and 15 seconds. In some specific examples, the predetermined length can be set to 12 seconds.

[0116] Step S2013: performing edge change rate detection on the video segments, and determining the video segments whose edge change rate detection results meet predetermined requirements as the training video segments.

[0117] When edge change rate detection is performed on a video segment, the edge change rate detection can be performed on the video segment by using an ECR edge change rate detection method. By using the ECR edge change rate detection method, the edge change rate of the video segment can be detected, and the video segment with an edge change rate greater than a predetermined edge change rate threshold is regarded as a video segment that meets the predetermined requirements, that is, it is regarded as a changed video segment, and the video segment is used as a training video segment. By taking the video segment with an edge change rate greater than the predetermined edge change rate threshold as a video segment that meets the predetermined requirements, the unchanged video segment can be eliminated accordingly, thereby avoiding the situation where the model crashes due to the unchanged video segment being trained.

[0118] It should be understood that there are a certain number of training video clips obtained based on the above method. In the process of obtaining the video fingerprint feature extractor, one training video clip can be obtained each time and enter the subsequent processing process.

[0119] Step S202: extracting a first sample video segment, a second sample video segment and a third sample video segment from the training video clip, wherein the first sample video segment and the second sample video segment are positive samples of each other, and the first sample video segment and the third sample video segment are negative samples of each other.

[0120] Among them, the first sample video segment and the second sample video segment are positive samples of each other, which means that the first sample video segment and the second sample video segment are similar samples, and the features of the two are similar; the first sample video segment and the third sample video segment are negative samples of each other, which means that the first sample video segment and the third sample video segment are dissimilar samples, and the features of the two are greatly different.

[0121] Among them, by extracting the first sample video segment, the second sample video segment and the third sample video segment from the same training video clip, two different video sample segments (the first sample video segment and the second sample video segment) derived from the same training video clip have similarities, while two different video sample segments (the first sample video segment and the third sample video segment) derived from the same training video clip have dissimilarities, which helps the subsequent training processing process and makes the training effect more accurate.

[0122] In one embodiment, reference Figure 3 As shown, extracting the first sample video segment, the second sample video segment and the third sample video segment from the training video clip may specifically include the following steps S2021 to S2023.

[0123] Step S2021: randomly selecting a predetermined number of frame images at equal intervals from the training video segment to obtain the first sample video segment.

[0124] The predetermined number may be set in combination with actual technical needs. For example, in some embodiments, the predetermined number may be set to 16. The number of randomly equally spaced interval frames may be set in combination with actual technical needs, and may be specifically set in combination with requirements such as training accuracy. In some embodiments, multiple interval frame numbers may be set, and when acquiring frame images, an interval frame number is randomly selected, and a predetermined number of frame images are acquired at equal intervals of the interval frame number.

[0125] Step S2021: In the training video clip and within the time coverage of the first sample video segment, randomly and evenly select the predetermined number of frame images to obtain the second sample video segment, where the second sample video segment is different from the first sample video segment, and an overlapping ratio between the first sample video segment and the second sample video segment is greater than a preset ratio.

[0126] Among them, the predetermined ratio is less than 1, and the specific numerical value of the predetermined ratio can be set in combination with the accuracy of the specific duplication check. For example, if it is considered that the two video segments are the same when the overlap degree reaches 40%, then the predetermined ratio can be set to 40%. If it is considered that the two video segments are the same only when the overlap degree reaches 70%, then the predetermined ratio can be set to 70%. Specifically in the relevant embodiments of the present application, the predetermined ratio can be set to 50%.

[0127] Step S2023: In the training video clip, outside the time coverage range of the first sample video segment, randomly and evenly select the predetermined number of frame images to obtain the third sample video segment, and the time interval between the third sample video segment and the time coverage range of the first sample video segment is greater than the preset time interval.

[0128] Among them, the time interval between the time coverage range of the third sample video segment and the first sample video segment refers to the time interval between the last frame of the third sample video segment (or the first frame of the third sample video segment) and the first frame of the first sample video segment (or the last frame of the first sample video segment). The specific size of the time interval can be set in combination with the type of video clip. If this type of video usually changes greatly in a short period of time, the time interval can be set relatively small. If this type of video usually does not change much in a short period of time, the time interval can be set relatively large. Considering that for videos, the content of the video screen usually changes after an interval of 2 seconds, especially for game videos, this change is more obvious. Therefore, in some embodiments of the present application, the time interval can be set to 2 seconds.

[0129] It should be understood that the process of obtaining the second sample video segment in step S2022 and the process of obtaining the third sample video segment in step S2023 may be performed simultaneously in any order.

[0130] As described above, the method of obtaining the first sample video segment, the second sample video segment and the third sample video segment in the embodiment of the present application is to obtain three sample video segments from the same training video segment, and the second sample video segment is within the coverage of the first sample video segment, and the third sample video segment is outside the coverage of the first sample video segment, thereby realizing a method of random time domain jittering of overlapping video segments, thereby realizing the input of three sample video segments, which can effectively avoid the problem of excessive features generated by CNN due to random equal time difference selection of 16 frames of pictures or similar operations in an entire video.

[0131] Step S203: extracting a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model.

[0132] In some embodiments, the video feature extraction model may include a first network and a second network, and the network structures of the first network and the second network are the same.

[0133] At this time, refer to Figure 5 As shown, the above-mentioned extraction of the first video feature of the first sample video segment, the second video feature of the second sample video segment, and the third video feature of the third sample video segment through the video feature extraction model may include the following steps S2031 and S2032.

[0134] Step S2031: extracting a first video feature of the first sample video segment and a third video feature of the third sample video segment through a first network.

[0135] Among them, the first network is a network for extracting video features (first video features, third video features) of the first sample video segment and the third sample video segment. In actual technical processing, there may be only one first network. In specific processing, through the control of timing, the video features of the first sample video segment and the third sample video segment are respectively extracted and output through the first network. In some embodiments, two completely identical first networks may also exist at the same time, and the first video features of the first sample video segment are extracted through one of the first networks, and the third video features of the third sample video segment are extracted through the other first network. In the model optimization process, when the video feature extraction model needs to be optimized, the two first networks can be optimized in the same way. For ease of explanation, in the following related embodiments of the present application, the example of the existence of two completely identical first networks is used for explanation.

[0136] Taking the extraction of the first video feature of the first sample video segment through the first network as an example, refer to Figure 6 As shown, extracting the first video feature of the first sample video segment through the first network may specifically include step S2131 and step S2231.

[0137] Step S2131: extracting features of the first sample video segment to obtain first sample segment features of the first sample video segment.

[0138] The process of extracting the features of the first sample video segment may be specifically to extract relevant features that can uniquely identify the first sample video segment. The first sample segment features extracted from the first sample video segment may be spatiotemporal fingerprint features of the first sample video segment.

[0139] In one embodiment, extracting the feature of the first sample video segment to obtain the first sample segment feature of the first sample video segment may include:

[0140] The first sample video segment is sequentially subjected to a first feature graph convolution process, a first maximum pooling layer process, a convolution process, a second feature graph convolution process, a first maximum pooling layer process, a first predetermined number of multi-dimensional feature connection processes, a second maximum pooling layer process, a second predetermined number of multi-dimensional feature connection processes, a third maximum pooling layer process, a third predetermined number of multi-dimensional feature connection processes, and an average pooling layer process to obtain features of the first sample video segment;

[0141] The multi-dimensional feature connection processing includes: processing the input features in at least two different processing methods to obtain sub-features corresponding to each processing method, performing feature connection on each of the sub-features, and obtaining features of the multi-dimensional feature connection processing.

[0142] Among them, the convolution kernels involved in the first feature map convolution processing, the convolution processing, and the second feature map convolution processing are three-dimensional convolution kernels, and the sampling windows in the first maximum pooling layer processing, the second maximum pooling layer processing, the third maximum pooling layer processing, and the average pooling layer processing are three-dimensional sampling windows.

[0143] Among them, the convolution kernel size and step size of the above-mentioned first feature map convolution processing and the second feature map convolution processing are different. Therefore, through the convolution processing of different convolution kernel sizes and different step sizes, the technical characteristics of different data block size dimensions and data block step sizes of the first sample video segment can be taken into account, thereby improving the diversity of the obtained technical features and making the obtained features more accurate. In a specific example, the convolution kernel of the first feature map convolution processing can be set to 7*7*7, and the step size can be set to 2, and the convolution kernel of the second feature map convolution processing can be set to 3*3*3, and the step size can be set to 1. It should be understood that in other embodiments, the convolution kernel size and step size of the first feature map convolution processing and the second feature map convolution processing can be set to other settings.

[0144] In some embodiments, the sampling windows of the first maximum pooling layer, the second maximum pooling layer, and the third maximum pooling layer have different sizes and the same step size. Thus, in the process of extracting features, by adopting pooling processing with different sampling windows, downsampling based on different sampling windows is performed, thereby effectively reducing the data dimension and avoiding overfitting. In some specific examples, the sampling window of the first maximum pooling layer can be set to 1*3*3, and the step size can be set to 2, the sampling window of the second maximum pooling layer can be set to 3*3*3, and the step size can be set to 2, and the sampling window of the third maximum pooling layer can be set to 2*2*2, and the step size can be set to 2.

[0145] In some embodiments, the first predetermined number of times, the second predetermined number of times, and the third predetermined number of times can be set differently. For example, in some embodiments, the first predetermined number of times can be set to 2, the second predetermined number of times can be set to 5, and the third predetermined number of times can be set to 2.

[0146] In the multi-dimensional feature connection processing, the input features are processed in at least two different ways to obtain sub-features corresponding to each processing method, and then the sub-features are connected to obtain features of the multi-dimensional feature connection processing. Therefore, the input features can be characterized in a variety of different ways, so that the features finally obtained after the feature connection can be more accurate, thereby making the video fingerprint features extracted by the video fingerprint feature extractor more accurate.

[0147] In some embodiments, during the multi-dimensional feature connection processing, the input feature is processed in at least two different processing modes to obtain sub-features corresponding to each processing mode, and each of the sub-features is feature connected to obtain features of the multi-dimensional feature connection processing, which may specifically include:

[0148] After performing maximum pooling processing on the input features, performing convolution processing on the features after the maximum pooling processing to obtain a first sub-feature; wherein the sampling window of the maximum pooling processing is a three-dimensional sampling window, and the convolution kernel of the convolution processing is a three-dimensional convolution kernel;

[0149] The input feature is convolved through a first convolution layer, a second convolution layer, and a third convolution layer to obtain a second sub-feature; the first convolution layer, the second convolution layer, and the third convolution layer are different from each other, and the convolution kernels of the first convolution layer, the second convolution layer, and the third convolution layer are all three-dimensional convolution kernels;

[0150] Performing convolution processing on the input feature through the first convolution layer, the second convolution layer, and the third convolution layer to obtain a third sub-feature;

[0151] Performing convolution processing on the input feature to obtain a fourth sub-feature; wherein the convolution kernel of the convolution processing is a three-dimensional convolution kernel;

[0152] Perform feature connection on the first sub-feature, the second sub-feature, the third sub-feature and the fourth sub-feature to obtain a feature obtained by multi-dimensional feature connection processing.

[0153] Among them, when the first sub-feature, the second sub-feature, the third sub-feature and the fourth sub-feature are connected, each sub-feature can be directly spliced ​​in a predetermined order, for example, the first sub-feature, the second sub-feature, the third sub-feature and the fourth sub-feature are connected in sequence to obtain a feature processed by multi-dimensional feature connection. In this process, the second sub-feature and the third sub-feature are obtained by processing the input feature twice by using the first convolution layer, the second convolution layer and the third convolution layer, respectively, and feature connection is performed on this basis, so that the features processed by the multi-dimensional feature connection finally obtained have redundant features, so that the obtained features are more accurate, so as to help the accuracy of the video fingerprint features extracted by the subsequent video fingerprint feature extractor.

[0154] Step S2231: performing feature mapping on the first sample segment feature to obtain a first video feature of the first sample video segment.

[0155] The process of performing feature mapping on the first sample segment feature can be regarded as the process of mapping the first sample segment feature to a high-dimensional feature. The specific feature mapping method can be performed in any possible feature mapping method, and the embodiment of the present application does not make any specific limitation.

[0156] The method of extracting the third video feature of the third sample video segment through the first network may be the same as the method of extracting the first video feature of the first sample video segment through the first network mentioned above, specifically, extracting the feature of the third sample video segment through the first network to obtain the third sample segment feature of the third sample video segment, and the extracted third sample segment feature of the third sample video segment may be the spatiotemporal fingerprint feature of the third sample video segment; and performing feature mapping on the third sample segment feature to obtain the third video feature of the third sample video segment, which will not be repeated here.

[0157] Since the video features obtained by extracting the first sample video segment and the third sample video segment using the first network are to be kept away from each other according to the training goal, by further performing feature mapping on the first sample segment features and the third sample segment features, they can be mapped to higher-dimensional features, thereby making the training accuracy higher and making the video fingerprint features extracted by the final video fingerprint feature extractor more accurate.

[0158] Step S2032: extracting a second video feature of the second sample video segment through a second network.

[0159] The second network is a network for extracting the video features of the second sample video segment. The second network has the same network structure as the first network. In one embodiment, extracting the second video features of the second sample video segment through the second network may include:

[0160] Extracting features of the second sample video segment to obtain second sample segment features of the second sample video segment; wherein the process of extracting features of the second sample video segment to obtain second sample segment features of the second sample video segment may be the same as the method of extracting features of the first sample video segment to obtain first sample segment features of the first sample video segment in step S2131 above, and the extracted second sample segment features of the second sample video segment may be spatiotemporal fingerprint features of the second sample video segment;

[0161] Feature mapping is performed on the second sample segment features to obtain second video features of the second sample video segment, wherein the process of performing feature mapping on the second sample segment features to obtain second video features of the second sample video segment may be the same as the method of performing feature mapping on the first sample segment features in the above step S2232 to obtain first video features of the first sample video segment.

[0162] Since the training goal is to make the second video features obtained by using the second network to process the second sample video segment similar to the first video features obtained by using the first network to process the first sample video segment, therefore, by further performing feature mapping on the second sample segment features, they can be mapped to higher-dimensional features, so that the training accuracy is higher and the video fingerprint features extracted by the final video fingerprint feature extractor are more accurate.

[0163] Step S204: With the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library, the video feature extraction model is optimized until the training end condition is reached; the video features in the memory library include video features extracted during historical iterative training.

[0164] Among them, the video features extracted during the historical iterative training process can be the video features extracted in each iterative process since the start of training, or can be the video features extracted in the most recent predetermined number of historical iterative training processes in a first-in-first-out manner, so as to avoid the problem of increased computational complexity caused by the continuous growth of video features in the memory library as the training process grows. Specifically, the video features in the memory library can be the video features of positive samples in the historical iterative training process. Taking the last iterative training process as an example, in the last iterative training process, the first video sample segment and the second video sample segment, which are positive samples of the last training video clip, and the third sample video segment, which are negative samples of the first sample video sample segment, will be obtained. After the last iterative training is completed, the video features of the second sample video segment of the last training video clip will be stored in the memory library.

[0165] With the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library, when optimizing the video feature extraction model, the following method may be used:

[0166] Calculate the loss value of the loss function with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library, wherein, in some specific examples, the loss value may also be the loss value of the loss function with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, the first video feature far away from the video features in the memory library, and the second video feature far away from the video features in the memory library. The specific form of the loss function may be designed in combination with the above-mentioned technical needs. For example, in some embodiments, the loss value may be calculated using a contrast loss function.

[0167] The video feature extraction model is optimized according to the loss value.

[0168] In some specific examples, when optimizing the video feature extraction model according to the loss value, the first network parameters of the first network in the video feature extraction model can be updated only according to the loss value. It should be understood that when there are two identical first networks that extract the first video feature and the third video feature respectively, the two first networks are updated simultaneously in the same manner. The specific process of updating the first network parameters of the first network can be performed using an existing method for updating the model.

[0169] As for the second network in the video feature extraction model, it can be performed through momentum update. The timing for updating the second network through momentum update can be after the first network parameters of the first network in the video feature extraction model are updated according to the loss value, and then the second network parameters of the second network are updated in combination with the updated first network parameters, and then the next iterative training process is entered. It can also be after the first network parameters of the first network in the video feature extraction model are updated according to the loss value, the next iterative training process is entered first, and before the second video features of the second sample video segment need to be extracted through the second network, the second network parameters of the second network are updated.

[0170] Taking updating the second network parameter of the second network before extracting the second video feature of the second sample video segment through the second network as an example, in some embodiments, before performing feature mapping on the second sample video segment through the second network to obtain the mapping feature, the method further includes:

[0171] Based on the first network parameters of the first network and the second network parameters of the second network, a momentum update is performed on the second network parameters of the second network.

[0172] Therefore, by performing momentum update on the network parameters of the second network, it is possible to effectively avoid the tendency that the first network and the second network are exactly the same, the features obtained by the first network and the second network are the same, and the loss is 0, thereby effectively avoiding the situation where the model collapses.

[0173] In some specific examples, based on the first network parameters of the first network and the second network parameters of the second network, the second network parameters of the second network are momentum updated. Specifically, the first network parameters in the first network and the second network parameters in the second network are weighted summed in combination with the momentum factor to perform momentum update on the network parameters of the second network. The momentum factor is a value less than 1, and the specific value of the momentum factor can be set in combination with actual technical needs.

[0174] In some embodiments, weighted summing of first network parameters in the first network and second network parameters in the second network in combination with a momentum factor to perform momentum update on the second network parameters of the second network includes:

[0175] The momentum factor is used as the weight coefficient of the second network parameter, the difference between 1 and the momentum factor is used as the weight coefficient of the first network parameter, the second network parameter and the first network parameter are weighted and summed, and the network parameter after the weighted sum is used as the second network parameter after the momentum update. It should be understood that the value of the momentum factor is less than 1.

[0176] Step S205: If the training end condition is met, a video fingerprint feature extractor is obtained based on the optimized video feature extraction model.

[0177] The training end condition can be set in combination with actual technical needs. In the embodiment of the present application, the training end condition can be set to the maximum number of iterations. If the number of training iterations reaches the maximum number of iterations, the training end condition is considered to be met. The specific value of the maximum number of iterations can be set in combination with actual technical needs, and can usually be comprehensively considered in combination with training efficiency and extraction accuracy of video fingerprint features.

[0178] In some embodiments, the first network may include a feature extraction subnetwork and a feature mapping subnetwork, wherein the feature extraction subnetwork extracts the first sample segment feature of the first sample video segment, and the feature mapping subnetwork performs feature mapping on the first sample segment feature to obtain the first video feature. When the iteration end condition is met, a video fingerprint feature extractor is obtained based on the video feature extraction model, specifically, the feature extraction subnetwork in the first network is extracted, and the feature extraction subnetwork is used as the video fingerprint feature extractor obtained by training.

[0179] In one embodiment, Figure 7 As shown, a video processing method is provided, which is applied to Figure 1 Taking the first device or the third device in the example as an example, the method includes the following steps S701 to S702.

[0180] Step S701: Obtain a target video clip.

[0181] The target video segment is a segment used to extract video fingerprint features. In some embodiments, the length of the target video segment may be the same as the length of the first sample video segment or the second sample video segment.

[0182] Among them, the target video segment obtained can be directly a segment containing the above-mentioned predetermined number of frame images, or it can be a video segment with a number of frame images greater than the predetermined number. The video segment with a predetermined number of frame images is extracted from the target video segment through random equal intervals or other extraction methods to enter the subsequent processing process.

[0183] Step S702: using a preset video fingerprint feature extractor to extract video fingerprint features from the target video clip to obtain video fingerprint features of the target video clip. The video fingerprint feature extractor is obtained by using the video processing method in any of the above-mentioned embodiments.

[0184] The present application also provides an application scenario, which applies the above-mentioned video processing method.

[0185] In the detection of video content, there are many scenarios where the similarity of video content needs to be judged, such as copyright detection, video duplication elimination, etc. The embodiment of the present application provides a video processing method to obtain a video fingerprint feature extractor for extracting the video fingerprint of a video clip. The obtained video fingerprint feature extractor can be used to extract the video fingerprint features of the video clip. By comparing the video fingerprints of two video clips, it can be determined whether the two video clips are the same or similar.

[0186] When training and obtaining a video fingerprint feature extractor, a target video source may be obtained from a related video content library. The video content library may be a video content library of a predetermined type, and the target video source may also be a certain number of video sources. For example, 100,000 (it should be understood that other numbers may be used in other embodiments) video sources may be selected from a video content library of the video game category as the target video source.

[0187] Subsequently, the obtained target video sources are cut into video clips of a predetermined length (e.g., 12 seconds). The edge change rate of each video clip is then detected to obtain video clips that meet predetermined requirements (e.g., video clips that are determined to be unchanged are eliminated, and video clips that are determined to be changed are retained as video clips that meet predetermined requirements), thereby finally obtaining training video clips that can be used in subsequent training processes.

[0188] When training video clips are used for training, one of the training video clips that has not been trained is selected each time to enter the subsequent training process. The specific architecture of the training process in a specific example can be as follows: Figure 8 shown.

[0189] For the selected training video segment, data augmentation is first performed on the training video segment, specifically, a predetermined number (for example, 16) of frame images are randomly and evenly selected in the training video segment as the first sample video segment v1, and within the time coverage of the first sample video segment v1, the predetermined number of frame images are randomly and evenly selected as the second sample video segment v′1, wherein the first sample video segment v1 is required to be different from the second sample video segment v′1, and the overlapping ratio of the second sample video segment v′1 to the first sample video segment v1 is greater than a predetermined ratio (for example, 50%) or the number of identical frame images reaches a certain set number (the set number can be a percentage of the above-mentioned predetermined number).

[0190] Furthermore, in the training video segment, a predetermined number of frame images are randomly selected at equal intervals at positions outside the time coverage range of the first sample video segment v1 and whose time interval with the time coverage range of the first sample video segment is greater than a preset time interval to obtain a third sample video segment v2. The preset time interval can be set in combination with actual technical needs, and can be set to 2 seconds in some embodiments.

[0191] Subsequently, the first network is used to process the first sample video segment v1 to obtain the first video feature corresponding to the first sample video segment v1 The third sample video segment v2 is processed by the first network to obtain a third video feature corresponding to the third sample video segment v2. The second network is used to process the second sample video segment v′1 to obtain the second video feature corresponding to the second sample video segment v′1

[0192] In a specific example, the feature extraction subnetwork 3D CNNf of the first network θ Extract the features of the first sample video segment v1 to obtain the first sample segment features Then, through the feature mapping subnetwork h θ For the first sample segment feature Perform feature mapping to obtain the first video feature

[0193] The feature extraction subnetwork of the first network is 3D CNNf θ Extract the features of the third sample video segment v2 to obtain the features of the third sample segment Then, through the feature mapping subnetwork h θ For the third sample segment feature Perform feature mapping to obtain the third video feature

[0194] The feature extraction subnetwork of the second network is 3D CNNf′ ξExtract the features of the second sample video segment v′1 to obtain the features of the second sample segment Then through the feature mapping sub-network h′ ξ For the second sample segment feature Perform feature mapping to obtain the second video feature

[0195] Among them, the feature extraction subnetwork 3D CNNf of the first network θ The feature extraction subnetwork 3DCNNf′ of the second network ξ The network structure can be the same, but the feature extraction subnetwork 3D CNNf θ With feature extraction subnetwork 3D CNNf′ ξ The model parameters of the first network can be different. θ The feature mapping subnetwork h′ of the second network ξ The network structure can be the same, but the feature mapping subnetwork h θ With the feature mapping subnetwork h′ ξ The network parameters of the second network may be different. In a specific example, the feature mapping subnetwork h′ of the second network ξ The network parameters of the first network can be based on the feature mapping sub-network h θ The network parameters are updated with momentum, and the feature mapping subnetwork h of the second network θ The network parameters of the first network can be based on the feature mapping subnetwork h′ ξ The network parameters are updated with momentum.

[0196] In a specific example, the feature extraction subnetwork (h θ and h′ ξ) may include a first feature map convolution processing layer, a first maximum pooling layer, a convolution layer, a second feature map convolution processing layer, a first maximum pooling layer, a first number (for example, 2) of multi-dimensional feature connection processing layers, a second maximum pooling layer, a second number (for example, 5) of the multi-dimensional feature connection processing layers, a third maximum pooling layer, a third number (for example, 2) of multi-dimensional feature connection processing layers, and an average pooling layer connected in sequence, and the convolution kernel involved in the first feature map convolution processing, the convolution processing, and the second feature map convolution processing is a three-dimensional convolution kernel, and the sampling windows in the first maximum pooling layer processing, the second maximum pooling layer processing, the third maximum pooling layer processing, and the average pooling layer processing are three-dimensional sampling windows. Thus, the first sample video segment v1 of the first sample video segment is obtained by sequentially performing the first feature map convolution processing, the first maximum pooling layer processing, the convolution processing, the second feature map convolution processing, the first maximum pooling layer processing, the first predetermined number of multi-dimensional feature connection processing, the second maximum pooling layer processing, the second predetermined number of multi-dimensional feature connection processing, the third maximum pooling layer processing, the third predetermined number of multi-dimensional feature connection processing, and the average pooling layer processing on the first sample video segment or the second sample video segment. The second sample segment features of the second sample video segment v′1 Or the third sample segment feature of the third sample video segment v2

[0197] In one embodiment, the convolution kernel of the first feature map convolution processing layer can be a three-dimensional 7*7*7 structure, and the step size can be 2. In a specific example, the first feature map convolution processing layer can include two convolution layers, and the convolution kernel of one convolution layer is 7*1*1, the step size is 1, and the convolution kernel of the other convolution layer is 1*7*7, the step size is 1. It can be understood that the first feature map convolution processing layer can also be configured in other ways, and the number and step size of the convolution layers included can also be configured in other ways, as long as the convolution kernels combined with each other are the three-dimensional convolution kernels and step sizes required by the first feature map convolution processing layer.

[0198] In one embodiment, the convolution kernel of the second feature map convolution processing layer can be a three-dimensional 3*3*3 structure, and the step size can be 2. In a specific example, the second feature map convolution processing layer can include two convolution layers, and the convolution kernel of one convolution layer is 3*1*1, the step size is 1, and the convolution kernel of the other convolution layer is 1*3*3, the step size is 1. It can be understood that the first feature map convolution processing layer can also be configured in other ways, and the number and step size of the convolution layers included can also be configured in other ways, as long as the convolution kernels combined with each other are the three-dimensional convolution kernels and step sizes required by the first feature map convolution processing layer.

[0199] Among them, the sampling windows of the first maximum pooling layer, the second maximum pooling layer, and the third maximum pooling layer have different sizes and the same step size. In a specific example, the sampling window of the first maximum pooling layer can be set to 1*3*3, the step size can be set to 2, the sampling window of the second maximum pooling layer can be set to 3*3*3, the step size can be set to 2, the sampling window of the third maximum pooling layer can be set to 2*2*2, the step size can be set to 2, and the convolution kernel of the convolution layer can be set to 1*1*1.

[0200] In one embodiment, the multi-dimensional feature connection processing layer may include four branch hierarchical structures arranged in parallel:

[0201] The first branch includes a maximum pooling layer and a convolution layer connected to each other. After the maximum pooling layer performs maximum pooling processing on the input feature, the convolution layer performs convolution processing on the feature after the maximum pooling processing to obtain the first sub-feature; wherein, in a specific example, the sampling window of the maximum pooling layer may be 3*3*3, and the convolution kernel of the convolution layer may be 1*1*1;

[0202] The second branch includes a first convolutional layer, a second convolutional layer, and a third convolutional layer, so as to obtain a second sub-feature by performing convolution processing on the input feature based on the first convolutional layer, the second convolutional layer, and the third convolutional layer, wherein the convolution kernel of the first convolutional layer, the second convolutional layer, and the third convolutional layer can be 3*3*3, and the step length is 2. In a specific example, the convolution kernels of the first convolutional layer, the second convolutional layer, and the third convolutional layer can be 1*1*1, 1*3*3, and 3*1*1, respectively, wherein the step length of the second convolutional layer is 1, and the step length of the third convolutional layer is 1;

[0203] The structure of the third branch may be the same as that of the second branch, so that the third sub-feature is obtained by performing convolution processing on the input feature through the first convolution layer, the second convolution layer, and the third convolution layer, so that the finally obtained feature may contain redundant feature information by adding one path of redundant feature information, so as to improve the accuracy of the obtained feature;

[0204] The fourth branch may only contain one convolutional layer, through which the input feature is convolved to obtain the fourth sub-feature. Fig.11 In the example shown, the convolution kernel of the convolution layer can be set to 1*1*1.

[0205] After obtaining the first sub-feature, the second sub-feature, the third sub-feature and the fourth sub-feature, feature connection can be performed on the first sub-feature, the second sub-feature, the third sub-feature and the fourth sub-feature through a feature connection layer to obtain features of multi-dimensional feature connection processing.

[0206] In the above, the first video feature of the first sample video segment v1 is obtained The second video feature of the second sample video segment v′1 And the third video feature of the third sample video segment v2 After that, the loss value can be calculated. In one embodiment of the present application, the loss value can be calculated using a contrast loss function, which can be expressed as:

[0207]

[0208] in, represents the loss value, Represents video features in the memory library.

[0209] Based on the calculated loss value Back propagation can be performed to update the feature extraction subnetwork 3D CNNf of the first network θ And the feature mapping subnetwork h θ , completing one iteration.

[0210] If the preset maximum number of iterations T is not reached, return to obtain the next training video segment that has not been trained, and repeat the above process.

[0211] In each iterative training process, the momentum of the second network is also updated simultaneously.

[0212] When the momentum of the second network is updated, the momentum of the network parameters of the second network can be updated by weighted summing the first network parameters in the first network and the second network parameters in the second network in combination with the momentum factor. In a specific example, the momentum factor can be used as the weighting coefficient of the second network parameter, and the difference between 1 and the momentum factor can be used as the weighting coefficient of the first network parameter. The second network parameter and the first network parameter are weighted summed, and the network parameter after the weighted sum is used as the second network parameter after the momentum update.

[0213] When the momentum of the second network parameters of the second network is updated, only the feature extraction subnetwork 3D CNNf′ of the second network may be updated. ξ The network parameters are updated, which can be expressed as:

[0214] ξ=τξ+(1-τ)θ

[0215] Where ξ is the feature extraction subnetwork 3D CNNf′ of the second network ξ The network parameters of , τ is the momentum factor, and τ is less than 1, θ is the feature extraction subnetwork 3D CNNf of the first network θThe network parameters. Among them, τ can be set to a value close to 1, such as a value with a difference from 1 within a certain range, so that each time the feature extraction subnetwork 3DCNNf′ of the second network ξ The adjustment range of τ is not too large. For example, in some specific examples, the value of τ can be set to 0.996.

[0216] It should be understood that the feature mapping subnetwork 3D CNNh′ of the second network can also be simultaneously ξ The network parameters are updated, which can also be expressed as:

[0217] ξ=τξ+(1-τ)θ

[0218] Where ξ is the feature mapping subnetwork 3D CNNh′ of the second network ξ The network parameters of , τ is the momentum factor, and τ is less than 1, θ is the feature mapping subnetwork 3D CNNh of the first network θ The network parameters. Among them, τ can be set to a value close to 1, such as a value with a difference from 1 within a certain range, so that each time the feature mapping subnetwork 3DCNNh′ of the second network ξ The adjustment range of τ is not too large. For example, in some specific examples, the value of τ can be set to 0.996.

[0219] If the preset maximum number of iterations is reached, the iteration process is completed and the feature extraction subnetwork 3D CNNf of the first network is replaced by θ Extracted as the obtained video fingerprint feature extractor.

[0220] The video fingerprint feature extractor obtained above can be used by itself, or distributed to other servers or terminals, or stored in a relevant location (such as the cloud) and then accessed by other servers or terminals to extract the video fingerprint features of the target video clip, and perform copyright detection or video duplication detection based on the extracted video fingerprint features.

[0221] For example, if it is necessary to detect whether the first target video segment M1 and the second target video segment M2 are the same or similar video segments, the first video fingerprint feature of the first target video segment M1 can be extracted by the video fingerprint feature extractor obtained above, and the second video fingerprint feature of the second target video segment M2 can be extracted by the video fingerprint feature extractor, and by comparing the similarities and differences between the first video fingerprint feature and the second video fingerprint feature, it is determined whether the first target video segment M1 and the second target video segment M2 are the same or similar video segments. For example, if the first video fingerprint feature is the same as the second video fingerprint feature, it can be determined that the first target video segment M1 and the second target video segment M2 are the same or similar video segments, otherwise they are different or dissimilar video segments. It should be understood that the first video fingerprint feature is the same as the second video fingerprint feature, which can be that the first video fingerprint feature and the second video fingerprint feature are exactly the same, or the error between the first video fingerprint feature and the second video fingerprint feature is within an acceptable error range.

[0222] For another example, if there is a third target video segment M3, and it is necessary to determine whether the third target video segment M3 is an existing video segment or is similar to an existing video segment, then it is possible to find out whether there is a video segment that is repeated with the third target video segment M3. Specifically, the third video fingerprint feature of the third target video segment M3 can be extracted by the video fingerprint feature extractor obtained above, and the third video fingerprint feature is compared with the video fingerprint features of each video segment in the database (which can be obtained by extracting each video segment in real time by the video fingerprint feature extractor obtained above, or can be a video fingerprint feature that is extracted and stored by the video fingerprint feature extractor obtained above in advance), to find out whether there is a video fingerprint feature that is the same as the third video fingerprint feature. If there is an identical video fingerprint feature, then it is determined that the video duplication check result is that there is a video segment that is the same as the third target video segment M3. Otherwise, it is determined that the video duplication check result is that there is no video segment that is the same as the third target video segment M3. It should be understood that the video fingerprint feature that is the same as the third video fingerprint feature can be completely the same as the third video fingerprint feature, or it can be a video fingerprint feature whose error with the third video fingerprint feature is within an acceptable error range.

[0223] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. For example, the processing of the above-mentioned steps S2022 and step S2023 can be performed simultaneously, and the processing of the above-mentioned steps S2031 and step S2032 can be performed simultaneously. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0224] In one embodiment, Fig. 9 As shown, a video processing device is provided, which can be a part of a computer device using a software module or a hardware module, or a combination of the two. The device specifically includes: a training video segment acquisition module 901, a data augmentation module 902, a feature extraction module 903 and a training iteration output module 904, wherein:

[0225] The training video segment acquisition module 901 is used to acquire the training video segment;

[0226] A data augmentation module 902 is used to extract a first sample video segment, a second sample video segment and a third sample video segment from the training video segment, wherein the first sample video segment and the second sample video segment are positive samples of each other, and the first sample video segment and the third sample video segment are negative samples of each other;

[0227] A feature extraction module 903, configured to extract a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model;

[0228] The training iteration output module 904 is used to optimize the video feature extraction model with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library until the training end condition is reached; the video features in the memory library include the video features extracted during the historical iterative training process; if the training end condition is reached, a video fingerprint feature extractor is obtained based on the optimized video feature extraction model.

[0229] In one embodiment, the data augmentation module 902 is used to randomly and evenly select a predetermined number of frame images in the training video segment to obtain the first sample video segment; randomly and evenly select the predetermined number of frame images in the training video segment and within the time coverage of the first sample video segment to obtain the second sample video segment, wherein the second sample video segment is different from the first sample video segment, and the overlap ratio between the first sample video segment and the second sample video segment is greater than a preset ratio; and randomly and evenly select the predetermined number of frame images in the training video segment and outside the time coverage of the first sample video segment to obtain the third sample video segment, wherein the time interval between the third sample video segment and the time coverage of the first sample video segment is greater than the preset time interval.

[0230] In one embodiment, the feature extraction module 803 is used to extract the first video feature of the first sample video segment and the third video feature of the third sample video segment through a first network; extract the second video feature of the second sample video segment through a second network, the second network has the same network structure as the first network, and the video feature extraction model includes the first network and the second network.

[0231] In one embodiment, the feature extraction module 803 extracts features of the first sample video segment through the first network to obtain first sample segment features of the first sample video segment; and performs feature mapping on the first sample segment features to obtain first video features of the first sample video segment.

[0232] In one embodiment, the training iteration output module 904 is used to update the first network parameters of the first network in the video feature extraction model with the goal of making the first video feature close to the second video feature, the first video feature away from the third video feature, and the first video feature away from the video features in the memory library.

[0233] In one embodiment, the apparatus further comprises: a momentum update module (not shown in the figure), wherein:

[0234] The momentum update module performs momentum update on the second network parameters of the second network based on the first network parameters of the first network and the second network parameters of the second network.

[0235] The feature extraction module 903 is used to extract the second video feature of the second sample video segment by using the second network updated by the momentum update module.

[0236] In some embodiments, the momentum update module is used to perform weighted summation of first network parameters of the first network and second network parameters of the second network in combination with a momentum factor to perform momentum update on the network parameters of the second network.

[0237] In some embodiments, the momentum update module is used to use the momentum factor as the weighted coefficient of the second network parameter, and the difference between 1 and the momentum factor as the weighted coefficient of the first network parameter, perform weighted summation on the second network parameter and the first network parameter, and use the network parameter after the weighted summation as the second network parameter after momentum update.

[0238] In one embodiment, the first network includes a feature extraction subnetwork and a feature mapping subnetwork; the first sample segment feature of the first sample video segment is extracted by the feature extraction subnetwork, and the first sample segment feature is feature mapped by the feature mapping subnetwork to obtain the first video feature;

[0239] The training iteration output module 904 is used to use the feature extraction sub-network in the first network as the trained video fingerprint feature extractor when the iteration end condition is reached.

[0240] In one embodiment, the device further comprises:

[0241] The training video segment determination module (not shown in the figure) is used to obtain a target video source; cut the target video source into video segments of a predetermined length; perform edge change rate detection on the video segments, and determine the video segments whose edge change rate detection results meet the predetermined requirements as the training video segments.

[0242] In one embodiment, Fig.10 As shown, a video processing device is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: a target video segment acquisition module 1001, a video fingerprint feature extraction module 1002, wherein:

[0243] The target video segment acquisition module 1001 is used to acquire the target video segment;

[0244] The video fingerprint feature extraction module 1002 is used to use a preset video fingerprint feature extractor to extract the video fingerprint feature of the target video segment to obtain the video fingerprint feature of the target video segment, and the acquisition method of the video fingerprint feature extractor includes: obtaining a training video segment; extracting a first sample video segment, a second sample video segment and a third sample video segment from the training video segment, wherein the first sample video segment and the second sample video segment are positive samples of each other, and the first sample video segment and the third sample video segment are negative samples of each other; extracting a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model; optimizing the video feature extraction model with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library until the training end condition is met; the video features in the memory library include video features extracted during the historical iterative training process; if the training end condition is met, obtaining a video fingerprint feature extractor based on the optimized video feature extraction model.

[0245] In one embodiment, the video processing device further includes the Fig. 9 The video processing device in any corresponding embodiment.

[0246] For the specific definition of the video processing device, please refer to the definition of the video processing method above, which will not be repeated here. Each module in the above video processing device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0247] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.11As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store one or more of training video clips, video features extracted during historical iterative training, and video fingerprint extractors obtained through training. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video processing method is implemented.

[0248] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.12 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a video processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0249] Those skilled in the art will understand that Fig.11 , 12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0250] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0251] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0252] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0253] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0254] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0255] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A video processing method, characterized in that: The method comprises: Get training video clips; Randomly selecting a predetermined number of frame images at equal intervals from the training video segment to obtain a first sample video segment; In the training video segment and within the time coverage of the first sample video segment, randomly and evenly select the predetermined number of frame images to obtain a second sample video segment, where the second sample video segment is different from the first sample video segment, and an overlapping ratio between the first sample video segment and the second sample video segment is greater than a preset ratio; In the training video segment, outside the time coverage range of the first sample video segment, randomly and evenly selecting the predetermined number of frame images to obtain a third sample video segment, wherein the time interval between the third sample video segment and the time coverage range of the first sample video segment is greater than a preset time interval; Extracting a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model; The video feature extraction model is optimized with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library until the training end condition is reached; the video features in the memory library include video features extracted during the historical iterative training process; If the training end condition is met, a video fingerprint feature extractor is obtained based on the optimized video feature extraction model.

2. The method according to claim 1, characterized in that Extracting a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model includes: Extracting a first video feature of the first sample video segment and a third video feature of the third sample video segment through a first network; A second video feature of the second sample video segment is extracted through a second network, the second network has the same network structure as the first network, and the video feature extraction model includes the first network and the second network.

3. The method according to claim 2, characterized in that The video feature extraction model is optimized with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from video features in the memory library, including: With the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library, the first network parameters of the first network in the video feature extraction model are updated.

4. The method according to claim 3, characterized in that Before extracting the second video feature of the second sample video segment through the second network, the method further includes: Based on the first network parameters of the first network and the second network parameters of the second network, momentum update is performed on the second network parameters of the second network.

5. The method according to claim 2, characterized in that: The first network includes a feature extraction subnetwork and a feature mapping subnetwork; the first sample segment feature of the first sample video segment is extracted by the feature extraction subnetwork, and the first sample segment feature is feature mapped by the feature mapping subnetwork to obtain the first video feature; If the iteration end condition is met, obtaining a video fingerprint feature extractor based on the video feature extraction model includes: If the iteration end condition is met, the feature extraction subnetwork in the first network is extracted as the trained video fingerprint feature extractor.

6. A video processing method, characterized in that: The method comprises: Get the target video clip; A preset video fingerprint feature extractor is used to extract video fingerprint features from the target video clip to obtain video fingerprint features of the target video clip, and the video fingerprint feature extractor is obtained by the method described in any one of claims 1-5.

7. A video processing device, characterized in that: The device comprises: A training video segment acquisition module is used to acquire training video segments; A data augmentation module is used to randomly and evenly select a predetermined number of frame images in the training video segment to obtain a first sample video segment; randomly and evenly select the predetermined number of frame images in the training video segment and within the time coverage of the first sample video segment to obtain a second sample video segment, wherein the second sample video segment is different from the first sample video segment, and the overlap ratio between the first sample video segment and the second sample video segment is greater than a preset ratio; randomly and evenly select the predetermined number of frame images in the training video segment and outside the time coverage of the first sample video segment to obtain a third sample video segment, and the time interval between the third sample video segment and the time coverage of the first sample video segment is greater than the preset time interval; a feature extraction module, configured to extract a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model; A training iteration output module is used to optimize the video feature extraction model with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library until the training end condition is reached; the video features in the memory library include video features extracted during historical iterative training; if the iteration end condition is reached, a video fingerprint feature extractor is obtained based on the video feature extraction model.

8. The device according to claim 7, characterized in that Extracting a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment through a video feature extraction model includes: Extracting a first video feature of the first sample video segment and a third video feature of the third sample video segment through a first network; A second video feature of the second sample video segment is extracted through a second network, the second network has the same network structure as the first network, and the video feature extraction model includes the first network and the second network.

9. The device according to claim 8, characterized in that The video feature extraction model is optimized with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from video features in the memory library, including: With the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library, the first network parameters of the first network in the video feature extraction model are updated.

10. The device according to claim 9, characterized in that Before extracting the second video feature of the second sample video segment through the second network, the device further includes: Based on the first network parameters of the first network and the second network parameters of the second network, momentum update is performed on the second network parameters of the second network.

11. The device according to claim 8, characterized in that The first network includes a feature extraction subnetwork and a feature mapping subnetwork; the first sample segment feature of the first sample video segment is extracted by the feature extraction subnetwork, and the first sample segment feature is feature mapped by the feature mapping subnetwork to obtain the first video feature; If the iteration end condition is met, obtaining a video fingerprint feature extractor based on the video feature extraction model includes: If the iteration end condition is met, the feature extraction subnetwork in the first network is extracted as the trained video fingerprint feature extractor.

12. A video processing device, characterized in that: The device comprises: A target video segment acquisition module is used to acquire a target video segment; The video fingerprint feature extraction module is used to use a preset video fingerprint feature extractor to extract the video fingerprint feature of the target video segment to obtain the video fingerprint feature of the target video segment, and the acquisition method of the video fingerprint feature extractor includes: obtaining a training video segment; randomly and evenly selecting a predetermined number of frame images in the training video segment to obtain a first sample video segment; randomly and evenly selecting the predetermined number of frame images within the time coverage of the first sample video segment in the training video segment to obtain a second sample video segment, wherein the second sample video segment is different from the first sample video segment, and the overlap ratio of the first sample video segment and the second sample video segment is greater than a preset ratio; randomly and evenly selecting the predetermined number of frame images outside the time coverage of the first sample video segment in the training video segment A third sample video segment is obtained by using a frame picture of a video clip, wherein the time interval between the time coverage range of the third sample video segment and the first sample video segment is greater than a preset time interval; a first video feature of the first sample video segment, a second video feature of the second sample video segment, and a third video feature of the third sample video segment are extracted by a video feature extraction model; with the goal of making the first video feature close to the second video feature, the first video feature far away from the third video feature, and the first video feature far away from the video features in the memory library, the video feature extraction model is optimized until a training end condition is reached; the video features in the memory library include video features extracted during historical iterative training; if the training end condition is reached, a video fingerprint feature extractor is obtained based on the optimized video feature extraction model.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

15. A computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images