Method, device and apparatus for video similarity detection

Abstract

The present application provides a method for video similarity detection, which relates to the field of artificial intelligence, and specifically to computer vision. The method includes: a detection device receives a first video, and determines the key frames of the first video based on the first video; inputs the key frames into a feature extraction model to obtain the features of the key frames; determines similar key frames and a second video based on the features of the key frames, wherein the first video is the video to be detected, and the second video is a video in the video library that is similar to the first video; further inputs the key frames and similar key frames into an editing type recognition model to obtain the editing type, wherein the editing type indicates the editing type used for editing between the first video and the second video. The aforementioned feature extraction model and editing type recognition model respectively use different neural network models. This method can more accurately determine the similar videos corresponding to the video to be detected, and further obtain the editing type between the video to be detected and the similar videos.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method for video similarity detection, and an apparatus and device for executing the method. Background Art

[0002] With the rapid development of mobile internet and the ubiquity of smart devices, the production, dissemination, and consumption of videos are ubiquitous in our lives. A wide range of video-based production, consumption, and learning applications and platforms are emerging, such as financial communication, online education, short videos, and entertainment variety shows. Traditional digital video processing software (such as VideoStudio), video transcoding tools (such as Format Factory), and currently developing AI technologies (such as generative adversarial networks) have made video editing simple and fast. Videos can be edited in a variety of ways, including cropping, splicing, rotating, mirroring, blurring, adding / removing text, adding / removing icons, changing color, brightness, and contrast, resizing, adding / removing borders, and applying filters. Edited videos can then be re-entered into the distribution chain.

[0003] Editing existing videos to create a variety of different styles brings greater content diversity and entertainment, but also poses greater challenges to video information security. Detecting similarities between existing videos, generated by various types of editing, and identifying the specific overlapping areas and editing types used are crucial for video-related services such as video similarity queries, copyright authentication, and ad recognition.

[0004] In the prior art, there are technologies for similarity detection of videos, such as: 1. Setting an adaptive threshold based on the correlation between the pattern noise distribution of the video to be detected and similar videos, and performing similarity detection and positioning of the video according to the adaptive threshold; 2. Calculating the similarity between the video to be detected and the existing video based on the hash algorithm; 3. Comparing the global features and local features of frames at fixed intervals in the video to obtain the location of the edited or tampered area. However, the prior art can only judge whether a video is obtained by editing one or more existing videos, and cannot determine what type of editing operation is performed on one or more existing videos to obtain the video to be detected. Therefore, how to determine videos similar to the video to be detected and detect the type of editing between the video to be detected and similar videos are technical problems that need to be solved in video similarity detection. Summary of the Invention

[0005] The present application provides a method for video similarity detection, which can determine videos similar to a video to be detected, and further determine the editing type used between the video to be detected and the similar videos.

[0006] In the first aspect, the present application provides a method for video similarity detection, the method comprising: receiving a first video, determining the key frames of the first video based on the first video; inputting the key frames into a feature extraction model to obtain the features of the key frames; determining similar key frames and a second video based on the features of the key frames, wherein the second video is the video where the similar key frames are located, and the second video is similar to the first video; inputting the key frames and the similar key frames into an editing type recognition model to obtain an editing type, wherein the editing type indicates the editing type used for editing between the first video and the second video. The method provided by the method for video similarity detection not only determines the similar videos corresponding to the video to be detected, but also further obtains the editing type between the video to be detected and the similar video based on the determination of the similar videos, which makes the method more advantageous when used in applications such as video copyright authentication and advertisement recognition.

[0007] In a possible implementation of the first aspect, the method further includes: outputting the second video or information about the second video to a display module, wherein the information about the second video includes a name of the second video; and outputting the edit type to the display module. The display module intuitively displays videos similar to the video to be detected and the edit types between the videos, allowing a user to intuitively access this information.

[0008] In a possible implementation of the first aspect, the edit type recognition model includes a first feature extraction branch, a second feature extraction branch, and a predictor; inputting the key frame and the similar key frame into the edit type recognition model, and obtaining the edit type specifically includes: inputting the key frame into the first feature extraction branch, and inputting the similar key frame into the second feature extraction branch; the first feature extraction branch extracts features from the key frame and outputs the edit features of the key frame; the second feature extraction branch extracts features from the similar key frame and outputs the edit features of the similar key frame; the key frame and the similar key frame are input into the predictor, and the predictor outputs the edit type. This method uses an edit type recognition module including a feature extraction branch and a predictor, so that the obtained edit type has a high accuracy rate.

[0009] In a possible implementation of the first aspect, the method further includes: calculating a similarity between the first video and the second video; and outputting the similarity to a display module. This method provides similarity information, further enriching detection results for the video to be detected, which can be used by users or other modules.

[0010] In one possible implementation of the first aspect, determining the similar key frame and the second video based on the features of the key frame specifically includes: querying a video library based on the features of the key frame, obtaining the similar key frame from the video library, wherein the features of the similar key frame are similar to those of the key frame; and determining the second video based on the similar key frame. This method of determining similar videos based on similar key frames improves the accuracy of video similarity detection.

[0011] In one possible implementation of the first aspect, the feature extraction model and the edit type recognition model each employ a different neural network model. Employing trained neural network models in both the feature extraction model and the edit type recognition model enables efficient video similarity detection in the present application and high accuracy of the detection results.

[0012] In a possible implementation manner of the first aspect, the editing type includes one or more of the following operations: cropping, splicing, rotating, mirroring, blurring, adding text, adding icons, changing colors, changing brightness, and changing contrast.

[0013] In one possible implementation of the first aspect, the method further includes: determining similar shots in the similar videos based on the similar key frames, wherein the similar shots are shots similar to the shot containing the key frames; and outputting a correspondence between the similar shots and the shot containing the key frames to a display module. This method can also accurately output the correspondence between the similar shots and the shot containing the key frames, thereby enriching the results of video similarity detection and facilitating further planning based on the detection results.

[0014] In a possible implementation manner of the first aspect, the editing type may also be an editing type used for editing between the similar shot and the shot where the key frame is located.

[0015] In a possible implementation manner of the first aspect, the similarity between the video and the similar videos further includes the similarity between shots of the video and similar shots in corresponding similar videos.

[0016] In a possible implementation of the first aspect, determining the key frames of the video based on the video specifically includes: performing a structural analysis on the video based on the content of the video to obtain shots of the video, where the shots are a collection of video frames in the video that express a section of continuous background picture content; and determining the key frames in the shots, where the key frames are video frames that represent the main picture content of the shots.

[0017] In the second aspect, the present application provides a detection device, including: a structural analysis module for receiving a first video and determining the key frames of the first video based on the first video; a feature extraction model for obtaining the features of the key frames based on the key frames; a comparative analysis module for determining similar key frames and a second video based on the features of the key frames, wherein the second video is the video where the similar key frames are located, and the second video is similar to the first video; an editing type recognition model for obtaining an editing type based on the key frames and the similar key frames, wherein the editing type indicates the editing type used for editing between the first video and the second video.

[0018] In a possible implementation of the second aspect, the detection device also includes: an output module, used to output the second video or the information of the second video to the display module, wherein the information of the second video includes the name of the second video; and also used to output the editing type to the display module.

[0019] In a possible implementation of the second aspect, the editing type recognition model includes a first feature extraction branch, a second feature extraction branch and a predictor; the first feature extraction branch is used to receive the key frame, perform feature extraction on the key frame, and output the editing features of the key frame; the second feature extraction branch is used to receive the similar key frame, perform feature extraction on the similar key frame, and output the editing features of the similar key frame; the predictor is used to obtain the editing type based on the editing features of the key frame and the editing features of the similar key frame.

[0020] In a possible implementation of the second aspect, the comparison and analysis module is further configured to calculate a similarity between the first video and the second video; and the output module is further configured to output the similarity to a display module.

[0021] In a possible implementation of the second aspect, the structural analysis module is specifically used to: query the video library based on the characteristics of the key frame, obtain the similar key frame in the video library, and the characteristics of the similar key frame are similar to the characteristics of the key frame; determine the second video based on the similar key frame.

[0022] In a possible implementation of the second aspect, the feature extraction model and the edit type recognition model respectively adopt different neural network models.

[0023] In a possible implementation manner of the second aspect, the editing type includes one or more of the following operations: cropping, splicing, rotating, mirroring, blurring, adding text, adding icons, changing colors, changing brightness, and changing contrast.

[0024] In a possible implementation of the second aspect, the comparative analysis module is further used to determine similar shots in the similar videos based on the similar key frames, wherein the similar shots are shots similar to the shots where the key frames are located; and the output module is further used to output the correspondence between the similar shots and the shots where the key frames are located to the display module.

[0025] In a possible implementation manner of the second aspect, the editing type may also be an editing type used for editing between the similar shot and the shot where the key frame is located.

[0026] In a possible implementation manner of the second aspect, the similarity between the video and the similar videos further includes the similarity between shots of the video and similar shots in corresponding similar videos.

[0027] In a possible implementation of the second aspect, the structural analysis module is specifically used to: perform structural analysis on the video according to the content of the video to obtain a shot of the video, where the shot is a collection of video frames in the video that express a section of continuous background picture content; and determine the key frame in the shot, where the key frame is a video frame that represents the main picture content of the shot.

[0028] In a third aspect, the present application provides a computing device system comprising at least one computing device, each computing device comprising a memory and a processor, the memory of the at least one computing device being used to store computer instructions; the processor of the at least one computing device executing the computer instructions stored in the memory to perform the method provided in the first aspect or any possible implementation of the first aspect.

[0029] In a fourth aspect, the present application provides a non-transitory readable storage medium. When the non-transitory readable storage medium is executed by a computing device, the computing device performs the method provided in the first aspect or any possible implementation of the first aspect. The storage medium stores a program. The storage medium includes, but is not limited to, volatile memory, such as random access memory, non-volatile memory, such as flash memory, hard disk drive (HDD), and solid state drive (SSD).

[0030] In a fifth aspect, the present application provides a computer program product, comprising computer instructions. When executed by a computing device, the computing device performs the method provided in the aforementioned first aspect or any possible implementation of the first aspect. The computer program product may be a software installation package. When the method provided in the aforementioned first aspect or any possible implementation of the first aspect is required, the computer program product may be downloaded and executed on the computing device. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0032] Figure 1 A schematic diagram illustrating the relationship between videos, video clips, shots, and key frames provided in an embodiment of the present application;

[0033] Figure 2 A schematic diagram of the deployment of a detection device provided in an embodiment of the present application;

[0034] Figure 3 A schematic diagram of another deployment of a detection device provided in an embodiment of the present application;

[0035] Figure 4 A schematic diagram of the structure of a computing device 100 equipped with a detection device provided in an embodiment of the present application;

[0036] Figure 5 A schematic diagram of the structures of a training device 200 and a detection device 300 provided in an embodiment of the present application;

[0037] Figure 6 A flowchart of a method for video similarity detection provided in an embodiment of the present application;

[0038] Figure 7 A schematic diagram of the structure of a feature extraction model provided in an embodiment of the present application;

[0039] Figure 8 A schematic diagram of the structure of an edit type identification model provided in an embodiment of the present application;

[0040] Figure 9 A schematic diagram of displaying information output by a detection device in text form provided in an embodiment of the present application;

[0041] Figure 10 A schematic diagram of displaying information output by a detection device in the form of a visual interface provided in an embodiment of the present application;

[0042] Figure 11A flowchart of a method for determining shots and key frames provided in an embodiment of the present application;

[0043] Figure 12 A schematic diagram of a computing device system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will describe the solutions in the embodiments provided in this application in conjunction with the drawings in this application.

[0045] Video is an electrical signal that stores continuous images of the real world. Figure 1 This is a diagram showing the relationship between videos, video clips, shots, and key frames. A video can be divided into multiple video clips based on the content of its screen. Each video clip records the video content of a relatively complete plot. A video clip can be divided into multiple shots. The video content in each shot is a single shot of the camera with a continuous background. A shot contains one or more video frames, and each video frame is an independent image. In a shot, the video frame that can describe the main content of the current shot is called the key frame of the shot. A shot can have one or more key frames. A variety of methods can be used to determine the key frames in a shot so that the content of a shot can be represented by the content in the key frames. By Figure 1 As we can see, a video can be divided into multiple levels, with the smallest unit being the video frame. In other words, one or more video frames (including keyframes) form a shot, shots of different scenes form a video clip, and one or more video clips form a complete video. The relationship between a video and its corresponding video clips, shots, and keyframes is called a video's multi-level structure. Multi-level structural analysis of a video can be performed based on its content.

[0046] This application provides a method for video similarity detection. The method analyzes and detects videos based on their multi-level structure, and determines whether the video to be detected is similar to videos in a video library. If the video is similar to one or more other videos, the method further obtains information about the locations where the video is similar to the similar videos and the types of editing used to determine the shots of the video relative to the similar videos, such as cropping, splicing, rotation, mirroring, blurring, adding text, adding icons, changing colors, brightness, and contrast.

[0047] It is worth noting that, in the embodiment of the present application, the similarity between two videos means that the two videos include one or more similar key frames, that is, one of the two videos contains one or more key frames that are obtained by editing one or more key frames contained in the other video through one or more types of editing methods. For example, all the key frames in the first video segment of video A are obtained by adding subtitles to the key frames in video B, and the key frames in the 2-N (N is a positive integer greater than or equal to 1) video segments of video A are obtained by removing icons from some key frames in video C, then video B and video C are considered to be similar videos to video A, that is, video A is similar to video B, and video A is also similar to video C.

[0048] The embodiment of the present application provides a method for video similarity detection, which is performed by a detection device. The deployment of the detection device is relatively flexible.

[0049] Figure 2 This is a deployment diagram of a detection device provided in an embodiment of the present application. The detection device can be deployed in a cloud environment. The cloud environment is an entity that uses basic resources to provide cloud services to users in a cloud computing model. The cloud environment includes a cloud data center and a cloud service platform. The cloud data center includes a large number of basic resources (including computing resources, storage resources, and network resources) owned by the cloud service provider. The computing resources included in the cloud data center can be a large number of computing devices (such as servers). The detection device can be a server in the cloud data center for detecting videos; the detection device can also be a virtual machine created in the cloud data center for detecting videos; the detection device can also be a software device deployed on a server or virtual machine in the cloud data center, the software device is used to detect videos, and the software device can be distributedly deployed on multiple servers, or distributedly deployed on multiple virtual machines, or distributedly deployed on virtual machines and servers. As Figure 2 As shown, the detection device is abstracted by the cloud service provider into a cloud service for video similarity detection on the cloud service platform and provided to users. After the user purchases the cloud service on the cloud service platform, the cloud environment uses the detection device to provide the user with the video similarity detection cloud service. The user can upload the video to be detected to the cloud environment through the application program interface (API) or the web interface provided by the cloud service platform. The detection device receives the video to be detected and detects the video to be detected. The detection result is returned by the detection device to the user's terminal, or the detection result is stored in the cloud environment, for example: presented on the web interface of the cloud service platform for the user to view.

[0050] When the detection device is a software device, the detection device can be logically divided into multiple parts, each of which has different functions (for example, the detection device includes a structure analysis module, a feature extraction model, a comparison analysis module, an edit type recognition model, and an output module). The several parts of the detection device can be deployed in different environments or devices, for example: Figure 3 As shown, part of the detection device is deployed on the terminal computing device (such as: terminal server, smart phone, laptop, tablet computer, personal desktop computer, smart camera), and the other part is deployed in the data center (specifically deployed on the server or virtual machine in the data center). The data center can be a cloud data center or an edge data center. The edge data center is a collection of edge computing devices deployed close to the terminal computing device.

[0051] The various parts of the detection device deployed in different environments or devices collaborate to realize the function of video editing type detection. For example, in one scenario, a structural analysis module in the detection device is deployed in a smart phone. The smart phone obtains a video and uses the structural analysis module to perform structural analysis on the video. The smart phone sends the data after structural analysis to the data center through the network. The data center is deployed with a feature extraction model, a comparative analysis module, an editing type recognition model, and an output module. These modules / models further process the data after structural analysis and finally obtain the detection results. The data center sends the detection results to the smart phone, so that users using the smart phone can obtain the detection results of the video. It should be understood that this application does not make a restrictive division between which parts of the detection device are deployed on the terminal computing device and which parts are deployed on the data center. In actual application, it can be adaptively deployed according to the computing power of the terminal computing device or the specific application requirements. It is worth noting that in one embodiment, the detection device can also be deployed in three parts, wherein one part is deployed on the terminal computing device, one part is deployed on the edge data center, and one part is deployed on the cloud data center.

[0052] When the detection device is a software device, the detection device can also be deployed separately on a computing device in any environment (for example, separately deployed on a terminal computing device or separately deployed on a computing device in a data center), such as Figure 4As shown, the computing device 100 includes a bus 101, a processor 102, a communication interface 103 and a memory 104. The processor 102, the memory 104 and the communication interface 103 communicate with each other through the bus 101. Among them, the processor 102 can be a central processing unit (CPU). The memory 104 can include a volatile memory (NVM), such as a random access memory (RAM). The memory 104 can also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, an HDD or an SSD. The memory 104 stores the executable code included in the detection device, and the processor 102 reads the executable code in the memory 104 to execute the method for video similarity detection. The memory 104 can also include software modules required for other running processes such as an operating system. The operating system can be LINUX TM , UNIX TM , WINDOWS TM wait.

[0053] When the detection device executes the method for video similarity detection provided in the embodiment of the present application, it is necessary to adopt a neural network model. The neural network model is a mathematical calculation model that imitates the structure and function of a biological neural network (the central nervous system of an animal). A neural network model can include multiple neural network layers with different functions, each layer including parameters and calculation formulas. Depending on the different calculation formulas or functions, different layers in the neural network model have different names. For example, the layer that performs convolution calculation is called a convolution layer, and the convolution layer is often used to extract features from the input signal (for example, an image). A neural network model can also be composed of a combination of multiple existing neural network models. Neural network models with different structures can be used in different scenarios (for example, classification, recognition) or provide different effects when used in the same scenario. The different structures of the neural network models specifically include one or more of the following: the number of network layers in the neural network model is different, the order of each network layer is different, and the weights, parameters or calculation formulas in each network layer are different. There are many different neural network models with high accuracy for application scenarios such as recognition or classification in the industry. Among them, some neural network models can be trained with a specific training set to complete a task alone or in combination with other neural network models (or other functional modules) to complete a task. Some neural network models can also be used directly to complete a task alone or combined with other neural network models (or other functional modules) to complete a task.

[0054] In one embodiment of the present application, a method for performing video similarity detection requires the use of two different neural network models: one is a neural network model for extracting features from the video to be detected, referred to as a feature extraction model; the other is a model for identifying the edit types between two similar videos, referred to as an edit type recognition model. Before being used for video edit type detection, the feature extraction model and the edit type recognition model can be trained by a training device. The training device uses different training sets to train the feature extraction model and the edit type recognition model, respectively. The feature extraction model and the edit type recognition model trained by the training device are deployed in a detection device, which is used by the detection device to detect video edit types.

[0055] Figure 5 A structural diagram of a training device 200 and a detection device 300 is provided. Figure 5 The structure and functions of the training device 200 and the detection device 300 are introduced. It should be understood that the embodiments of the present application are only exemplary divisions of the structure and functional modules of the training device 200 and the detection device 300, and the present application does not impose any limitations on their specific divisions.

[0056] The training device 200 is used to train the feature extraction model 203 and the edit type recognition model 204 respectively. Two different training sets are required to train the feature extraction model 203 and the edit type recognition model 204, which are respectively called the feature extraction training set and the edit type recognition training set. The obtained feature extraction training set and edit type recognition training set are stored in a database. The acquisition device can acquire multiple training videos or training images, and the acquired multiple training videos or training images are processed and annotated manually or by the acquisition device to form a training set. When the acquisition device acquires multiple training videos, the acquisition device divides each training video into shots, determines key frames in the divided shots, uses the determined key frames as training images, and then processes and annotates the training images to construct a training set. When the training device 200 starts training the feature extraction model 203, the initialization module 201 first initializes the parameters of each layer in the feature extraction model 203 (i.e., assigns an initial value to each parameter), and then the training module 202 reads the training images in the feature extraction training set in the database to train the feature extraction model 203 until the loss function in the feature extraction model 203 converges or all the training images in the feature extraction training set are used for training, and then the training of the feature extraction model 203 is completed. Similarly, when the training device 200 starts training the edit type recognition model 204, the initialization module 201 first initializes the parameters of each layer in the edit type recognition model 204 (i.e., assigns an initial value to each parameter), and then the training module 202 reads the training images in the edit type recognition training set in the database to train the edit type recognition model 204 until the loss function in the edit type recognition model 204 converges or all the training images in the edit type recognition training set are used for training, and then the training of the edit type recognition model 204 is completed. It is worth noting that the feature extraction model 203 and the edit type recognition model 204 can also be trained separately by two training devices, and the feature extraction model 203 and / or the edit type recognition model 204 do not need to be trained by the training device 200. For example, the feature extraction model 203 and / or the edit type recognition model 204 uses a neural network model that has been trained by a third party and has good accuracy for feature extraction and / or type recognition. In one embodiment of the present application, it is also not necessary for the acquisition device to acquire training images or training videos and construct a feature extraction training set and / or edit type recognition training set. For example, the feature extraction training set and / or the edit type recognition training set is obtained directly from a third party.

[0057] It is worth noting that in the embodiment of the present application, the basic feature extraction part of any one of the neural network models such as Alexnet, Resnet, Mobilenet, Densenet, etc. can be used as the feature extraction model 203, and the loss function used to train the feature extraction model 203 can be a triplet loss function. The feature extraction training set includes two types of image groups, one is labeled as a similar image group, that is, the label of the similar image group is set to similar, and the other is labeled as a dissimilar image group, that is, the label of the dissimilar image group is set to dissimilar. A feature extraction training set includes multiple similar image groups and multiple dissimilar image groups. The similar image group includes two images, one of which is the original image and the other is a generated image obtained by editing the original image through one or more editing types (for example, the original image and the generated image after rotating and cropping the original image constitute a similar image group). Since the original image and the generated image are obtained through one or more editing types, the original image and the generated image have certain similar features. The dissimilar image group also includes two (or more) images, and there is no relationship between the two images before and after editing. The feature extraction training set is used to train the feature extraction model. Similar image groups are used as positive samples for training the feature extraction model, allowing the feature extraction model to learn the similar features between two images in the similar image group. Dissimilar image groups are used as negative samples for training the feature extraction model, making the feature extraction model more capable of distinguishing similar from dissimilar image groups. A feature extraction model trained with the feature extraction training set can more accurately extract features from the detected keyframes.

[0058] It is worth noting that in an embodiment of the present application, the editing type recognition training set also includes multiple image groups, and each image group consists of an original image and a generated image corresponding to the original image. The generated image corresponding to the original image is the original image obtained by one or more types of editing operations. Each image group is provided with one or more labels, and the label of each image group is the editing type of the generated image in this image group. For example: an image group includes an original image and a generated image obtained by rotating and cropping the original image, then the image group has two labels, namely rotation and cropping (or labels of other representations corresponding to rotation and cropping). It should be understood that the images in the image group in the editing type recognition training set can be the same as the images in the similar image group in the feature extraction training set, that is, the obtained training images can be reused by the two training sets, but the labels of the image group in the editing type recognition training set are different from those of the similar image group in the feature extraction training set. The edit type recognition training set includes image groups with multiple labels, with multiple image groups for each label. Each image group can be a single-label image group (for example, if there are multiple image groups with the label "rotate," then the generated images in these multiple image groups are all obtained by rotating the original image), or a multi-label image group (for example, if there are multiple image groups with the labels "rotate," "crop," and "add icon," then the generated images in these multiple image groups are all obtained by rotating, cropping, and adding icons to the original image). The edit type recognition model 204 can use multi-label cross entropy as the loss function when being trained.

[0059] The feature extraction model 203 and the editing type recognition model 204 trained by the training device 200 can be used to perform feature extraction and video / image editing type recognition respectively. Figure 5 As shown, the trained feature extraction model 203 and the edit type recognition model 204 are deployed to the detection device 300. In the detection device 300, the trained feature extraction model 203 is called the feature extraction model 302, and the trained edit type recognition model 204 is called the edit type recognition model 304.

[0060] like Figure 5 As shown, the detection device 300 includes a structure analysis module 301, a feature extraction model 302, a comparison analysis module 303, an edit type recognition model 304, and an output module 305.

[0061] The structural analysis module 301 is used to receive a video to be tested (which can be a complete video file or a video clip, such as a real-time video stream), perform structural analysis on the video, and decompose the video into one or more shots (or decompose the video into one or more video clips, and then decompose each video clip into one or more shots). The structural analysis module 301 is also used to determine one or more key frames in each shot that can represent the content of the shot. The structural analysis module 301 outputs a piece of structural data corresponding to the video structure, which indicates the position of each shot in the video and the location of the key frames of each shot.

[0062] The feature extraction model 302 is connected to the structure analysis module 301 via a communication path, and is used to read the key frame of each shot in the video according to the structure data, extract features from each key frame, and output the features of each key frame.

[0063] Comparative analysis module 303 is connected to feature extraction module 302 via a communication path and is configured to query a video library based on the features corresponding to each keyframe, and retrieve one or more videos similar to the video to be detected from the video library. Comparative analysis module 303 is also configured to determine the similarity between the similar videos and the video to be detected, as well as the correspondence and similarity between keyframes or shots in the similar videos that are similar to the video to be detected.

[0064] The editing type recognition model 304 is connected to the comparison and analysis module 303. The editing type recognition model 304 inputs each similar key frame together with the corresponding key frame in the video to be detected into the editing type recognition model 304 based on the similar key frames in the similar videos obtained by the comparison and analysis module 303, and obtains the editing type between the similar key frames in the similar videos and the corresponding key frames in the video to be detected according to the editing type recognition model 304.

[0065] Output module 305 is connected to comparison and analysis module 303 and edit type identification model 304 via communication paths. Output module 305 outputs the edit types between one or more videos similar to the video to be detected, as determined by comparison and analysis module 303, and the edit types between similar key frames in the similar videos and corresponding key frames in the video to be detected. Optionally, output module 305 outputs the similarity between the similar videos and the video to be detected, as well as the location information of key frames or shots in the similar videos that are similar to the video to be detected.

[0066] Both the training device 200 and the detection device 300 can be software devices. When both the training device 200 and the detection device 300 are software devices, the training device 200 can be deployed on the same computing device as the detection device 300 (for example, deployed on the same server, or deployed on two different virtual machines in the same server). The training device 200 can also be deployed on different computing devices from the detection device 300 (for example, the training device 200 is deployed on one or more servers in a cloud environment, and the detection device 300 is deployed on one or more servers in an edge environment). The deployment of the training device 200 is also relatively flexible. Like the deployment method of the detection device described above, it can be deployed entirely on the same computing device, or its parts can be deployed separately on different computing devices. Different computing devices work together to run the various parts of the training device 200 to realize all the functions of the training device 200.

[0067] The following combination Figure 6 A method for video similarity detection provided by an embodiment of the present application is specifically described.

[0068] S401: Receive a video to be detected, and determine shots and key frames according to the content of the video to be detected.

[0069] Specifically, the detection device 300 obtains a video to be tested for edit type (for example, the detection device 300 receives a video uploaded by a user or administrator, or receives a video captured by another device in real time). The device then performs a structural analysis of the video based on its content, identifying shots and keyframes within each shot. A video to be tested can include multiple shots. A shot is a single shot captured by a camera with a continuous background. A shot typically represents a scene, and the content within a shot can be represented by keyframes.

[0070] In this step, the method for determining the shot can adopt a sliding window method to determine the shot boundary based on the grayscale histogram difference between the video frames at the previous and next moments in the video to be detected. The video to be detected is segmented based on the shot boundary, and key frames are selected in each segmented shot based on the video frame content within the shot. In this step, a piece of structural data corresponding to the video is obtained by performing structural analysis on the video to be detected. The structural data can be expressed as {[s1, e1, k 10 , k 11 ,…],[s2,e2,k 20 , k 21 ,…],…,[s n , e n , k n0 , k n1 ,…]}, where [s1, e1, k10 , k 11 , ...] represents a shot, s1 is the frame sequence of the starting video frame of the shot in the entire video, e1 is the frame sequence of the ending video frame of the shot in the entire video, k 10 The offset number of frames of a key frame in the shot relative to the starting video frame.

[0071] It should be understood that the present application does not limit the specific implementation method for determining shots and key frames based on the content of the video to be detected. Different methods can be used to determine shots and key frames for different videos to be detected. A specific solution for determining shots and key frames based on the content of the video to be detected will be introduced later. This step can obtain all key frames in the video to be detected.

[0072] It should be noted that in this step, the shots contained in the video to be detected are first determined, and then the key frames in the shots are further determined. Optionally, in other embodiments, the shots may be not determined and the key frames may be directly obtained.

[0073] S402: The feature extraction model extracts features from the key frames.

[0074] Specifically, the key frame of each shot obtained in the aforementioned step S401 is a two-dimensional image, such as Figure 7 As shown, each key frame is input into the trained feature extraction model, and the feature extraction model outputs the features of each key frame. The features of the key frame can be a multidimensional matrix, and the features of the key frame represent the implicit characteristics of the picture content of the key frame. This application does not impose specific restrictions on the structure of the neural network model used by the feature extraction model. The feature extraction model can use the industry-common neural network model for image classification or the backbone of the neural network model for image recognition, or it can be an improved neural network model. Figure 7The feature extraction model shown uses a convolutional neural network model, which includes multiple convolutional layers, each of which includes one or more convolution kernels, and each of which includes multiple parameters. The size of each convolution kernel can be the same or different (for example, the first convolution layer of the feature extraction model can have 16 convolution kernels of size 7*7). After the key frame (or tensor) is input into a convolution layer and convolved with each convolution kernel in the convolution layer, the convolution layer outputs a tensor. The tensor output by the convolution layer is a three-dimensional array, including multiple values, for example: a tensor of scale W*H*L (where W represents the width of the tensor, H represents the height of the tensor, and L represents the number of channels of the tensor, and W, H, and L are all large numbers). A convolution layer (a natural number greater than 0) includes W*H*L values. The number of convolution kernels included in the convolution layer determines the number of channels of the tensor output by the convolution layer. For example, a tensor of scale W*H*L (where W represents the width of the tensor, H represents the height of the tensor, and L represents the number of channels of the tensor, and W, H, and L are all natural numbers greater than 0) is input into a convolution layer containing J convolution kernels of size 1*1, and then convolved with J 1*1 convolution kernels in the convolution layer. The scale of the tensor output by the convolution layer is W*H*J (J is a natural number greater than 0). The size and number of convolution kernels in different convolution layers can be the same or different. The scale of the tensor output by each convolution layer is determined by the key frame (or tensor) input to the convolution layer, the size and number of convolution kernels in the convolution layer, and the convolution calculation method. After the key frame is input into the feature extraction network, the tensor output by the last convolution layer is used as the feature of the key frame and output by the feature extraction model.

[0075] It is worth noting that all key frames in the video to be detected obtained in step S401 are subjected to the operation of step S402. Therefore, step S402 obtains the features of all key frames in the video to be detected.

[0076] S403: Determine a video similar to the video to be detected based on the features of the key frames.

[0077] Specifically, step S402 obtains the features of all key frames in the video to be detected, compares the features of each key frame with the features of key frames in all videos in the video library, determines similar key frames that are similar to the key frames in the video to be detected, and determines the video to which the similar key frames belong. The video to which the similar key frames belong is called a similar video, and the similar video is similar to the video to be detected. The similarity between the video to be detected and the similar video is further determined, as well as the correspondence and position between the shot containing the similar key frame and the shot containing the corresponding key frame.

[0078] It is worth noting that the video library used in step S403 is a pre-organized and calculated video library, which includes multiple videos. The same video structure analysis operation as the aforementioned step S401 is performed on each video to determine the shots and key frames in each video, that is, each video in the video library corresponds to a piece of structural data, and the structural data indicates the start and end frames of each shot in the video and the key frames within each shot. The present application also performs the same method as the aforementioned step S402 on each key frame in the video library, that is, it also performs feature extraction on the key frames of each video in the video library to obtain the features of each key frame. Therefore, the video library in the present application stores multiple videos, the structural data corresponding to each video, and the features of the key frames in each video.

[0079] This application does not limit the source of the videos in the video library. When creating a video library, videos can be adaptively collected according to the specific application scenarios of the video similarity detection method provided by this application. For example, if the film and television work protection department uses the video similarity detection method to identify pirated videos, the videos in the video library can be the original film and television works that can be collected. The richer the video library, the greater the probability of obtaining similar videos with high similarity to the video to be detected. It is worth noting that the operation of performing video structure analysis on the videos in the video library to determine the shots and key frames of the video can be performed at any time before executing step S403.

[0080] The specific process of step S403 is described as follows:

[0081] S4031: Determine similar key frames in the video library based on the acquired features of each key frame in the video to be detected.

[0082] The features of each key frame in the video to be detected are compared with the features of the key frames of all videos in the video library. The comparison method can be adopted to calculate the similarity one by one, and the key frames of the videos in the video library whose similarity is greater than a preset threshold value are determined as similar key frames of the key frames to be compared. This application does not limit the specific method of similarity calculation. If there are no similar key frames in the video library that are similar to any key frame in the video to be detected, the detection of the editing type of the video is terminated. If there are similar key frames in the video library that are similar to the key frames in the video to be detected, the subsequent steps are carried out. It should be understood that there may be one or more similar key frames for each key frame of the video to be detected in the video library.

[0083] S4032: Determine similar videos that are similar to the video to be detected based on similar key frames.

[0084] Specifically, in one embodiment, a graph search method may be used to determine similar videos that are similar to the video to be detected:

[0085] The key frames in the video to be detected are arranged in time sequence, and each key frame is matched with all similar key frames in the video library that are similar to the key frame. All key frames in the video to be detected and the similar key frames corresponding to the key frames can form a graph. The key frames in the video to be detected and their corresponding similar key frames are regarded as nodes in the graph. According to the time sequence of the key frames, the similar key frames corresponding to each key frame in the graph are used as nodes to construct a path for each similar key frame. Each path includes multiple nodes and edges connecting the nodes. When determining the path, the nodes are determined in sequence according to the time sequence of the key frames of the video to be detected. For each key frame, a similar key frame corresponding to the key frame is found as a node on a path. The determined similar key frame belongs to the same video as the similar key frames already on the path. Therefore, the nodes on each path of the multiple paths obtained meet the condition that the similar key frames on the same path are in the same video (if one or more similar key frames corresponding to a key frame do not belong to the same video as the similar key frames already on any path, the key frame is skipped). Thus, each path corresponds to a video, which is called a similar video. Based on the similar keyframes on each path and the structural data of the video corresponding to that path stored in the video library, the shot of each similar keyframe in the similar video is determined. This shot is called a similar shot to the shot containing the corresponding keyframe. The similar shot and the shot containing the corresponding keyframe in the video to be detected are called a similar shot pair. Optionally, the similarity between the keyframe contained in one shot and the similar keyframe contained in the other shot in a similar shot pair can be determined as the similarity of the similar shot pair.

[0086] Calculate the similarity between each similar video and the video to be detected. The similarity between a similar video and the video to be detected can be obtained by taking a weighted average of the similarities between each similar shot pair consisting of shots in the similar video and shots in the video to be detected. Alternatively, the ratio of the sum of the durations of the shots containing similar key frames in the similar video to the total duration of the video can be used as the similarity between the similar video and the video to be detected.

[0087] S404: Input the key frames in the video to be detected and the similar key frames in the corresponding similar videos into an edit type recognition model, and the edit type recognition model outputs the edit type.

[0088] Specifically, after step S403, one or more similar videos similar to the video to be detected are obtained, and similar key frames in the similar videos are obtained. The similar key frames in each similar video and the key frames in the corresponding video to be detected are formed into a key frame group, and each similar video corresponds to one or more key frame groups. Each key frame group is input into the editing type recognition model, and the editing type recognition model extracts and predicts the editing features between similar key frames and key frames in the key frame group, and outputs one or more editing types between the key frames in the key frame group and the similar key frames. The one or more editing types between the key frames and the similar key frames represent one or more editing types used for editing between the key frames and the similar key frames. The conversion between the key frames and the similar key frames can be achieved through the editing of these one or more editing types. Each key frame group in each similar video is sequentially subjected to editing feature extraction and prediction by the editing type recognition model, and the editing types between the similar key frames in each similar video and the corresponding key frames in the video to be detected are obtained. It is worth noting that since key frames and similar key frames are video frames representing the content of the shots in the video to be detected and the content of similar shots in similar videos respectively, one or more editing types between the key frames and similar key frames can be used to represent one or more editing types between the shot where the key frame is located and the similar shot in the corresponding similar video.

[0089] The edit type recognition model uses a pre-trained neural network model. Figure 8 A model is identified for an exemplary edit type. Figure 8 The illustrated edit type recognition model includes two feature extraction branches, called a first feature extraction branch and a second feature extraction branch, and a predictor. The outputs of the two feature extraction branches serve as inputs to the predictor, which outputs one or more predicted edit types. The two feature extraction branches in the edit type recognition model are constructed using the same multiple convolutional layers (with the same number of convolutional layers and the same parameters). Keyframes and similar keyframes in a keyframe group are input into the first and second feature extraction branches, respectively. The first feature extraction branch performs convolution calculations on the keyframes, and the last convolution layer of the first feature extraction branch outputs the edit features of the keyframes. The second feature extraction branch performs convolution calculations on similar keyframes, and the last convolution layer of the second feature extraction branch outputs the edit features of similar keyframes. The edit features of similar keyframes and keyframes serve as inputs to the predictor, which calculates and predicts the edit types between the keyframes and similar keyframes. Based on the relationship between the keyframes and similar keyframes, the predictor can output one or more edit types.

[0090] It is worth noting that the one or more edit types output by the predictor of the edit type recognition model represent the one or more edit types used for editing between the input keyframe and similar keyframes, and also represent the one or more edit types used for editing between the shot containing the keyframe and the similar shot containing the similar keyframe. If all keyframe groups corresponding to a similar video are evaluated by the edit type recognition model to obtain the same one or more edit types, then the edit types between the similar video and the video to be detected are the one or more edit types. If multiple keyframe groups corresponding to a similar video are evaluated by the edit type recognition model to obtain different types of edit types, then the edit types between the similar video and the video to be detected are all different types of edit types. For example, if there are three keyframe groups between the similar video and the video to be detected, i.e., there are three similar keyframes in the similar video that are similar to the keyframes of the video to be detected, and the three keyframe groups are respectively evaluated by the edit type recognition model to obtain three outputs, namely (rotate, crop), (add icon), and (mirror, crop), then the edit types between the similar video and the video to be detected are (rotate, crop, add icon, mirror).

[0091] It is worth noting that there are three situations for one or more editing types used for editing between the video to be detected and similar videos: 1. The video to be detected can be obtained by a similar video through editing operations of one or more editing types; 2. The video to be detected can be obtained by a similar video through the opposite editing operations corresponding to one or more editing types, that is, the video to be detected can obtain a similar video through editing operations of one or more editing types; 3. When the obtained editing types include multiple, the video to be detected can be obtained by a similar video through editing operations of one or more editing types and the opposite editing operations of another one or more editing types.

[0092] This application does not limit the specific classification and name of the editing type. In one way of dividing editing types, for some editing operations that are opposite to each other, such as adding or removing icons, adding or removing filters, the two editing operations that are opposite to each other can be divided into two editing types. For example, adding or removing icons are respectively set to add icon editing type, and remove icon editing type. For the case where the editing types are divided in this way, the editing type recognition model can output (1) the editing type adopted by the video to be detected obtained from similar videos, or it can output (2) the editing type adopted by the similar video to be detected, or it can output (3) these two editing types that are opposite to each other. For some editing operations that do not have opposite operations, there is only one editing type, and the editing type recognition model outputs this editing type, indicating that the similar video is edited by this editing type to obtain the video to be detected.

[0093] Here are some examples to help you understand:

[0094] Suppose the first video has a first keyframe and a third keyframe, and the second video has a second keyframe and a fourth keyframe. If the difference between the first and second keyframes is that the second keyframe has an additional icon, then it can be assumed that the first video is the source video, the second video is the video generated by editing the first video (source video), and the edit type used for editing between the first and second videos is "add icon." Alternatively, it can be assumed that the second video is the source video, the first video is the video generated by editing the second video (source video), and the edit type used for editing between the first and second videos is "remove icon." In other words, because "add icon" and "remove icon" are opposite operations, there may be two detection results, and the role of "source video" is exactly opposite in these two detection results. In actual applications, one or both detection results can be output to the user. In addition to editing operations such as adding (or removing) icons, this situation also occurs with editing operations such as adding text and applying filters.

[0095] As another example, suppose the difference between the third keyframe and the fourth keyframe is that the fourth keyframe can be obtained by performing a "mosaic" operation on the third keyframe. Unlike the above example, there is no reverse operation for the "mosaic" operation, so the editing type in this example is unique. Therefore, in this example: the first video is the source video, the second video is the video generated after the first video (source video) is edited, and the editing type used for editing between the first video and the second video is "mosaic". The editing type recognition model can output the editing type of "mosaic".

[0096] In another way of dividing editing types, two editing operations that are opposite to each other can also be collectively referred to as one editing type. The name of this editing type can be represented by only the name of one of the editing operations that are opposite to each other, or the name of the editing type can reflect the two editing operations that are opposite to each other. For the case where the editing types are divided in this way, the output of this editing type by the editing type recognition model indicates that the video to be detected is obtained by the positive editing operation corresponding to this editing type, or that the video to be detected is obtained by the opposite editing operation corresponding to this editing type. For example: the two opposite editing operations of adding an icon or deleting an icon are collectively referred to as one editing type, and the name of this editing type is "add icon". The key frame of the first video and the key frame of the second video are input into the editing type recognition model. If the editing type output by the editing type recognition model is "add icon", then the editing type used for editing between the first video and the second video is "add icon". There are two possible specific relationships between the first video and the second video: the first video can be obtained by adding an icon to the second video, or the first video can be obtained by deleting an icon to the second video.

[0097] S405: Output similar videos and editing types.

[0098] After the aforementioned step S403, one or more similar videos are obtained, and after step S404, the editing type between each similar video and the video to be detected is obtained. Each similar video or the information of each similar video (for example: the name of the similar video), and the editing type between each similar video and the video to be detected can be output to the display module. The display module can be a module in the detection device, or a module of other devices or equipment other than the detection device. The display device can display each similar video or the information of each similar video and the editing type corresponding to each similar video through a visual interface or text.

[0099] Optionally, the similarity between each similar video and the video to be detected may also be output to a display module.

[0100] Optionally, the correspondence between similar shots in each similar video and corresponding shots in the video to be detected can also be output to the display module, and the display module can display the correspondence between the similar videos and the video to be detected in various forms.

[0101] Optionally, the similarity between the shot of the video to be detected and the similar shot in the corresponding similar video may also be output to the display module.

[0102] The above-mentioned similar videos or information of similar videos, the editing type between the similar video and the video to be detected, the similarity between the similar video and the video to be detected, the correspondence between similar shots in the similar video and the corresponding shots in the video to be detected, and the similarity between the shots of the video to be detected and the similar shots in the corresponding similar video are collectively referred to as relevant information corresponding to the similar videos.

[0103] It is worth noting that, with respect to the multiple similar videos obtained in the aforementioned step S403, the similar videos may be further screened based on their similarity, and only the relevant information corresponding to the screened similar videos may be output to the display module. For example, the similarity between the similar videos and the video to be detected may be compared with a pre-set screening threshold, and only the relevant information corresponding to the similar videos that are greater than or equal to the pre-set screening threshold may be output to the display module.

[0104] The display module can display the information in various forms according to the information obtained.

[0105] Figure 9 A display module provided in an embodiment of the present application displays information output by a detection device in text form. Figure 9 As shown, the text contains relevant information corresponding to the top K similar videos with the highest similarity to the video Q to be detected that are queried in the video library, including: the name of each similar video, the similarity of each similar video to video Q, a list of similar shots in each similar video, and a list of overall editing types between the similar videos and video Q. Among them, the similar shot list in each similar video contains the start and end frame sequence or time of the shot in video Q, the start and end frame sequence or time of the corresponding similar shot in the similar video, shot similarity, shot editing type and other information. The information in the similar shot list represents the correspondence between the similar shot and the shot in the corresponding video Q.

[0106] Figure 10 A display module provided in another embodiment of the present application is a schematic diagram showing information output by a detection device in the form of a visual interface. Figure 10 As shown, the visual interface displays the video Q to be detected, similar videos similar to the video Q, corresponding similar shots and their similarities, as well as the editing type corresponding to each similar shot and the similarity of each shot.

[0107] Optionally, the relevant information corresponding to the similar videos obtained by the aforementioned steps S403 and S404 can also be output to a processing module. The processing module can be a functional module in the detection device or a functional module in other devices or equipment. The processing module can further process the relevant information corresponding to the similar videos.

[0108] The editing type of the video to be detected can be detected by the method described in steps S401-S405. It should be understood that the specific implementation of each step described in the above method is only an exemplary description and does not impose any limitation on the method for video similarity detection provided in this application.

[0109] The following combination Figure 11 The specific implementation method of determining the shots and key frames according to the content of the video to be detected in the aforementioned step S401 is described as follows:

[0110] S4011: Read video frames in the video to be detected, and perform similarity comparison on the video frames in the video to be detected.

[0111] Specifically, first read the first video frame and the second video frame in the video to be detected in time sequence, and use the image hash algorithm or other similarity comparison algorithm to perform similarity comparison between the latter video frame and the former video frame of the two video frames. If the two frames are similar, continue to read a new video frame, and perform similarity comparison between the new video frame and the latter video frame of the first two video frames, until the new video frame is dissimilar to the latter video frame of the first two video frames, and then store the two dissimilar video frames in the cache.

[0112] S4012: Calculate the grayscale color histogram difference of two adjacent video frames in the buffer.

[0113] Specifically, grayscale color histograms of two adjacent video frames are calculated respectively, and the grayscale color histograms of the two video frames are correspondingly subtracted to obtain a grayscale color histogram difference of the two video frames, and the grayscale color histogram difference is stored in a cache.

[0114] S4013: Determine the relationship between the number of video frames cached in the buffer and the preset minimum lens frame number. If the number of video frames cached in the buffer is greater than the preset minimum lens frame number, execute step S4014; otherwise, execute step S4011.

[0115] S4014: Determine the shot boundary based on the maximum value and average value of all grayscale color histogram differences in the buffer area.

[0116] Specifically, the maximum value M and average value S of all grayscale color histogram differences in the buffer are calculated (where M and S are real numbers greater than 0). If M > n*S (where 0 > n >= 1), the later of the two video frames corresponding to the maximum value M of the grayscale color histogram difference is determined as the shot boundary, and step S4015 is executed. If M <= n*S (where 0 > n >= 1), step S4016 is executed. It is worth noting that the value of n can be pre-set according to different application requirements.

[0117] S4015: In the video to be detected, the frame following the previously determined shot boundary (or the first frame of the video to be detected), the currently determined shot boundary, and all frames therebetween are identified as a shot, and key frames are determined within the shot. Key frames are determined by calculating the grayscale color histogram difference between adjacent video frames within the shot, selecting video frames whose difference exceeds a pre-set difference threshold, and then filtering the selected video frames to select clear, moderately bright video frames as key frames.

[0118] It is worth noting that after step S4015 is completed, the buffer area is cleared and the above step S4011 is continued.

[0119] S4016: Calculate the gradient values of all video frames in the buffer and compare the maximum gradient value with a preset gradient threshold. If the maximum gradient value is greater than the preset threshold, the video frame corresponding to the maximum gradient value is determined as a shot boundary and step S4015 is executed. If the maximum gradient value is less than or equal to the preset gradient threshold, step S4017 is executed.

[0120] S4017: Determine the relationship between the number of video frames cached in the buffer and the preset maximum number of lens frames. If the number of video frames cached in the buffer is greater than the preset maximum number of lens frames, execute step S4015; otherwise, execute step S4011.

[0121] The above steps are executed according to their respective execution conditions until all video frames in the video to be detected are processed, and then the execution is terminated.

[0122] It is worth noting that in this application, different methods can be used to determine the shots and key frames based on the content of the video to be detected, depending on the type of the video content to be detected. For example, when the video to be detected is a lecture, variety show, or other video with the same background, the video to be detected can be segmented with a fixed length, and each fixed-length video segment after segmentation is a shot. Then, key frames are determined within each segmented shot. There are various methods for determining key frames. For example, several key frames can be selected within a shot based on a fixed video frame interval, or edge detection can be performed on each video frame within the shot, and video frames with edges that differ greatly from those of adjacent video frames are selected as key frames.

[0123] In another embodiment of the present application, the method for video similarity detection is slightly different from the method described in the aforementioned steps S401-S405. The trained edit type recognition model can be split into two parts, one part including a first feature extraction branch and a second feature extraction branch, and the other part including a predictor. The two parts of the edit type recognition model can be stored in different locations (for example, different virtual machines, different physical computing devices). The operation of extracting features from key frames and similar key frames using the first and second feature extraction branches can be completed before the edit type recognition model performs edit type recognition. For example, in the aforementioned step S403, the first feature extraction branch and the second feature extraction branch can be used to extract editing features from key frames and similar key frames respectively, and the obtained editing features of the key frames and the editing features of the similar key frames are temporarily stored in a storage module. In step S404, the editing features of the key frames and the editing features of the similar key frames in the storage module are input into the predictor, and the predictor outputs the existing editing types between the key frames and the similar key frames.

[0124] This application provides a Figure 5 The detection device 300 shown in the figure includes modules and functions as described above and will not be repeated here. In one embodiment, the structural analysis module 301 in the detection device 300 is specifically used to execute the method described in the aforementioned step S401; the feature extraction model 302 is specifically used to execute the method described in the aforementioned step S402; the comparative analysis module 303 is specifically used to execute the method described in the aforementioned step S403; the edit type recognition model 304 is specifically used to execute the method described in the aforementioned step S404; and the output module 305 is specifically used to execute the method described in the aforementioned step S405.

[0125] This application also provides a Figure 4 In the computing device 100 shown, the processor 102 in the computing device 100 reads the executable code included in the detection apparatus 300 stored in the memory 104 to execute the aforementioned video similarity detection method.

[0126] Since each module in the detection device 300 of the present application can be deployed on multiple computing devices respectively, the present application also provides a method as follows: Figure 12 The computing device system shown includes multiple computing devices 500, each of which includes a bus 501, a processor 502, a communication interface 503, and a memory 504. The processor 502, the memory 504, and the communication interface 503 communicate with each other via the bus 501.

[0127] The processor 502 may be a CPU. The memory 504 may include a volatile memory (English: volatile memory), such as a RAM. The memory 504 may also include a non-volatile memory, such as a ROM, a flash memory, a HDD or an SSD. The memory 504 stores executable code, and the processor 502 executes the executable code to perform part of the method for video similarity detection. The memory 504 may also include an operating system and other software modules required for running processes. The operating system may be LINUX TM , UNIX TM , WINDOWS TM wait.

[0128] Each computing device 500 establishes a communication path through the communication network. Each computing device 500 runs any one or more of the structure analysis module 301, feature extraction model 302, comparative analysis module 303, edit type recognition model 304, and output module 305. Any computing device 500 can be a computing device in a cloud data center, a computing device in an edge data center, or a terminal computing device.

[0129] The descriptions of the processes corresponding to the above figures have different focuses. For parts that are not described in detail in a certain process, please refer to the relevant descriptions of other processes.

[0130] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product for video similarity detection includes one or more computer instructions for video similarity detection. When these computer program instructions are loaded and executed on a computer, all or part of the embodiments of the present invention are generated. Figure 6 The process or function described.

[0131] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium is a readable storage medium storing computer program instructions for video similarity detection. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., an SSD).

Claims

1. A method for video similarity detection, characterized in that: include: receiving a first video, and determining a key frame of the first video according to the first video; Inputting the key frame into a feature extraction model to obtain features of the key frame; Determining a similar key frame and a second video based on the features of the key frame, wherein the second video is a video containing the similar key frame, and the second video is similar to the first video, and determining the similar key frame and the second video based on the features of the key frame includes: querying a video library based on the features of the key frame, obtaining the similar key frame in the video library, wherein the features of the similar key frame are similar to the features of the key frame; and determining the second video based on the similar key frame; Input the key frame and the similar key frame into the edit type recognition model to obtain the edit type, wherein the edit type indicates the edit type used for editing between the first video and the second video, and the edit type recognition model is a pre-trained neural network model. Inputting the key frame and the similar key frame into the edit type recognition model to obtain the edit type includes: performing feature extraction on the key frame to obtain the feature of the key frame, performing feature extraction on the similar key frame to obtain the feature of the similar key frame, and obtaining the edit type according to the feature of the key frame and the feature of the similar key frame.

2. The method according to claim 1, wherein The method further comprises: Outputting the second video or information about the second video to a display module, wherein the information about the second video includes a name of the second video; Outputting the editing type to the display module.

3. The method according to claim 1 or 2, wherein: The edit type recognition model includes a first feature extraction branch, a second feature extraction branch and a predictor; Inputting the key frame and the similar key frame into the edit type recognition model to obtain the edit type specifically includes: Inputting the key frame to the first feature extraction branch, and inputting the similar key frame to the second feature extraction branch; The first feature extraction branch extracts features from the key frame and outputs editing features of the key frame, and the second feature extraction branch extracts features from the similar key frame and outputs editing features of the similar key frame; The edit features of the key frame and the edit features of the similar key frames are input to the predictor, and the predictor outputs the edit type.

4. The method according to claim 1 or 2, wherein: The method further comprises: Calculating the similarity between the first video and the second video; Output the similarity to a display module.

5. The method according to claim 1 or 2, wherein: The feature extraction model and the edit type recognition model respectively adopt different neural network models.

6. The method according to claim 1 or 2, wherein: The editing type includes one or more of the following operations: Crop, stitch, rotate, mirror, blur, add text, add icons, change colors, brightness and contrast.

7. A detection device, characterized in that: include: a structure analysis module, configured to receive a first video and determine a key frame of the first video according to the first video; A feature extraction model, configured to obtain features of the key frames according to the key frames; a comparative analysis module, configured to determine a similar key frame and a second video based on the features of the key frame, wherein the second video is a video containing the similar key frame, and the second video is similar to the first video, and determining the similar key frame and the second video based on the features of the key frame comprises: querying a video library based on the features of the key frame, obtaining the similar key frame in the video library, wherein the features of the similar key frame are similar to the features of the key frame; and determining the second video based on the similar key frame; An editing type recognition model is used to obtain an editing type based on the key frames and the similar key frames, wherein the editing type indicates the editing type used for editing between the first video and the second video. The editing type recognition model is a pre-trained neural network model. Obtaining the editing type based on the key frames and the similar key frames includes: performing feature extraction on the key frames to obtain features of the key frames, performing feature extraction on the similar key frames to obtain features of the similar key frames, and obtaining the editing type based on the features of the key frames and the features of the similar key frames.

8. The device according to claim 7, wherein The detection device also includes: An output module is used to output the second video or the information of the second video to the display module, wherein the information of the second video includes the name of the second video; and is also used to output the editing type to the display module.

9. The device according to claim 7 or 8, characterized in that The edit type recognition model includes a first feature extraction branch, a second feature extraction branch and a predictor; The first feature extraction branch is used to receive the key frame, perform feature extraction on the key frame, and output the editing feature of the key frame; The second feature extraction branch is used to receive the similar key frames, perform feature extraction on the similar key frames, and output editing features of the similar key frames; The predictor is used to obtain the edit type according to the edit features of the key frame and the edit features of the similar key frames.

10. The device according to claim 7, wherein The comparison and analysis module is further configured to calculate the similarity between the first video and the second video; The detection device further includes an output module, and the output module is used to output the similarity to a display module.

11. The device according to claim 7 or 8, characterized in that The feature extraction model and the edit type recognition model respectively adopt different neural network models.

12. The device according to claim 7 or 8, characterized in that The editing type includes one or more of the following operations: Crop, stitch, rotate, mirror, blur, add text, add icons, change colors, brightness and contrast.

13. A computing device system, comprising at least one computing device, characterized in that: Each computing device includes a memory and a processor, the memory of at least one computing device being configured to store computer instructions; The processor of the at least one computing device executes the computer instructions stored in the memory to perform the method according to any one of claims 1 to 6.

14. A non-transitory readable storage medium, characterized in that When the non-transitory readable storage medium is executed by a computing device, the computing device executes the method according to any one of claims 1 to 6.

15. A computer program product, characterized in that When the computer program product is executed by a computing device, the computing device performs the method according to any one of claims 1 to 6.

Images