Video content similarity calculation method and device, equipment, medium and product

Abstract

The invention provides a video content similarity calculation method and device, equipment, a medium and a product, and the method comprises the steps: carrying out the key frame sampling of an obtained target video and a to-be-recognized video, and obtaining a first key frame sequence of the target video and a second key frame sequence of the to-be-recognized video; converting the first key frame sequence and the second key frame sequence into text description to obtain a first semantic coding text sequence and a second semantic coding text sequence; encoding the first semantic encoding text sequence and the second semantic encoding text sequence to obtain a first semantic encoding vector and a second semantic encoding vector; and calculating the similarity between the target video and the to-be-identified video according to the first semantic coding vector and the second semantic coding vector. According to the scheme, the semantic features of the video content are extracted through semantic conversion, the semantic similarity of the video can be better captured, and calculation of the video content correlation can be more efficiently and accurately achieved.

Description

Technical Field

[0001] The present invention relates to the field of video recognition technology, and in particular to a method, device, equipment, medium and product for calculating video content similarity. Background Art

[0002] With the development of the Internet and social media, the speed and scope of video content dissemination have greatly increased, which has led to more widespread and difficult to control infringement. Therefore, the market urgently needs a technical solution that can quickly and accurately identify infringing content.

[0003] The existing solution adopts the video feature code recognition solution, which identifies the feature code of the video, searches and compares the feature codes of different videos, and determines the correlation between different videos. This solution relies heavily on the accuracy of the feature code extraction algorithm. If there are defects in the feature extraction algorithm, it may lead to misjudgment or missed judgment. In addition, the data processing volume of the extraction process is huge, and the recognition cost is high. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a video content similarity calculation method, device, equipment, medium and product, which can calculate the relevance of video content more efficiently and accurately.

[0005] An embodiment of the present invention provides a method for calculating video content similarity, the method comprising:

[0006] Perform key frame sampling on the acquired target video and the video to be identified, and obtain a first key frame sequence of the target video and a second key frame sequence of the video to be identified;

[0007] Converting the first key frame sequence and the second key frame sequence into text descriptions to obtain a first semantically coded text sequence and a second semantically coded text sequence;

[0008] Encoding the first semantically coded text sequence and the second semantically coded text sequence to obtain a first semantically coded vector and a second semantically coded vector;

[0009] The similarity between the target video and the to-be-identified video is calculated according to the first semantic coding vector and the second semantic coding vector.

[0010] Preferably, the method further comprises:

[0011] The similarity level between the target video and the video to be identified is determined according to the similarity and a preset threshold level interval.

[0012] As a preferred solution, encoding the first semantically coded text sequence and the second semantically coded text sequence to obtain a first semantically coded vector and a second semantically coded vector specifically includes:

[0013] Inputting the first semantically encoded text sequence and the second semantically encoded text sequence into a context encoder based on a Transformer layer respectively to obtain a first global semantic encoding representation and a second global semantic encoding representation;

[0014] The first global semantic coding representation and the second global semantic coding representation are hash-coded to obtain the first semantic coding vector and the second semantic coding vector.

[0015] As a preferred solution, converting the first key frame sequence and the second key frame sequence into text description to obtain a first semantically coded text sequence and a second semantically coded text sequence specifically includes:

[0016] Inputting the first key frame sequence and the second key frame sequence into an image-text reconstruction model based on an encoder-decoder structure respectively, converting the image information into corresponding text descriptions, and obtaining a first text expression sequence and a second text expression sequence;

[0017] The first text expression sequence and the second text expression sequence are semantically encoded and semantically optimized respectively to obtain the first semantically encoded text sequence and the second semantically encoded text sequence.

[0018] Furthermore, the first text expression sequence and the second text expression sequence are semantically encoded and semantically optimized respectively to obtain the first semantically encoded text sequence and the second semantically encoded text sequence, including:

[0019] Inputting the first text expression sequence and the second text expression sequence into a semantic encoder for semantic encoding respectively, converting the text expression into a vector representation, and obtaining a first semantic encoding vector sequence and a second semantic encoding vector sequence;

[0020] The first semantic coding vector sequence and the second semantic coding vector sequence are respectively input into a semantic optimizer for semantic optimization to obtain the first semantic coding text sequence and the second semantic coding text sequence.

[0021] Furthermore, the optimization model of the semantic optimizer is

[0022] Among them, v i ′ represents the optimized i-th semantic encoding vector, x i ,x jRespectively represent the i-th and j-th semantic coding vectors in the semantic coding vector sequence, Count(x j →x i ) represents x j With x i The number of semantic encoding vectors between them, f(x i ,x j ) represents x i and x j The similarity between them is , and M is the total number of feature vectors in the semantic encoding vector sequence minus 1.

[0023] Preferably, performing key frame sampling on the acquired target video and the video to be identified to acquire a first key frame sequence of the target video and a second key frame sequence of the video to be identified includes:

[0024] Acquire the video to be identified, and extract the target video from a preset target video library;

[0025] Key frame sampling is performed on the target video and the video to be identified at the same key frame sampling frequency to obtain the first key frame sequence and the second key frame sequence.

[0026] As a preferred solution, calculating the similarity between the target video and the to-be-identified video according to the first semantic coding vector and the second semantic coding vector includes:

[0027] The Euclidean distance between the first semantic encoding vector and the second semantic encoding vector is calculated as the similarity.

[0028] The embodiment of the present invention further provides a video content similarity calculation device, the device comprising:

[0029] A sampling module, used to perform key frame sampling on the acquired target video and the video to be identified, and obtain a first key frame sequence of the target video and a second key frame sequence of the video to be identified;

[0030] A text conversion module, used for converting the first key frame sequence and the second key frame sequence into text descriptions to obtain a first semantically coded text sequence and a second semantically coded text sequence;

[0031] An encoding module, used for encoding the first semantically encoded text sequence and the second semantically encoded text sequence to obtain a first semantic encoding vector and a second semantic encoding vector;

[0032] A calculation module is used to calculate the similarity between the target video and the video to be identified based on the first semantic coding vector and the second semantic coding vector.

[0033] Preferably, the device further comprises:

[0034] The level module is used to determine the similarity level between the target video and the video to be identified according to the similarity and a preset threshold level interval.

[0035] Furthermore, the encoding module is specifically used for:

[0036] Inputting the first semantically encoded text sequence and the second semantically encoded text sequence into a context encoder based on a Transformer layer respectively to obtain a first global semantic encoding representation and a second global semantic encoding representation;

[0037] The first global semantic coding representation and the second global semantic coding representation are hash-coded to obtain the first semantic coding vector and the second semantic coding vector.

[0038] Preferably, the text conversion module is specifically used for:

[0039] Inputting the first key frame sequence and the second key frame sequence into an image-text reconstruction model based on an encoder-decoder structure respectively, converting the image information into corresponding text descriptions, and obtaining a first text expression sequence and a second text expression sequence;

[0040] The first text expression sequence and the second text expression sequence are semantically encoded and semantically optimized respectively to obtain the first semantically encoded text sequence and the second semantically encoded text sequence.

[0041] Furthermore, the text conversion module is also used for:

[0042] Inputting the first text expression sequence and the second text expression sequence into a semantic encoder for semantic encoding respectively, converting the text expression into a vector representation, and obtaining a first semantic encoding vector sequence and a second semantic encoding vector sequence;

[0043] The first semantic coding vector sequence and the second semantic coding vector sequence are respectively input into a semantic optimizer for semantic optimization to obtain the first semantic coding text sequence and the second semantic coding text sequence.

[0044] Furthermore, the optimization model of the semantic optimizer is

[0045] Among them, v i ′ represents the optimized i-th semantic encoding vector, x i ,x j Respectively represent the i-th and j-th semantic coding vectors in the semantic coding vector sequence, Count(x j→x i ) represents x j With x i The number of semantic encoding vectors between them, f(x i ,x j ) represents x i and x j The similarity between them is , and M is the total number of feature vectors in the semantic encoding vector sequence minus 1.

[0046] Preferably, the sampling module is specifically used for:

[0047] Acquire the video to be identified, and extract the target video from a preset target video library;

[0048] Key frame sampling is performed on the target video and the video to be identified at the same key frame sampling frequency to obtain the first key frame sequence and the second key frame sequence.

[0049] Preferably, the calculation module is used for:

[0050] The Euclidean distance between the first semantic encoding vector and the second semantic encoding vector is calculated as the similarity.

[0051] An embodiment of the present invention further provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for calculating the similarity of video content as described in any of the above embodiments is implemented.

[0052] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the video content similarity calculation method as described in any of the above embodiments.

[0053] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of any of the above methods when executed by a processor.

[0054] The present invention provides a method, device, equipment, medium and product for calculating video content similarity, which obtains a first key frame sequence of the target video and a second key frame sequence of the video to be identified by performing key frame sampling on an acquired target video and a video to be identified; converts the first key frame sequence and the second key frame sequence into text descriptions to obtain a first semantically coded text sequence and a second semantically coded text sequence; encodes the first semantically coded text sequence and the second semantically coded text sequence to obtain a first semantic coding vector and a second semantic coding vector; and calculates the similarity between the target video and the video to be identified based on the first semantic coding vector and the second semantic coding vector. The present application scheme extracts the semantic features of the video content through semantic conversion, which can better capture the semantic similarity of the video and can more efficiently and accurately calculate the relevance of the video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of a method for calculating video content similarity provided by an embodiment of the present invention;

[0056] Figure 2 It is a structural schematic diagram of a video content similarity calculation device provided by an embodiment of the present invention;

[0057] Figure 3 It is a structural diagram of a terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The existing scheme for video related identification is to judge whether the video to be identified is related to the target video according to the feature code library, wherein, when the number of feature codes of the video to be identified that hits the feature code library reaches a preset number, the method of determining that the video to be identified is related to the target video is to identify the feature code of the video and search and compare it with the feature code of the target video. When the number of feature codes of the video to be identified and the feature code of the target video that are the same reaches a certain preset number, it is determined that the video to be identified is related to the target video, thereby achieving the purpose of automatically identifying and determining whether the video to be identified is related to the target video, thereby achieving the technical effect of improving the efficiency of identifying videos, and then solving the technical problem of low efficiency in the method of manually identifying whether a video infringes on a film and television work in the related technology. However, many technical difficulties were found in the specific implementation of the above technical scheme, including: the extraction of feature codes requires a high degree of accuracy to ensure that the video content can be correctly identified and compared. If there are defects in the feature extraction algorithm, it may lead to misjudgment or missed judgment. It is a challenge to build a comprehensive and discriminative feature code library. It is necessary to ensure that the feature codes in the library can cover various possible infringing videos while avoiding the situation of accidentally damaging legitimate use. If video is used as comparison material, the accuracy of the comparison can be improved, but the amount of data processing will be too large and the dependence on computing resources will be high, affecting the comparison efficiency and cost.

[0060] In response to the above technical problems, this case provides a method for calculating video content similarity, see Figure 1 , is a flow chart of a method for calculating video content similarity provided by an embodiment of the present invention, the method comprising:

[0061] Step S1, performing key frame sampling on the acquired target video and the video to be identified, and obtaining a first key frame sequence of the target video and a second key frame sequence of the video to be identified;

[0062] Step S2, converting the first key frame sequence and the second key frame sequence into text descriptions to obtain a first semantically coded text sequence and a second semantically coded text sequence;

[0063] Step S3, encoding the first semantically coded text sequence and the second semantically coded text sequence to obtain a first semantically coded vector and a second semantically coded vector;

[0064] Step S4: calculating the similarity between the target video and the video to be identified according to the first semantic coding vector and the second semantic coding vector.

[0065] During the specific implementation of this embodiment, a target video and a video to be identified are obtained, and the correlation between the video to be identified and the target video is compared.

[0066] Video is composed of a large number of continuous image frames. If the entire video is processed and compared directly, the amount of calculation will be very large and the efficiency will be very low. Accordingly, by adopting the key frame sampling method, representative key frames can be extracted from the video, greatly reducing the amount of data for subsequent processing. Therefore, it is first necessary to extract key frames from the target video and the video to be identified, and obtain the first key frame sequence of the target video and the second key frame sequence of the video to be identified.

[0067] Considering that if the time sequence of the key frames of the target video and the time sequence of the key frames of the video to be identified are directly used as the comparison material, the amount of data to be compared is too large. Therefore, the present application performs textual expression on the target video and the video to be identified, converts the first key frame sequence and the second key frame sequence into textual descriptions, and obtains a first semantically coded text sequence and a second semantically coded text sequence.

[0068] The obtained text representation uses high-dimensional semantic coding, which will be very time-consuming and computationally resource-intensive. Therefore, in the specific implementation of this application, the first semantically coded text sequence and the second semantically coded text sequence are encoded to obtain a first semantic coding vector and a second semantic coding vector, which greatly reduces the storage space requirements and also reduces the subsequent calculation overhead.

[0069] After obtaining the semantic coding vector, calculations are performed based on the obtained semantic coding vector to determine the similarity between the target video and the video to be identified, thereby achieving an assessment of the similarity and plagiarism risk between different videos.

[0070] Extracting the semantic features of video content through semantic transformation can better capture the semantic similarity of videos, thereby improving the accuracy of infringement identification.

[0071] In another embodiment provided in this case, the method further includes:

[0072] Based on a comparison between the similarity and a predetermined threshold, a similarity level of the first video to be identified is determined.

[0073] Specifically, different threshold level intervals are set for different similarity levels, and a corresponding relationship between similarity levels and similarities is constructed. After obtaining the similarity, the similarity is compared with the threshold level interval to determine the interval where the similarity lies, and then the similarity level between the target video and the video to be identified is determined, thereby determining the infringement risk level of the video to be identified. By determining the infringement risk level through similarity calculation and threshold judgment, automated infringement identification and risk assessment can be achieved, greatly improving work efficiency.

[0074] In another embodiment provided by the present invention, the step S3 specifically includes the following steps:

[0075] After the text description is converted, the semantic encoding vector of each key frame is obtained. Therefore, a global semantic representation is needed to better perform semantic matching between videos.

[0076] The first semantically encoded text sequence of the target video and the second semantically encoded text sequence of the video to be identified are respectively input into the context encoder based on the Transformer layer to obtain the global text semantic encoding representation vector of the target video and the global text semantic encoding representation vector of the video to be identified, namely the first global semantic encoding representation and the second global semantic encoding representation.

[0077] By performing global semantic encoding on the first semantically encoded text sequence and the second semantically encoded text sequence, the global semantic features of the text transformation of the entire video are captured, rather than just the text semantic features of a single key frame.

[0078] It should be noted that the global semantic encoder in this case is a context encoder using the Transformer layer. Transformer is a very powerful sequence modeling architecture that can effectively model the contextual relationship between sequence data. By inputting the optimized semantic encoding vector sequence into the context encoder of the Transformer layer, the global semantic encoding representation of the entire video can be obtained.

[0079] When the context encoder works specifically, the input is a mask image (that is, an image with a center missing). The encoder is input to obtain the encoded features, and then a channel fully connected layer is placed between the encoded features and the decoded features to obtain better semantic features. Finally, the decoder uses the features to reconstruct the missing parts. The encoder uses the AlexNet structure and trains the network from scratch with randomly initialized weights. Compared with the original AlexNet architecture and the autoencoder, the main difference is the channel fully connected layer in the middle. If there are only convolutional layers in the network, the features of spatial positions far away on the feature map cannot be utilized. To solve this problem, a fully connected layer can be used, that is, the value of each neuron in the current layer depends on the values ​​of all neurons in the previous layer. However, the fully connected layer will introduce many parameters, 8192x8192=67.1M, which is difficult to train even on a GPU, so a channel fully connected layer can be used to solve this problem. The channel fully connected layer connects each channel completely independently instead of all channels. For example, there are m feature maps of size n*n. If a standard fully connected layer is used, there will be m 2 n parameters, and for the channel-level fully connected layer, there are only mn parameters. Therefore, features can be captured at distant spatial locations without adding so many extra parameters. The decoder is just the reverse of the encoding process. A series of transposed convolutions can be used to obtain a reconstructed image of the desired size.

[0080] After the global semantic code is obtained, the first global semantic code representation and the second global semantic code representation are hash-coded to obtain a first semantic code vector and a second semantic code vector.

[0081] Hash coding can compress high-dimensional semantic coding representations into a more compact hash coding form. This compression can greatly improve the efficiency of subsequent semantic matching, while also reducing storage and computing overhead. It should be understood that directly comparing high-dimensional semantic coding representations is very time-consuming and computationally resource-intensive. By compressing them into hash codes, similarity matching can be quickly performed using technologies such as hash indexing, greatly improving matching efficiency.

[0082] It should be noted that high-dimensional semantic coding representation requires a large amount of storage space, and after compressing it into a hash code, the storage space requirement can be greatly reduced, and the subsequent calculation overhead is also reduced.

[0083] Using a Transformer-based context encoder to extract the global semantic representation of the video can better reflect the overall semantic characteristics of the video, thereby further improving the recognition performance.

[0084] In another embodiment provided by the present invention, the step S2 specifically includes the following steps:

[0085] Considering that the amount of data to be compared is too large if the time series of the first target video key frames and the time series of the first to-be-identified video key frames are directly used as comparison materials, this solution uses text descriptions to represent the image semantics of the key frames during specific implementation.

[0086] The first key frame sequence and the second key frame sequence are respectively converted into corresponding text descriptions through an image-text reconstruction model based on an encoder-decoder structure to obtain a first text expression sequence of the target video and a second text expression sequence of the video to be identified.

[0087] In the technical solution of the present application, an encoder-decoder based image-text reconstruction module is used to complete this conversion.

[0088] The encoder-decoder model is a commonly used deep learning architecture that can effectively convert image information into corresponding text descriptions. The encoder and decoder layers are the two main components that make up the sequence-to-sequence (Seq2Seq) model, which is widely used in natural language processing (NLP) tasks such as machine translation, text summarization, question-answering systems, etc. The design of these layers enables the model to process input sequences and produce corresponding output sequences, that is, the conversion from one domain (such as source language text) to another domain (such as target language text).

[0089] In a specific example of the present application, the encoder is a convolutional neural network model, and the decoder is a RNN model.

[0090] It should be noted that the image-text reconstruction model relies on constructing multimodal semantic association information between text and image embeddings. The core of the reconstruction model is also Transformer, which uses the attention mechanism to learn and understand multimodal context understanding. The pre-training model of the reconstruction model consists of four pre-training tasks: conditional masked language modeling (MLM) on images; masked region modeling (MRM) on text; image-text matching (ITM); and text region alignment (WAR).

[0091] The image-text reconstruction model has independent embedding processing for images and texts. First, the regional image is encoded. The work here is two-fold: the visual features and bounding box features are each embedded into a common embedding space using the corresponding embedder. Here, a special modal embedding is also used in the embedder processing of the image and text. This operation is completed before passing through the respective LN layers and is not shown in the model diagram above. The purpose of this is to help the model better distinguish the difference between the input content (image, text). This layer is also similar to the "segment embedding" layer in BERT. Finally, the information of each image region and text is passed through the Transformer module to complete the learning of the contextual content. During the pre-training process, the learning task based on the optimized transmitter (OT) can achieve the performance of optimized distribution matching by minimizing the loss of transferring the original data distribution to other data distribution types.

[0092] After completing the reconstruction of the image text, the image information is converted into a corresponding text description to obtain a first text expression sequence and a second text expression sequence;

[0093] After the text description is completed, the text expression itself contains a large amount of vocabulary and grammatical information, and the direct comparison effect may not be good. Therefore, in this case, the first text expression sequence and the second text expression sequence are semantically encoded and semantically optimized respectively.

[0094] Textual expressions themselves contain a lot of vocabulary and grammatical information, so direct comparison may not be effective. Through semantic encoding, more concise and semantic feature representations can be extracted. Semantic optimization can further improve the quality and reliability of semantic encoding vectors, providing a more accurate basis for subsequent semantic matching.

[0095] Through semantic coding and semantic optimization, a first semantically coded text sequence and a second semantically coded text sequence are obtained. By extracting semantic features of video key frames and performing global semantic matching, more accurate and robust video infringement identification can be achieved.

[0096] In another embodiment provided by the present invention, the process of semantic encoding and semantic optimization specifically includes:

[0097] Inputting the first text expression sequence and the second text expression sequence into a semantic encoder respectively to obtain a sequence of semantic coding vectors of key frame text expression of the first target video and a sequence of semantic coding vectors of key frame text expression of the video to be identified, i.e., a first semantic coding vector sequence and a second semantic coding vector sequence;

[0098] The first semantic coding vector sequence and the second semantic coding vector sequence are input into a semantic optimizer that integrates semantic relevance and semantic span to obtain an optimized sequence of semantic coding vectors of target video key frame text expressions and an optimized sequence of semantic coding vectors of video key frame text expressions to be identified, namely, the first semantic coding text sequence and the second semantic coding text sequence.

[0099] It should be noted that the semantic encoder can be generated based on a neural network. The semantic information extracted from the input raw data is extracted through a semantic extraction neural network, that is, part of the feature information of the raw data. Subsequently, the extracted semantic information is respectively passed through source coding and channel coding. Use a semantic source encoder and decoder for source coding. The semantic extraction, source coding and channel coding are jointly designed using a neural network to realize an end-to-end system framework. The input of the semantic encoder neural network includes the transmitter signal and other information. The transmitter protects the transmitted signal by retaining a part of the redundant information in the signal. During the training phase, the receiver introduces a feedback mechanism to feed back the influence of the channel on the signal to the receiver, thereby realizing the joint coding of the source channel.

[0100] It should be noted that the semantic optimizer adopts the algorithm of minimizing the loss function. In the deep learning supervised learning model, the gradient descent method is often used to solve the optimal parameters, smooth the current gradient according to the local historical gradient, and optimize the obtained semantic encoding vector.

[0101] Through semantic coding, text expressions can be converted into more abstract and semantic vector representations, so that more accurate semantic comparisons can be performed later. It should be understood that text expressions themselves contain a lot of vocabulary and grammatical information, and direct comparisons may not be effective. Through semantic coding, more concise and semantic feature representations can be extracted.

[0102] Simple semantic coding cannot fully capture the semantic relevance and span information between key frames. Therefore, in the technical solution of the present application, semantic optimization is used to further improve the quality and reliability of the semantic coding vector, providing a more accurate basis for subsequent semantic matching.

[0103] In another embodiment provided by the present invention, the optimization model of the semantic optimizer is

[0104] The semantic optimizer integrating semantic relevance and semantic span uses the above model to process each semantic coding vector in the first semantic coding vector sequence and the second semantic coding vector sequence to obtain the first semantic coding text sequence and the second semantic coding text sequence.

[0105] Among them, v i ′ represents the optimized i-th semantic encoding vector, x i ,x j Respectively represent the i-th and j-th semantic coding vectors in the semantic coding vector sequence, Count(x j →x i ) represents x j With x i The number of semantic encoding vectors between them, f(x i ,x j ) represents x i and x j The similarity between them is , and M is the total number of feature vectors in the semantic encoding vector sequence minus 1.

[0106] It should be noted that similarity refers to the degree of semantic connection between adjacent key frames, expressed as f(v i ,v j ) is used to represent, for example, in a specific example of the present application, the f(v i ,v j ) is the cosine similarity between two semantic encoding vectors. The semantic span refers to the semantic distance between key frames, expressed as Count(x j →x i ) is used to represent, in a specific example of this application, Count(x j →x i ) represents x j With x i The number of semantic encoding vectors between them.

[0107] In another embodiment provided by the present invention, the step S1 specifically includes the following steps:

[0108] Get the video to be identified and extract the target video from the target video library.

[0109] First, a target video is extracted from an existing target video library. The target video library is the basic knowledge base of the system, which contains some known copyrighted video content. The video to be identified is the video that needs to be judged whether it is infringing.

[0110] The key frames of the target video and the video to be identified are sampled at the same key frame sampling frequency to obtain a time sequence of the key frames of the target video and a time sequence of the key frames of the video to be identified.

[0111] Video is composed of a large number of continuous image frames. If the entire video is processed and compared directly, the amount of calculation will be very large and the efficiency will be very low. Accordingly, by adopting the key frame sampling method, representative key frames can be extracted from the video, greatly reducing the amount of data for subsequent processing. At the same time, in order to ensure the accuracy of subsequent video-semantic conversion and semantic comparison, it is necessary to use the same key frame sampling frequency for the target video and the video to be identified, that is, to keep the key frame sampling frequency of the target video and the video to be identified consistent, so that the two videos can be guaranteed to correspond in time series, which is conducive to the subsequent semantic feature extraction and comparison.

[0112] In another embodiment provided by the present invention, the step S3 specifically includes:

[0113] When calculating the similarity, the Euclidean distance between the first semantic encoding vector and the second semantic encoding vector is used as the similarity.

[0114] The Euclidean distance between the global text semantic coding representation vector of the hash-coded first target video and the global text semantic coding representation vector of the hash-coded first video to be identified is calculated as the similarity between the first target video and the first video to be identified.

[0115] Euclidean distance is a commonly used vector similarity measurement method that can reflect the similarity between two vectors. By calculating the Euclidean distance of the global semantic encoding representation vector after hash encoding, the similarity between the target video and the video to be identified can be obtained.

[0116] It should be noted that the smaller the Euclidean distance is, the more similar the two vectors are, which means that the content of the target video and the video to be identified is more similar, that is, the similarity is higher. Therefore, the Euclidean distance can be directly used as an indicator to represent the similarity between the two videos. Therefore, the calculated Euclidean distance can be used as a basis for judging whether the target video and the video to be identified are infringing.

[0117] See also Figure 2 , is a schematic diagram of the structure of a video content similarity calculation device provided by an embodiment of the present invention, the device comprising:

[0118] A sampling module, used to perform key frame sampling on the acquired target video and the video to be identified, and obtain a first key frame sequence of the target video and a second key frame sequence of the video to be identified;

[0119] A text conversion module, used for converting the first key frame sequence and the second key frame sequence into text descriptions to obtain a first semantically coded text sequence and a second semantically coded text sequence;

[0120] An encoding module, used for encoding the first semantically encoded text sequence and the second semantically encoded text sequence to obtain a first semantic encoding vector and a second semantic encoding vector;

[0121] A calculation module is used to calculate the similarity between the target video and the video to be identified based on the first semantic coding vector and the second semantic coding vector.

[0122] The video content similarity calculation device provided in this embodiment can execute all the steps and functions executed in the video content similarity calculation method provided in any of the above embodiments, and the specific functions of the device will not be described in detail herein.

[0123] See also Figure 3 , is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention. The terminal device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a video content similarity calculation program. When the processor executes the computer program, the steps in each of the above-mentioned video content similarity calculation method embodiments are implemented, such as Figure 1 Alternatively, the processor implements the functions of each module in the above-mentioned device embodiments when executing the computer program.

[0124] Exemplarily, the computer program may be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the video content similarity calculation device. For example, the computer program may be divided into several modules, and the specific functions of each module have been described in detail in a video content similarity calculation method provided in any of the above embodiments, and the specific functions of the device will not be repeated here.

[0125] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The terminal device may include, but is not limited to, a processor and a memory. A person skilled in the art may understand that the schematic diagram is only an example of a terminal device and does not constitute a limitation on a video content similarity calculation device. The terminal device may include more or fewer components than shown in the diagram, or may combine certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0126] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the video content similarity calculation device, and uses various interfaces and lines to connect the various parts of the entire video content similarity calculation device.

[0127] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the video content similarity calculation device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0128] Wherein, if the module integrated in the video content similarity calculation device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0129] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of implementing the method by the functional network element in the above embodiment.

[0130] The computer program product provided in this embodiment can execute all the steps and functions of the video content similarity calculation method provided in any of the above embodiments, and the specific functions of the product will not be described in detail here.

[0131] It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the protection scope of the present invention.

Claims

1. A method for calculating video content similarity, characterized in that: The method comprises: Perform key frame sampling on the acquired target video and the video to be identified, and obtain a first key frame sequence of the target video and a second key frame sequence of the video to be identified; Converting the first key frame sequence and the second key frame sequence into text descriptions to obtain a first semantically coded text sequence and a second semantically coded text sequence; Encoding the first semantically coded text sequence and the second semantically coded text sequence to obtain a first semantically coded vector and a second semantically coded vector; The similarity between the target video and the to-be-identified video is calculated according to the first semantic coding vector and the second semantic coding vector.

2. The video content similarity calculation method according to claim 1, characterized in that: The method further comprises: The similarity level between the target video and the video to be identified is determined according to the similarity and a preset threshold level interval.

3. The video content similarity calculation method according to claim 1, characterized in that: Encoding the first semantically coded text sequence and the second semantically coded text sequence to obtain a first semantically coded vector and a second semantically coded vector specifically includes: Inputting the first semantically encoded text sequence and the second semantically encoded text sequence into a context encoder based on a Transformer layer respectively to obtain a first global semantic encoding representation and a second global semantic encoding representation; The first global semantic coding representation and the second global semantic coding representation are hash-coded to obtain the first semantic coding vector and the second semantic coding vector.

4. The video content similarity calculation method according to claim 1, characterized in that: Converting the first key frame sequence and the second key frame sequence into text descriptions to obtain a first semantically coded text sequence and a second semantically coded text sequence specifically includes: Inputting the first key frame sequence and the second key frame sequence into an image-text reconstruction model based on an encoder-decoder structure respectively, converting the image information into corresponding text descriptions, and obtaining a first text expression sequence and a second text expression sequence; The first text expression sequence and the second text expression sequence are semantically encoded and semantically optimized respectively to obtain the first semantically encoded text sequence and the second semantically encoded text sequence.

5. The video content similarity calculation method according to claim 4, characterized in that: The first text expression sequence and the second text expression sequence are semantically encoded and semantically optimized respectively to obtain the first semantically encoded text sequence and the second semantically encoded text sequence, including: Inputting the first text expression sequence and the second text expression sequence into a semantic encoder for semantic encoding respectively, converting the text expression into a vector representation, and obtaining a first semantic encoding vector sequence and a second semantic encoding vector sequence; The first semantic coding vector sequence and the second semantic coding vector sequence are respectively input into a semantic optimizer for semantic optimization to obtain the first semantic coding text sequence and the second semantic coding text sequence.

6. The video content similarity calculation method according to claim 5, characterized in that: The optimization model of the semantic optimizer is: Among them, v i ′ represents the optimized i-th semantic encoding vector, x i ,x j Respectively represent the i-th and j-th semantic coding vectors in the semantic coding vector sequence, Count(x j →x i 0 represents x j With x i The number of semantic encoding vectors between them, f(x i ,x j ) represents x i and x j The similarity between them is , and M is the total number of feature vectors in the semantic encoding vector sequence minus 1.

7. The video content similarity calculation method according to claim 1, characterized in that: Key frame sampling is performed on the acquired target video and the video to be identified to obtain a first key frame sequence of the target video and a second key frame sequence of the video to be identified, including: Acquire the video to be identified, and extract the target video from a preset target video library; Key frame sampling is performed on the target video and the video to be identified at the same key frame sampling frequency to obtain the first key frame sequence and the second key frame sequence.

8. The video content similarity calculation method according to claim 1, characterized in that: Calculating the similarity between the target video and the to-be-identified video according to the first semantic coding vector and the second semantic coding vector includes: The Euclidean distance between the first semantic encoding vector and the second semantic encoding vector is calculated as the similarity.

9. A video content similarity calculation device, characterized in that: The device comprises: A sampling module, used to perform key frame sampling on the acquired target video and the video to be identified, and obtain a first key frame sequence of the target video and a second key frame sequence of the video to be identified; A text conversion module, used for converting the first key frame sequence and the second key frame sequence into text descriptions to obtain a first semantically coded text sequence and a second semantically coded text sequence; An encoding module, used for encoding the first semantically encoded text sequence and the second semantically encoded text sequence to obtain a first semantic encoding vector and a second semantic encoding vector; A calculation module is used to calculate the similarity between the target video and the video to be identified based on the first semantic coding vector and the second semantic coding vector.

10. A terminal device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the video content similarity calculation method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the video content similarity calculation method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images