Video similarity determination method and device, equipment, medium and product

Abstract

The invention relates to a video similarity determination method and device, equipment, a medium and a product, and is applied to the technical field of computers. The method comprises the following steps: acquiring video data of a to-be-compared video and video data of an anchor point video; the video data of the to-be-compared video and the video data of the anchor point video are subjected to data processing, a first video feature vector of the to-be-compared video and a second video feature vector of the anchor point video are obtained, and the first video feature vector and the second video feature vector both comprise feature vectors of multiple dimensions; and performing similarity comparison based on each first video feature vector and each second video feature vector to obtain a similarity result of the to-be-compared videos. By comparing the similarity of the multi-dimensional feature vectors, multiple types of features in videos with complex contents can be covered, so that the similarity of the video contents is comprehensively analyzed, and the situation that a single feature similarity result is inaccurate when videos with complex video contents are compared is avoided. Therefore, the video similarity accuracy is improved.

Description

Technical Field

[0001] This application belongs to the field of computer technology, specifically relating to a method, apparatus, device, medium, and product for determining video similarity. Background Technology

[0002] With the booming development of internet multimedia video, video content formats such as short videos and micro-dramas are becoming increasingly diversified. The application of emerging technologies such as 5G, AI, and big data has provided support for the realization of functions such as personalized recommendations and intelligent editing.

[0003] Generally, the similarity between two videos can be compared by examining a single feature, such as color or shape similarity between images. However, this method of determining similarity relies on a limited set of features and cannot meet the needs of comparing videos with more complex content. Therefore, it often results in lower accuracy for video similarity comparisons. Summary of the Invention

[0004] This disclosure is made in view of the above-mentioned problems. This disclosure provides a method, apparatus, device, medium, and product for determining video similarity, which helps to improve the accuracy of video similarity determination.

[0005] According to one aspect of this disclosure, a video similarity determination method is provided, comprising: Obtain the video data of the video to be compared and the video data of the anchor video; The video data of the video to be compared and the video data of the anchor video are processed respectively to obtain the first video feature vector of the video to be compared and the second video feature vector of the anchor video. Both the first video feature vector and the second video feature vector include feature vectors of multiple dimensions. The similarity results of the videos to be compared are obtained by comparing the similarity of each first video feature vector and each second video feature vector.

[0006] Optionally, the video data includes at least two types of data selected from video attribute data, keyframe image data, audio data, and text data; wherein the text data includes first text data and / or second text data, the first text data being video subtitle text data, and the second text data being audio transcription text data.

[0007] Optionally, the step of processing the video data of the video to be compared and the video data of the anchor video to obtain the first video feature vector of the video to be compared and the second video feature vector of the anchor video includes: Feature extraction is performed on the video data to obtain a first video feature vector of the video to be compared and a second video feature vector of the anchor video; wherein, the feature extraction includes at least two of the following: video attribute feature extraction, image feature extraction, audio feature extraction and text feature extraction.

[0008] Optionally, the step of processing the video data of the video to be compared and the video data of the anchor video to obtain the first video feature vector of the video to be compared and the second video feature vector of the anchor video includes: Image features are extracted from the keyframe image data of the video to be compared and the keyframe image data of the anchor video, respectively, to obtain the frame image features of the video to be compared and the frame image features of the anchor video. The frame image features of the video to be compared are encoded to obtain a first image feature vector, and the first image feature vector is used as the first video feature vector. The frame image features of the anchor point video are encoded to obtain a second image feature vector, and the second image feature vector is used as the second video feature vector.

[0009] Optionally, the step of processing the video data of the video to be compared and the video data of the anchor video to obtain the first video feature vector of the video to be compared and the second video feature vector of the anchor video includes: Text features are extracted from the text data of the video to be compared and the text data of the anchor video to obtain the text tag features of the video to be compared and the text tag features of the anchor video. Based on the text tagging features of the video to be compared, a text classification vector of the video to be compared is generated, and the text classification vector of the video to be compared is used as the first video feature vector. Based on the text tagging features of the anchor video, a text classification vector of the anchor video is generated, and the text classification vector of the anchor video is used as the second video feature vector.

[0010] Optionally, the step of processing the video data of the video to be compared and the video data of the anchor video to obtain the first video feature vector of the video to be compared and the second video feature vector of the anchor video includes: Audio features are extracted from the audio data of the video to be compared and the audio data of the anchor video to obtain the frequency domain features of the video to be compared and the frequency domain features of the anchor video. The frequency domain features of the video to be compared are encoded to obtain a first frequency domain feature vector, and the first frequency domain feature vector is used as the first video feature vector. The frequency domain features of the anchor video are encoded to obtain a second frequency domain feature vector, and the second frequency domain feature vector is used as the second video feature vector.

[0011] Optionally, the step of performing similarity comparison based on each of the first video feature vectors and each of the second video feature vectors to obtain the similarity result of the videos to be compared includes: Based on the first video feature vector and the corresponding second video feature vector, a similarity comparison is performed to obtain multiple sub-similarity results; The similarity results of the multiple sub-similarity results are fused to obtain the similarity result of the video to be compared.

[0012] According to another aspect of this disclosure, a video similarity determination apparatus is provided, comprising: The acquisition module is used to acquire the video data of the video to be compared and the video data of the anchor video; The processing module is used to process the video data of the video to be compared and the video data of the anchor video respectively to obtain the first video feature vector of the video to be compared and the second video feature vector of the anchor video. The first video feature vector and the second video feature vector both include feature vectors of multiple dimensions. The comparison module is used to perform similarity comparison based on the feature vectors of each of the first videos and the feature vectors of each of the second videos to obtain the similarity result of the videos to be compared.

[0013] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the video similarity determination method described above.

[0014] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, the program being executed by a processor to implement the above-described video similarity determination method.

[0015] According to another aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above-described video similarity determination method.

[0016] In this disclosure, video data of the video to be compared and video data of the anchor video are acquired. The video data of the video to be compared and the video data of the anchor video are processed separately to obtain a first video feature vector of the video to be compared and a second video feature vector of the anchor video. Both the first and second video feature vectors include feature vectors with multiple dimensions. Similarity comparison is performed based on each of the first and second video feature vectors to obtain the similarity result of the video to be compared. By comparing the similarity of multi-dimensional feature vectors, multiple features in videos with complex content can be covered, enabling a comprehensive analysis of the similarity of video content and avoiding inaccuracies in the similarity result of a single feature when comparing videos with complex content. Therefore, the accuracy of video similarity can be improved.

[0017] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0018] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 This is a flowchart of a video similarity determination method provided in this disclosure.

[0020] Figure 2 Another flowchart is provided for a video similarity determination method disclosed herein.

[0021] Figure 3 This is a schematic diagram of the structure of the video similarity determination method provided in this disclosure.

[0022] Figure 4 This is a schematic diagram of the structure of a video similarity determination device provided in this disclosure.

[0023] Figure 5 This is a hardware block diagram of an electronic device provided in this disclosure.

[0024] Figure 6 This is a schematic diagram of a computer program product provided in this disclosure. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solution of this application, the application scenario of this application will be described first below.

[0026] With the booming development of internet multimedia video, video content formats such as short videos and micro-dramas are becoming increasingly diversified. The application of emerging technologies such as 5G, AI, and big data has provided support for the realization of functions such as personalized recommendations and intelligent editing. Video similarity comparison is widely used in video search and recommendation, video editing, and video copyright infringement detection.

[0027] Currently, similarity between two videos can be compared by examining single features. For example, visual features can be used to compare color and shape similarities; temporal features can be used to compare spectral features; and semantic information can be used to compare video similarity. However, such similarity determination methods rely on relatively singular features for comparison and cannot meet the needs of comparing videos with more complex content. Therefore, this results in lower accuracy for video similarity determination.

[0028] To address the aforementioned technical problems, this disclosure provides a method, apparatus, device, medium, and product for determining video similarity. In this disclosure, video data of the video to be compared and video data of the anchor video are acquired; the video data of the video to be compared and the video data of the anchor video are processed separately to obtain a first video feature vector of the video to be compared and a second video feature vector of the anchor video. Both the first and second video feature vectors include feature vectors with multiple dimensions; similarity comparison is performed based on each of the first and second video feature vectors to obtain the similarity result of the video to be compared. By comparing the similarity of multi-dimensional feature vectors, multiple features in videos with complex content can be covered, enabling a comprehensive analysis of the similarity of video content and avoiding inaccuracies in the similarity result of a single feature when comparing videos with complex content. Therefore, the accuracy of video similarity can be improved.

[0029] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0030] Figure 1 This is a flowchart illustrating a video similarity determination method provided in this disclosure. Figure 1 As shown, the method includes: S101: Obtain the video data of the video to be compared and the video data of the anchor video.

[0031] Specifically, the video to be compared is the video for which similarity comparison is required, and the anchor video is the video used as a reference. The files of the video to be compared and the anchor video can be imported from local storage or the cloud, respectively. The video formats can be MP4, AVI, MKV, etc. The acquired video files are then parsed to extract the video data of the video to be compared and the video data of the anchor video.

[0032] S102: Perform data processing on the video data of the video to be compared and the video data of the anchor video respectively to obtain the first video feature vector of the video to be compared and the second video feature vector of the anchor video.

[0033] Specifically, by extracting different features from the video to be compared and the anchor video, a first video feature vector and a second video feature vector can be obtained. Both the first and second video feature vectors include feature vectors with multiple dimensions. In this embodiment, the multi-dimensional feature vectors include feature vectors with at least two of the following dimensions: frequency domain feature vector, image feature vector, audio feature vector, and text classification vector. The frequency domain feature vector is used to evaluate the similarity of the basic attributes of the videos; the image feature vector is used to evaluate the similarity of the static visual information of the videos; the audio feature vector is used to evaluate the audio similarity of the videos; and the text classification vector is used to evaluate the similarity of the text content of the videos and / or the similarity of the audio-transcribed text. Comparing from multiple dimensions can improve the accuracy of video similarity comparison.

[0034] S103: Based on the feature vectors of each first video and each feature vector of each second video, perform similarity comparison to obtain the similarity results of the videos to be compared.

[0035] Specifically, the extracted first video feature vectors and second video feature vectors are compared for similarity to obtain sub-similarity results for multiple video feature vectors. These sub-similarity results are then fused to obtain a comprehensive similarity result for the videos to be compared.

[0036] In this way, we can overcome the shortcomings of single feature analysis in video similarity detection, such as the difficulty of capturing dynamic information in videos based on visual features, the high computational complexity of time-series feature-based methods, and the high data dependence of deep learning-based methods. This reduces the risk of semantic loss in the similarity comparison process and improves the accuracy of similarity results.

[0037] In this disclosure, video data of the video to be compared and video data of the anchor video are acquired. The video data of the video to be compared and the video data of the anchor video are processed separately to obtain a first video feature vector of the video to be compared and a second video feature vector of the anchor video. Both the first and second video feature vectors include feature vectors with multiple dimensions. Similarity comparison is performed based on each of the first and second video feature vectors to obtain the similarity result of the video to be compared. By comparing the similarity of multi-dimensional feature vectors, multiple features in videos with complex content can be covered, enabling a comprehensive analysis of the similarity of video content and avoiding inaccuracies in the similarity result of a single feature when comparing videos with complex content. Therefore, the accuracy of video similarity can be improved.

[0038] In one possible implementation, the video data includes at least two types of data selected from video attribute data, keyframe image data, audio data, and text data; wherein the text data includes first text data and / or second text data, the first text data being video subtitle text data, and the second text data being audio transcription text data.

[0039] Specifically, the files of the video to be compared and the anchor video are parsed separately to extract keyframe image data, text data, audio data, and video attribute data. The text data includes video subtitle text data and audio transcription text data. In this embodiment, FFmpeg can be used to extract keyframe image data frame by frame from the video file at set intervals, with a default interval of 1 second per frame. FFmpeg can also be used to separate the audio track from the video file, with the output audio format being WAV or MP3. After extracting the audio track, the audio separation tool Spleeter can be used to segment the audio into background music and vocals. The background music is the audio data, and the vocals are used for transcription text recognition. The Whisper speech recognition model is used to convert the vocals portion of the audio track into text content, i.e., the second text data. The Subtitle Extractor tool is used to extract subtitles from the video file, i.e., the first text data.

[0040] In one possible implementation, an exemplary method for processing the video data of the video to be compared and the video data of the anchor video separately to obtain a first video feature vector of the video to be compared and a second video feature vector of the anchor video includes: Feature extraction is performed on the video data to obtain the first video feature vector of the video to be compared and the second video feature vector of the anchor video.

[0041] Specifically, key features are extracted from the video data of each modality obtained from the video to be compared and the anchor video to obtain the first video feature vector of the video to be compared and the second video feature vector of the anchor video for similarity calculation. Feature extraction includes at least two methods: video attribute feature extraction, image feature extraction, audio feature extraction, and text feature extraction. Video attribute feature extraction can extract basic attribute features of the video, including at least one of the following: video duration, file size, resolution, and frame rate, used to evaluate the similarity of the overall video structure; image feature extraction can extract static features such as color, texture, and shape through visual feature analysis; audio feature extraction can analyze background music to extract features such as spectrum, pitch, and rhythm; text feature extraction can perform word segmentation and syntactic analysis on the first and second text data respectively to extract text tag features. The extracted features are then encoded to obtain video feature vectors, where the first video feature vector corresponds to the video to be compared and the second video feature vector corresponds to the anchor video.

[0042] In one possible implementation, an exemplary method for processing the video data of the video to be compared and the video data of the anchor video separately to obtain a first video feature vector of the video to be compared and a second video feature vector of the anchor video includes: Image features are extracted from the keyframe image data of the video to be compared and the keyframe image data of the anchor video, respectively, to obtain the frame image features of the video to be compared and the frame image features of the anchor video. The frame image features of the video to be compared are encoded to obtain the first image feature vector, and the first image feature vector is used as the first video feature vector. The frame image features of the anchor video are encoded to obtain the second image feature vector, and the second image feature vector is used as the second video feature vector.

[0043] Specifically, in this embodiment, a VGG16 convolutional neural network (CNN) model can be used to extract features from the extracted keyframe image data. VGG16 has 16 convolutional and fully connected layers, and its hierarchical structure can effectively extract features such as edges, textures, and colors from the image, encoding the keyframe image data into image feature vectors. These image feature vectors are used to compare the similarity between keyframe image data. The image feature vectors are then used as video feature vectors.

[0044] This process involves comparing the similarity of feature vectors between any two adjacent images, using the cosine similarity between the feature vectors to measure the similarity between adjacent images. This removes keyframe image data with minimal content differences, reducing the impact of redundant keyframe image data on subsequent video similarity comparisons. The image feature vectors corresponding to the redundant keyframe image data are then used as the final image feature vectors.

[0045] In one possible implementation, an exemplary method for processing the video data of the video to be compared and the video data of the anchor video separately to obtain a first video feature vector of the video to be compared and a second video feature vector of the anchor video includes: Text features are extracted from the text data of the video to be compared and the text data of the anchor video, respectively, to obtain the text tag features of the video to be compared and the text tag features of the anchor video. Based on the text tag features of the video to be compared, a text classification vector of the video to be compared is generated, and the text classification vector of the video to be compared is used as the first video feature vector. Based on the text tag features of the anchor video, a text classification vector of the anchor video is generated, and the text classification vector of the anchor video is used as the second video feature vector.

[0046] Specifically, in this embodiment, features can be extracted from text data using the Transformer encoding architecture of a pre-trained Sentence-BERT model. Sentences in the text data are segmented into sub-word units, and then the semantics of each sub-word unit are integrated to label the sentences. A multi-layer self-attention mechanism is used to generate a text classification vector for each label; that is, the text classification vector is a feature representation of the text data at the sentence level. This text classification vector is then used as the video feature vector.

[0047] In one possible implementation, an exemplary method for processing the video data of the video to be compared and the video data of the anchor video separately to obtain a first video feature vector of the video to be compared and a second video feature vector of the anchor video includes: Audio features are extracted from the audio data of the video to be compared and the audio data of the anchor video to obtain the frequency domain features of the video to be compared and the frequency domain features of the anchor video. The frequency domain features of the video to be compared are encoded to obtain the first frequency domain feature vector, and the first frequency domain feature vector is used as the first video feature vector. The frequency domain features of the anchor video are encoded to obtain the second frequency domain feature vector, and the second frequency domain feature vector is used as the second video feature vector.

[0048] Specifically, in this embodiment, a Fast Fourier Transform can be performed on the audio data, i.e., background music, to extract frequency domain features. Encoding these frequency domain features yields a frequency domain feature vector. For example, the frequency domain features can be determined using the following formula:

[0049] in, These are frequency domain features, representing the components of a frequency domain signal at different frequencies; It is a complex exponent used to decompose the frequency components; It is a continuous signal in the time domain; It is a time variable; For frequency; It is the imaginary unit.

[0050] In one possible implementation, an exemplary method for obtaining the similarity result of the videos to be compared by performing similarity comparison based on each first video feature vector and each second video feature vector includes: Based on the feature vectors of each first video and the corresponding feature vectors of each second video, similarity comparisons are performed to obtain multiple sub-similarity results; the multiple sub-similarity results are then fused to obtain the similarity result of the video to be compared.

[0051] Specifically, the first video feature vector and the second video feature vector each include at least two of the following feature vectors: video attribute feature vector, image feature vector, text classification vector, and frequency domain feature vector. Similarity comparisons are performed on these feature vectors to obtain the corresponding sub-similarity results.

[0052] The video attribute feature vectors of the video to be compared are compared with the video attribute feature vectors of the anchor video. The video attribute feature vectors can be represented by hash values. In this embodiment, the similarity of the video attribute features is determined by calculating the Hamming distance between the hash values, obtaining the first sub-similarity result. The smaller the Hamming distance, the higher the similarity of the video attributes.

[0053] The image feature vectors of the video to be compared are compared with the image feature vectors of the anchor video. In this embodiment, the image feature vectors can be compared frame by frame. The similarity between frames is evaluated by calculating the cosine similarity of the image feature vectors. Finally, the comparison results of each key frame are combined to determine the sub-similarity result of the image features, and the second sub-similarity result is obtained.

[0054] The text classification vectors of the video to be compared are compared with the text classification vectors of the anchor video. In this embodiment, the Manhattan distance can be used to calculate the absolute difference between the text classification vectors in each dimension. The differences of all text classification vectors are summed to obtain the total distance between the text classification vectors of the two videos. The smaller the Manhattan distance, the higher the similarity of the text data of the two videos. Performing similarity comparison separately can reduce the possibility of missing video semantics and improve the accuracy and reliability of video similarity comparison. It can be understood that the first text data and the second text data can be compared separately using the same method to obtain the third sub-similarity result and the fourth sub-similarity result.

[0055] The frequency domain feature vectors of the video to be compared are compared with those of the anchor video. Euclidean distance can be used to assess the difference in audio data between the two videos. By calculating the Euclidean distance using the frequency domain feature vectors after Fourier transform, the similarity of different audio data can be compared, yielding the fifth sub-similarity result.

[0056] After obtaining the sub-similarity results, each sub-similarity result is normalized, and then the sub-similarity results are fused together. In this embodiment, the Min-Max method can be used to normalize to the same scale range, where the scale range is [0,1]. For example, the following formula can be used for normalization to obtain the normalized sub-similarity results:

[0057] in, This represents the maximum value of each sub-similarity result. The minimum value of each sub-similarity result. For the sub-similarity results, The results are normalized sub-similarity scores.

[0058] In this embodiment, a random forest algorithm can be used to perform similarity fusion calculation on the normalized sub-similarity results. Multiple decision trees are trained to predict the similarity label between each pair of videos, i.e., the video to be compared and the anchor video. For example, a label of 1 is used for similarity, and a label of 0 is used for dissimilarity. Each decision tree independently determines whether the video to be compared and the anchor video are similar or dissimilar. Finally, the final similarity result is determined by calculating the label ratio of all decision trees.

[0059] The number of decision trees can be set to 300 by default, the maximum depth of each decision tree is 15, the minimum number of samples required for node splits is 2, and the minimum number of samples required for leaf nodes is 5. These parameter settings prevent overfitting. It is understood that the above data is only an example and can be modified according to actual needs. Furthermore, the above random forest algorithm is only an example of similarity fusion; other methods such as maximum value, average value, and hard voting mechanisms can also be used, and this disclosure does not impose specific limitations on these methods.

[0060] Figure 2 This is another flowchart of a video similarity determination method provided in this disclosure. Figure 2 As shown, the method includes: S201: Input the video to be compared and the anchor video.

[0061] Specifically, such as Figure 3 As shown, Figure 3This is a schematic diagram of the video similarity determination method provided in this disclosure. The anchor video is a reference video, while the video to be compared is the input video for similarity analysis with the anchor video.

[0062] S202: Perform video parsing on the video to be compared and the anchor video to obtain video data.

[0063] Specifically, video parsing involves video preprocessing and audio transcription text extraction. Video preprocessing includes extracting keyframes from the input video file to obtain keyframe image data; segmenting the audio to obtain audio data; extracting subtitle text from the video as the first text data; and audio transcription text extraction uses tools to transcribe the audio voice content from the video into text to obtain the second text data, which is used for subsequent feature extraction and similarity comparison.

[0064] S203: Extract features from the video data to obtain different video feature vectors for the two videos.

[0065] Specifically, different features are extracted from the video. Video attribute feature extraction can extract overall feature information such as video resolution, duration, and frame rate to obtain video attribute feature vectors, which are used to assess the basic similarity of the videos. Image feature extraction can analyze visual features such as color, texture, and shape of images to capture the static visual information of the video and obtain image feature vectors. Audio feature extraction can extract information such as spectral features, pitch, and rhythm to obtain frequency domain feature vectors, which are used to determine audio similarity. Text feature extraction can use natural language processing techniques to extract semantic and contextual information from subtitles and semantic information from audio-transcribed text, which are used to analyze the text content similarity of the videos and obtain text classification vectors.

[0066] S204: Based on different video feature vectors, perform similarity comparison to obtain similarity comparison results.

[0067] Specifically, the extracted features across various dimensions are compared. Video attribute similarity comparison compares overall attributes such as resolution, duration, and frame rate to assess basic video similarity. Image similarity comparison compares image feature vectors from two videos to assess visual content similarity. Audio similarity comparison compares frequency domain feature vectors from two videos to determine audio content similarity. Text similarity comparison compares subtitles from two videos to analyze semantic relevance and compares audio-to-text transcriptions to analyze semantic consistency of speech content. Finally, through similarity normalization and similarity fusion calculation, the similarity values ​​of each feature vector are standardized or normalized to meet the input requirements of similarity fusion on a uniform scale. A random forest algorithm is then used to weight and fuse the similarities of each feature vector, ultimately outputting a comprehensive similarity result.

[0068] Figure 4 This is a schematic diagram of the structure of a video similarity determination device provided in this disclosure. Figure 4 As shown, the device 400 includes: an acquisition module 410, a processing module 420, and a comparison module 430.

[0069] The acquisition module 410 is used to acquire the video data of the video to be compared and the video data of the anchor video; Processing module 420 is used to process the video data of the video to be compared and the video data of the anchor video respectively to obtain a first video feature vector of the video to be compared and a second video feature vector of the anchor video. The first video feature vector and the second video feature vector both include feature vectors of multiple dimensions. The comparison module 430 is used to perform similarity comparison based on each of the first video feature vectors and each of the second video feature vectors to obtain the similarity result of the video to be compared.

[0070] Optionally, the processing module is used to: Feature extraction is performed on the video data to obtain a first video feature vector of the video to be compared and a second video feature vector of the anchor video; wherein, the feature extraction includes at least two of the following: video attribute feature extraction, image feature extraction, audio feature extraction and text feature extraction.

[0071] Optionally, the processing module is used to: Image features are extracted from the keyframe image data of the video to be compared and the keyframe image data of the anchor video, respectively, to obtain the frame image features of the video to be compared and the frame image features of the anchor video. The frame image features of the video to be compared are encoded to obtain a first image feature vector, and the first image feature vector is used as the first video feature vector. The frame image features of the anchor point video are encoded to obtain a second image feature vector, and the second image feature vector is used as the second video feature vector.

[0072] Optionally, the processing module is used to: Text features are extracted from the text data of the video to be compared and the text data of the anchor video to obtain the text tag features of the video to be compared and the text tag features of the anchor video. Based on the text tagging features of the video to be compared, a text classification vector of the video to be compared is generated, and the text classification vector of the video to be compared is used as the first video feature vector. Based on the text tagging features of the anchor video, a text classification vector of the anchor video is generated, and the text classification vector of the anchor video is used as the second video feature vector.

[0073] Optionally, the processing module is used to: Audio features are extracted from the audio data of the video to be compared and the audio data of the anchor video to obtain the frequency domain features of the video to be compared and the frequency domain features of the anchor video. The frequency domain features of the video to be compared are encoded to obtain a first frequency domain feature vector, and the first frequency domain feature vector is used as the first video feature vector. The frequency domain features of the anchor video are encoded to obtain a second frequency domain feature vector, and the second frequency domain feature vector is used as the second video feature vector.

[0074] Optionally, the comparison module is used for: Based on the first video feature vector and the corresponding second video feature vector, a similarity comparison is performed to obtain multiple sub-similarity results; The similarity results of the multiple sub-similarity results are fused to obtain the similarity result of the video to be compared.

[0075] This application also provides an electronic device for performing the video similarity determination method described above. Please refer to... Figure 5 It illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 5As shown, the electronic device 50 includes: a processor 500, a memory 501, a bus 502, and a communication interface 503. The processor 500, the communication interface 503, and the memory 501 are connected via the bus 502. The memory 501 stores a computer program that can run on the processor 500. When the processor 500 runs the computer program, it executes the video similarity determination method provided in any of the foregoing embodiments of this application.

[0076] The memory 501 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface 503 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0077] Bus 502 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 501 is used to store programs. After receiving an execution instruction, the processor 500 executes the program. The video similarity determination method disclosed in any of the foregoing embodiments of this application can be applied to the processor 500, or implemented by the processor 500.

[0078] The processor 500 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 500 or by instructions in software form. The processor 500 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 501. The processor 500 reads the information in memory 501 and, in conjunction with its hardware, completes the steps of the above method.

[0079] The electronic device provided in this application embodiment and the video similarity determination method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0080] This application also provides a computer-readable storage medium corresponding to the video similarity determination method provided in the foregoing embodiments. The computer-readable storage medium shown can be an optical disc, on which a computer program is stored. When the computer program is run by a processor, it executes the video similarity determination method provided in any of the foregoing embodiments.

[0081] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0082] The computer-readable storage medium provided in the above embodiments of this application and the video similarity determination method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0083] This application also provides a computer program product 600, such as... Figure 6 As shown. This computer program product carries a computer program 601, the instructions of which can be used to execute the steps of the video similarity determination method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0084] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0085] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0086] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0087] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0088] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0089] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0090] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0091] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for determining video similarity, characterized in that, include: Obtain the video data of the video to be compared and the video data of the anchor video; The video data of the video to be compared and the video data of the anchor video are processed respectively to obtain the first video feature vector of the video to be compared and the second video feature vector of the anchor video. Both the first video feature vector and the second video feature vector include feature vectors of multiple dimensions. The similarity results of the videos to be compared are obtained by comparing the similarity of each first video feature vector and each second video feature vector.

2. The method according to claim 1, characterized in that, The video data includes at least two types of data selected from video attribute data, keyframe image data, audio data, and text data; wherein the text data includes first text data and / or second text data, the first text data being video subtitle text data, and the second text data being audio transcription text data.

3. The method according to claim 2, characterized in that, The step of processing the video data of the video to be compared and the video data of the anchor video to obtain the first video feature vector of the video to be compared and the second video feature vector of the anchor video includes: Feature extraction is performed on the video data to obtain a first video feature vector of the video to be compared and a second video feature vector of the anchor video; wherein, the feature extraction includes at least two of the following: video attribute feature extraction, image feature extraction, audio feature extraction and text feature extraction.

4. The method according to claim 3, characterized in that, The step of processing the video data of the video to be compared and the video data of the anchor video to obtain the first video feature vector of the video to be compared and the second video feature vector of the anchor video includes: Image features are extracted from the keyframe image data of the video to be compared and the keyframe image data of the anchor video, respectively, to obtain the frame image features of the video to be compared and the frame image features of the anchor video. The frame image features of the video to be compared are encoded to obtain a first image feature vector, and the first image feature vector is used as the first video feature vector. The frame image features of the anchor point video are encoded to obtain a second image feature vector, and the second image feature vector is used as the second video feature vector.

5. The method according to claim 3, characterized in that, The step of processing the video data of the video to be compared and the video data of the anchor video to obtain the first video feature vector of the video to be compared and the second video feature vector of the anchor video includes: Text features are extracted from the text data of the video to be compared and the text data of the anchor video to obtain the text tag features of the video to be compared and the text tag features of the anchor video. Based on the text tagging features of the video to be compared, a text classification vector of the video to be compared is generated, and the text classification vector of the video to be compared is used as the first video feature vector. Based on the text tagging features of the anchor video, a text classification vector of the anchor video is generated, and the text classification vector of the anchor video is used as the second video feature vector.

6. The method according to claim 3, characterized in that, The step of processing the video data of the video to be compared and the video data of the anchor video to obtain the first video feature vector of the video to be compared and the second video feature vector of the anchor video includes: Audio features are extracted from the audio data of the video to be compared and the audio data of the anchor video to obtain the frequency domain features of the video to be compared and the frequency domain features of the anchor video. The frequency domain features of the video to be compared are encoded to obtain a first frequency domain feature vector, and the first frequency domain feature vector is used as the first video feature vector. The frequency domain features of the anchor video are encoded to obtain a second frequency domain feature vector, and the second frequency domain feature vector is used as the second video feature vector.

7. The method according to claim 1, characterized in that, The step of performing similarity comparison based on the feature vectors of each of the first video and each of the feature vectors of the second video to obtain the similarity result of the video to be compared includes: Based on the first video feature vector and the corresponding second video feature vector, a similarity comparison is performed to obtain multiple sub-similarity results; The similarity results of the multiple sub-similarity results are fused to obtain the similarity result of the video to be compared.

8. A video similarity determination device, characterized in that, include: The acquisition module is used to acquire the video data of the video to be compared and the video data of the anchor video; The processing module is used to process the video data of the video to be compared and the video data of the anchor video respectively to obtain the first video feature vector of the video to be compared and the second video feature vector of the anchor video. The first video feature vector and the second video feature vector both include feature vectors of multiple dimensions. The comparison module is used to perform similarity comparison based on the feature vectors of each of the first videos and the feature vectors of each of the second videos to obtain the similarity result of the videos to be compared.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the method as described in any one of claims 1-7.

Images