Video clip content similarity evaluation method and system, electronic equipment and medium

Abstract

The invention discloses a video clip content similarity assessment method and system, electronic equipment and a medium, and the method comprises the steps: obtaining a to-be-assessed video, carrying out the preprocessing of the to-be-assessed video, and obtaining the preprocessed video data; according to the preprocessed video data, performing video segmentation on the to-be-evaluated video to obtain a plurality of short video clips; extracting a key frame in each short video clip to obtain a key frame corresponding to each short video clip; constructing a nearest neighbor graph corresponding to each key frame in each short video clip according to the key frame corresponding to each short video clip; constructing a bipartite graph corresponding to each short video clip according to the nearest neighbor graph corresponding to each key frame in each short video clip; calculating the similarity between the two short video clips based on the bipartite graph; and evaluating the similarity of the two short video clips according to the similarity. According to the method, the accuracy of evaluating the similarity of the video clip contents can be improved, and the interpretability of the whole method is improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method, system, electronic device and medium for evaluating the similarity of video clip content. Background Art

[0002] Existing related research methods are limited to a single keyframe. Frame-level feature methods often extract image features based on a single frame and can only reflect the spatial characteristics of static scenes. They rely heavily on the visual features of the image and are easily affected by image quality (such as clarity, brightness, background noise, etc.). They ignore the temporal relationship between frames in the video and cannot capture motion trends or behavioral characteristics in dynamic scenes. The essence of video is dynamic changes in a continuous temporal sequence, and frame-level feature methods have no direct modeling capabilities for these changes.

[0003] In addition, existing related research methods require a large amount of training data. The spatiotemporal feature method usually relies on the training of deep learning models, which requires a large amount of labeled data to ensure the generalization ability of the model. The computing resource requirements are high. Deep learning models (such as 3D-CNN) require a lot of computing resources and memory, and may face performance bottlenecks in real-time processing and large-scale data processing. The spatiotemporal feature method has high requirements on the quality of the video (such as resolution and frame rate), and low-quality videos may lead to inaccurate feature extraction. Although it can capture the spatial and temporal features of the video at the same time, it is completely left to the model to learn the features of the video content, and the overall interpretability is poor.

[0004] Therefore, the existing related research methods are not accurate enough in evaluating the similarity of video clip content, and the overall method has poor interpretability. Summary of the invention

[0005] The present application aims to propose a method, system, electronic device and medium for evaluating the similarity of video clip content, which can improve the accuracy of evaluating the similarity of video clip content and improve the interpretability of the overall method.

[0006] In a first aspect, an embodiment of the present application provides a method for evaluating content similarity of video clips, the method comprising: Acquire a video to be evaluated, and preprocess the video to be evaluated to obtain preprocessed video data; According to the preprocessed video data, the video to be evaluated is segmented to obtain multiple short video clips; Extracting key frames from each short video clip to obtain key frames corresponding to each short video clip, wherein each short video clip corresponds to at least one key frame; According to the key frames corresponding to each short video segment, construct a nearest neighbor graph corresponding to each key frame in each short video segment; Constructing a bipartite graph corresponding to each short video segment according to the nearest neighbor graph corresponding to each key frame in each short video segment; Based on the bipartite graph, calculating the similarity between the two short video clips; The similarity between the two short video clips is evaluated according to the similarity.

[0007] Compared with the prior art, the first aspect of the present application has the following beneficial effects: The method obtains the video to be evaluated and preprocesses the video to be evaluated to obtain the preprocessed video data; according to the preprocessed video data, the video to be evaluated is segmented to obtain multiple short video clips; the key frames in each short video clip are extracted to obtain the key frames corresponding to each short video clip, wherein each short video clip corresponds to at least one key frame; according to the key frames corresponding to each short video clip, the nearest neighbor graph corresponding to each key frame in each short video clip is constructed; according to the nearest neighbor graph corresponding to each key frame in each short video clip, the bipartite graph corresponding to each short video clip is constructed; based on the bipartite graph, the similarity between two short video clips is calculated; according to the similarity, the similarity of the two short video clips is evaluated. In this way, the bipartite graph corresponding to each short video clip is constructed through a series of step methods. Since the bipartite graph can clearly represent the relationship and change between objects in the video clip, the bipartite graph modeling effectively captures the structural change information of the video clip. Therefore, based on the bipartite graph, the similarity between two short video clips is calculated, which can improve the accuracy of evaluating the similarity of the content of the video clips. And by analyzing the changes of nodes and edges in the bipartite graph, the appearance, disappearance and changes in the relationship between objects can be directly observed. This structured representation method is easier to understand and interpret than existing related technologies, improving the interpretability of the overall method.

[0008] In some implementations, obtaining a video to be evaluated and preprocessing the video to be evaluated to obtain preprocessed video data includes: Converting the format of the video to be evaluated to obtain a video after the format conversion; Converting the speech in the video after the format conversion into speech text; Extracting visual text from the video converted into the format; Performing data processing of the text on the speech text and the visual text to obtain processed speech text and processed visual text; The processed speech text and the processed visual text are analyzed by natural language processing to obtain preprocessed video data.

[0009] In some implementations, the video to be evaluated is segmented according to the preprocessed video data to obtain a plurality of short video clips, including: Determining the time segmentation points of the video to be evaluated based on the preprocessed video data; Calculating the pixel difference between two adjacent time division points, and determining the scene change point and the target time division point according to the pixel difference; The video to be evaluated is segmented according to the target time segmentation point and the scene change point to obtain a plurality of short video clips.

[0010] In some implementations, extracting the key frame in each short video segment to obtain the key frame corresponding to each short video segment includes: Performing target detection on each frame in each short video clip to obtain multiple objects, and labeling each of the objects; Calculating the geometric center point of each of the objects to obtain the position coordinates of the object, and using the geometric center point to represent the object; Calculating a relative position change value between two objects according to the position coordinates of the objects; Compare all objects in the next frame with all objects in the previous frame. If any of the following conditions exists: a new label object appears, an existing label object disappears, or the relative position change between two objects exceeds a preset threshold, the next frame is taken as a key frame. If any of the following conditions is met in the short video clip: no new label object appears, an existing label object disappears, or the relative position change between two objects exceeds a preset threshold, the first and last two frames of the short video clip are taken as key frames to extract the key frames corresponding to each short video clip.

[0011] In some implementations, constructing a nearest neighbor graph corresponding to each key frame in each short video segment according to the key frame corresponding to each short video segment includes: Obtain all objects in each key frame, and use the geometric center point of each object as a node of the graph; Calculate the nearest node of each object according to the geometric center point of each object; An undirected edge is constructed between each of the objects and the nearest node corresponding thereto, so as to construct a nearest neighbor graph corresponding to each key frame in each of the short video clips.

[0012] In some implementations, constructing a bipartite graph corresponding to each short video segment according to a nearest neighbor graph corresponding to each key frame in each short video segment includes: Representing all objects in the nearest neighbor graph corresponding to each key frame by using a set to obtain a set corresponding to each key frame; According to the set corresponding to each key frame, construct the edge type between each object in two key frames in each short video clip, wherein the edge type includes the edge where the object itself moves, the edge where the neighboring object moves, and the edge where the object appears or disappears; Constructing a bipartite graph representation between two key frames according to the set corresponding to each key frame and the edge type; According to the bipartite graph representation, a bipartite graph corresponding to each short video clip is constructed, and the bipartite graph includes a degree distribution of nodes.

[0013] In some implementations, calculating the similarity between the two short video clips based on the bipartite graph includes: Calculating the object distribution similarity between the two first key frames based on the first key frames respectively corresponding to the two short video clips, wherein the first key frame is the key frame that is arranged at the front in the short video clip; Based on the bipartite graphs corresponding to the two short video clips, respectively, calculating the similarity of the degree distribution of nodes in the two bipartite graphs; The degree distribution similarity and the object distribution similarity are weightedly summed to obtain the similarity between the two short video clips.

[0014] In a second aspect, an embodiment of the present application further provides a system for evaluating the similarity of video clip content, the system comprising: A data processing unit, used to obtain a video to be evaluated, and preprocess the video to be evaluated to obtain preprocessed video data; A video segmentation unit, configured to segment the video to be evaluated according to the preprocessed video data to obtain a plurality of short video clips; A key frame extraction unit, used to extract key frames from each short video segment to obtain key frames corresponding to each short video segment, wherein each short video segment corresponds to at least one key frame; A nearest neighbor graph construction unit, configured to construct a nearest neighbor graph corresponding to each key frame in each short video segment according to the key frame corresponding to each short video segment; A bipartite graph construction unit, configured to construct a bipartite graph corresponding to each short video segment according to a nearest neighbor graph corresponding to each key frame in each short video segment; A similarity calculation unit, used for calculating the similarity between the two short video clips based on the bipartite graph; The similarity evaluation unit is used to evaluate the similarity between the two short video clips according to the similarity.

[0015] In a third aspect, an embodiment of the present application further provides an electronic device comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute a method for evaluating the similarity of video clip content as described above.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method for evaluating the content similarity of video clips as described above.

[0017] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which: Figure 1 is a flowchart of an embodiment of a method for evaluating the similarity of video clip contents provided by the present application; Figure 2 It is a schematic diagram of the overall architecture of the best embodiment of the method for evaluating the similarity of video clip contents provided by the present application; Figure 3 is a schematic diagram of video scene segmentation in a best embodiment of the method for evaluating video clip content similarity provided by the present application; Figure 4 It is a schematic diagram of extracting and detecting key frames in the best embodiment of the method for evaluating the similarity of video clip contents provided by the present application; Figure 5 is a schematic diagram of a nearest neighbor graph in a best embodiment of a method for evaluating video clip content similarity provided by the present application; Figure 6 It is a schematic diagram of a bipartite graph representation in a best embodiment of the method for evaluating the similarity of video clip contents provided by the present application; Figure 7 Schematic diagram of the structure of an embodiment of a system for evaluating the similarity of video clip contents provided by the present application. DETAILED DESCRIPTION

[0019] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0020] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0021] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0022] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0023] First, some nouns involved in this application are analyzed: Video frame: A video frame is the basic unit in a video. A video frame represents a still image of a video. A video consists of a series of continuous video frames.

[0024] Keyframe: A video frame that has special importance in a video sequence, usually used to represent the content summary of the video or for video retrieval purposes.

[0025] Video retrieval: The process of retrieving specific video clips or key frames from a video database through a certain query method. The retrieval process may involve technologies such as content analysis and indexing.

[0026] Video analysis: Perform in-depth analysis of video content, including object detection, tracking, action recognition, etc., to obtain more specific information about the video content.

[0027] Parallel computing: A computing method that uses multiple computers or multi-core CPUs to process tasks simultaneously to improve computing efficiency and performance. In addition, it also has advantages such as scalability and flexibility.

[0028] Video scene segmentation technology: A technology that combines the information obtained from video analysis to divide the video to be detected into several short video clips according to the changes in the scene.

[0029] Automatic Speech Recognition (ASR): is a technology that automatically converts human speech into text.

[0030] Visual text recognition technology: a technology that recognizes and extracts text content from image files.

[0031] Natural Language Processing (NLP): is a technology that enables computers to understand, interpret and generate data in human language.

[0032] Target tracking technology: It is a computer vision method that identifies, locates and continuously tracks one or more moving targets in a video sequence or image sequence. It is widely used in video surveillance, autonomous driving, motion analysis and other fields. Specifically, this technology detects and tracks the target object by analyzing the spatial and temporal relationship between consecutive frames. It can keep tracking even when the target is partially blocked, the lighting changes or it moves rapidly.

[0033] Earth Mover's Distance (EMD): is a method to measure the similarity between two sets of distributions, which is widely used in image processing, computer vision, machine learning and other fields. The core idea of ​​EMD comes from the earth moving problem. Intuitively, it is to measure the minimum amount of work required to "move" one set of distributions to another. Mathematically, EMD is defined as the minimum transportation cost between two sets of distributions. It calculates the cost of moving "mass" from one distribution to another, and the cost is calculated by multiplying the transportation distance and the mass.

[0034] Hausdorff Distance: is a mathematical measure used to measure the similarity between two sets. Specifically, it is used to measure the maximum "minimum distance" between two sets, that is, the minimum distance from a point in one set to the farthest point in the other set. It is often used to compare differences between shapes, point sets, paths, etc.

[0035] Bipartite Graph: A special type of graph in which the nodes can be divided into two disjoint subsets, and each edge connects the nodes in one subset to the nodes in the other subset. Bipartite graph is a structured graph model that is widely used in various practical problems, especially when the problem involves two types of objects and their relationship. Bipartite graph provides a clear and effective way of representation and analysis.

[0036] Node Degree Distribution: It is an important concept in graph theory, which is used to describe the connection of nodes in a graph. Specifically, node degree distribution refers to the distribution of the degree of each node in the graph (that is, the number of edges connected to the node). It describes the number or proportion of nodes with different degrees in the graph. It is one of the important indicators for analyzing the graph structure and reflects the topological structure of a graph to a certain extent.

[0037] Existing related research methods are limited to a single keyframe. Frame-level feature methods often extract image features based on a single frame and can only reflect the spatial characteristics of static scenes. They rely heavily on the visual features of the image and are easily affected by image quality (such as clarity, brightness, background noise, etc.). They ignore the temporal relationship between frames in the video and cannot capture motion trends or behavioral characteristics in dynamic scenes. The essence of video is dynamic changes in a continuous temporal sequence, and frame-level feature methods have no direct modeling capabilities for these changes.

[0038] In addition, existing related research methods require a large amount of training data. The spatiotemporal feature method usually relies on the training of deep learning models, which requires a large amount of labeled data to ensure the generalization ability of the model. The computing resource requirements are high. Deep learning models (such as 3D-CNN) require a lot of computing resources and memory, and may face performance bottlenecks in real-time processing and large-scale data processing. The spatiotemporal feature method has high requirements on the quality of the video (such as resolution and frame rate), and low-quality videos may lead to inaccurate feature extraction. Although it can capture the spatial and temporal features of the video at the same time, it is completely left to the model to learn the features of the video content, and the overall interpretability is poor.

[0039] Therefore, the existing related research methods are not accurate enough in evaluating the similarity of video clip content, and the overall method has poor interpretability.

[0040] In order to solve the above-mentioned problem of inaccurate evaluation of video clip content similarity and the problem of poor interpretability of the overall method, the present application proposes a method, system, electronic device and medium for evaluating video clip content similarity.

[0041] Reference Figure 1 The present application provides a method for evaluating the similarity of video clip contents, the method comprising the following steps: Step S100: obtaining a video to be evaluated, and preprocessing the video to be evaluated to obtain preprocessed video data; Step S200: segment the video to be evaluated according to the preprocessed video data to obtain multiple short video clips; Step S300: extracting key frames from each short video segment to obtain key frames corresponding to each short video segment, wherein each short video segment corresponds to at least one key frame; Step S400: construct a nearest neighbor graph corresponding to each key frame in each short video segment according to the key frame corresponding to each short video segment; Step S500: construct a bipartite graph corresponding to each short video segment according to the nearest neighbor graph corresponding to each key frame in each short video segment; Step S600: Calculate the similarity between two short video clips based on the bipartite graph; Step S700: Evaluate the similarity between two short video clips according to the similarity.

[0042] In this embodiment, by obtaining a video to be evaluated and preprocessing the video to be evaluated to obtain preprocessed video data; according to the preprocessed video data, the video to be evaluated is segmented to obtain multiple short video clips; the key frames in each short video clip are extracted to obtain the key frames corresponding to each short video clip, wherein each short video clip corresponds to at least one key frame; according to the key frames corresponding to each short video clip, a nearest neighbor graph corresponding to each key frame in each short video clip is constructed; according to the nearest neighbor graph corresponding to each key frame in each short video clip, a bipartite graph corresponding to each short video clip is constructed; based on the bipartite graph, the similarity between two short video clips is calculated; according to the similarity, the similarity of the two short video clips is evaluated. In this way, a bipartite graph corresponding to each short video clip is constructed through a series of step methods. Since the bipartite graph can clearly represent the relationship and change between objects in the video clip, the bipartite graph modeling effectively captures the structural change information of the video clip. Therefore, based on the bipartite graph, the similarity between two short video clips is calculated, which can improve the accuracy of evaluating the similarity of the content of the video clips. And by analyzing the changes in nodes and edges in the bipartite graph, we can directly observe the appearance, disappearance, and changes in relationships of objects. This structured representation method is easier to understand and explain than existing related technologies, improving the interpretability of the overall method.

[0043] The above-mentioned method of performing video segmentation on the video to be evaluated based on the preprocessed video data to obtain multiple short video clips can be performed by combining the preprocessed video data with the video content of the video to be evaluated, roughly determining the time segmentation points of the video, and then determining the scene change points, and finally performing video segmentation on the video to be evaluated based on the time segmentation points and the scene change points to obtain multiple short video clips.

[0044] The key frames in each short video clip may be extracted by preset conditions, for example, frames that meet the preset conditions are used as key frames; and the key frames in each short video clip are extracted by parallel computing, which can improve the efficiency of key frame extraction. The key frames in each short video clip may also be extracted by existing technologies known to those skilled in the art. This embodiment is not specifically limited.

[0045] The above-mentioned method of evaluating the similarity of two short video clips according to the similarity may be to evaluate the similarity of the two short video clips according to the similarity using a threshold method. It may also be to evaluate the similarity of the two short video clips according to the similarity using a top-k query method. It may also be possible to evaluate the similarity of the two short video clips using existing technologies known to those skilled in the art. This embodiment is not specifically described or limited.

[0046] In some implementations, obtaining a video to be evaluated and preprocessing the video to be evaluated to obtain preprocessed video data includes: Convert the format of the video to be evaluated to obtain a video after the format conversion; Convert the speech in the format-converted video into speech text; Extract visual text from converted videos; Perform data processing of this article on the speech text and the visual text to obtain processed speech text and processed visual text; Natural language processing is used to analyze the processed speech text and the processed visual text to obtain preprocessed video data.

[0047] In this embodiment, the format of the video to be evaluated is converted to obtain a format-converted video; the speech in the format-converted video is converted into speech text; the visual text in the format-converted video is extracted; the speech text and the visual text are processed in this paper to obtain processed speech text and processed visual text; the processed speech text and processed visual text are analyzed by natural language processing to obtain pre-processed video data. In this way, by performing video analysis on the video, the understanding of the video scene is achieved, which can lay a good data foundation for the later video segmentation, thereby potentially screening out more representative key frames.

[0048] In some implementations, the video to be evaluated is segmented according to the preprocessed video data to obtain a plurality of short video clips, including: Based on the preprocessed video data, determine the time segmentation points of the video to be evaluated; Calculate the pixel difference between two adjacent time division points, and determine the scene change point and the target time division point based on the pixel difference; According to the target time segmentation points and scene change points, the video to be evaluated is segmented to obtain multiple short video clips.

[0049] In this embodiment, based on the pre-processed video data, the time segmentation point of the video to be evaluated is determined; the pixel difference at two adjacent time segmentation points is calculated, and the scene change point and the target time segmentation point are determined according to the pixel difference; according to the target time segmentation point and the scene change point, the video to be evaluated is segmented to obtain multiple short video clips. In this way, according to the target time segmentation point and the scene change point, the video to be evaluated is segmented, which can achieve efficient scene segmentation and ensure scene consistency. It can lay a good data foundation for the later key frame extraction, that is, the video is segmented into a series of short video clips, which can improve the efficiency and accuracy of key frame extraction.

[0050] In some implementations, extracting a key frame from each short video segment to obtain a key frame corresponding to each short video segment includes: Perform target detection on each frame in each short video clip to obtain multiple objects and label each object; Calculate the geometric center point of each object, obtain the position coordinates of the object, and use the geometric center point to represent the object; According to the position coordinates of the objects, the relative position change value between the two objects is calculated; Compare all objects in the next frame with all objects in the previous frame. If any of the following conditions exists: a new label object appears, an existing label object disappears, or the relative position change between two objects exceeds a preset threshold, the next frame is taken as a key frame. If any of the following conditions exists in the short video clip: no new label objects appear, existing label objects disappear, or the relative position change between two objects exceeds a preset threshold, the first and last frames of the short video clip are taken as key frames to extract the key frames corresponding to each short video clip.

[0051] In this embodiment, by performing target detection on each frame in each short video clip, multiple objects are obtained, and a label is given to each object; the geometric center point of each object is calculated to obtain the position coordinates of the object, and the geometric center point is used to represent the object; the relative position change value between two objects is calculated based on the position coordinates of the object; all objects in the latter frame are compared with all objects in the previous frame, and if there is any condition of the appearance of a new label object, the disappearance of an existing label object, and the relative position change value between the two objects exceeds a preset threshold, the latter frame is used as a key frame; if there is no new label object, the disappearance of an existing label object, and the relative position change value between the two objects exceeds a preset threshold in the short video clip, the first and last two frames of the short video clip are used as key frames to extract the key frames corresponding to each short video clip. In this way, by using the geometric center point to represent the object, the position relationship between objects can be better reflected, thereby improving the accuracy of key frame extraction.

[0052] The above-mentioned object detection is performed on each frame in each short video clip to obtain multiple objects. Different short video clips can be input into different processors or computing nodes for parallel computing, that is, target tracking technology is used to perform object detection on each frame in each short video clip to obtain multiple objects. Parallel computing can improve the speed and efficiency of video detection, thereby improving the efficiency of key frame extraction.

[0053] The target tracking technology can adopt the existing technology known to those skilled in the art, for example, ByteTrack and deepsort target tracking algorithms, and is generally used together with target detection such as YOLO to form a whole.

[0054] In some implementations, constructing a nearest neighbor graph corresponding to each key frame in each short video segment according to the key frame corresponding to each short video segment includes: Get all objects in each keyframe and use the geometric center point of each object as a node of the graph; Calculate the nearest node of each object based on the geometric center point of each object; Undirected edges are constructed between each object and its corresponding nearest node to build a nearest neighbor graph corresponding to each key frame in each short video clip.

[0055] In this embodiment, all objects in each key frame are obtained, and the geometric center point of each object is used as a node of the graph; the nearest node of each object is calculated based on the geometric center point of each object; an undirected edge is constructed between each object and its corresponding nearest node to construct a nearest neighbor graph corresponding to each key frame in each short video clip. In this way, it is only necessary to convert the key frame into a nearest neighbor graph to determine the connection relationship between nodes, without connecting each node to all other nodes. It can better understand the relationship and changes between objects, and lay a good data foundation for the later construction of bipartite graphs and similarity calculations.

[0056] In some implementations, constructing a bipartite graph corresponding to each short video segment according to a nearest neighbor graph corresponding to each key frame in each short video segment includes: All objects in the nearest neighbor graph corresponding to each key frame are represented by sets to obtain the set corresponding to each key frame; According to the set corresponding to each key frame, the edge type between each object in two key frames in each short video clip is constructed, where the edge type includes the edge where the object moves itself, the edge where the neighbor object moves, and the edge where the object appears or disappears; According to the set and edge type corresponding to each key frame, a bipartite graph representation between two key frames is constructed; According to the bipartite graph representation, a bipartite graph corresponding to each short video clip is constructed, and the bipartite graph contains the degree distribution of the nodes.

[0057] In this embodiment, all objects in the nearest neighbor graph corresponding to each key frame are represented by a set to obtain a set corresponding to each key frame; according to the set corresponding to each key frame, the edge type between each object in two key frames in each short video clip is constructed, wherein the edge type includes the edge of the object itself moving, the edge of the neighbor object moving, and the edge of the object appearing or disappearing; according to the set and edge type corresponding to each key frame, a bipartite graph representation between two key frames is constructed; according to the bipartite graph representation, a bipartite graph corresponding to each short video clip is constructed, and the bipartite graph contains the degree distribution of the nodes. In this way, the bipartite graph is constructed because the bipartite graph can clearly represent the relationship and change between objects in the video clip, and the bipartite graph modeling effectively captures the structured change information of the video clip. A good data foundation is laid for calculating the similarity between two short video clips to improve the accuracy of evaluating the similarity of video clip content. And by analyzing the changes in nodes and edges in the bipartite graph, the appearance, disappearance and changes in the relationship between objects can be directly observed. This structured representation method is easier to understand and explain than the existing related technologies, and improves the interpretability of the overall method.

[0058] In some implementations, calculating the similarity between two short video clips based on a bipartite graph includes: Based on the first key frames corresponding to the two short video clips, respectively, calculating the object distribution similarity between the two first key frames, wherein the first key frame is the key frame that is arranged at the front in the short video clip; Based on the bipartite graphs corresponding to the two short video clips, calculate the similarity of the degree distribution of the nodes in the two bipartite graphs; The similarity between two short video clips is obtained by performing a weighted sum calculation on the degree distribution similarity and the object distribution similarity.

[0059] In this embodiment, based on the bipartite graphs corresponding to the two short video clips, the degree distribution similarity of the nodes in the two bipartite graphs is calculated; the degree distribution similarity and the object distribution similarity are weighted and calculated to obtain the similarity between the two short video clips. In this way, the degree distribution similarity and the object distribution similarity are comprehensively considered to obtain a more reasonable and accurate similarity between the two short video clips, thereby improving the accuracy of evaluating the similarity of the video clip content.

[0060] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below: The existing technical solutions have the following deficiencies: (1) Shortcomings of similarity assessment methods based on frame-level features.

[0061] This method is limited to a single keyframe. Frame-level feature methods often extract image features based on a single frame and can only reflect the spatial characteristics of static scenes. It relies heavily on the visual features of the image and is easily affected by image quality (such as clarity, brightness, and background noise). It ignores the temporal relationship between frames in the video and cannot capture motion trends or behavioral characteristics in dynamic scenes. The essence of video is dynamic changes in a continuous temporal sequence, and frame-level feature methods have no direct modeling capabilities for these changes.

[0062] (2) The shortcomings of similarity assessment methods based on spatiotemporal features.

[0063] This method requires a large amount of training data. The spatiotemporal feature method usually relies on the training of deep learning models, which requires a large amount of labeled data to ensure the generalization ability of the model. The computing resource requirements are high. Deep learning models (such as 3D-CNN) require a lot of computing resources and memory, and may face performance bottlenecks in real-time processing and large-scale data processing. The spatiotemporal feature method has high requirements on the quality of the video (such as resolution, frame rate, etc.). Low-quality videos may lead to inaccurate feature extraction. Although it can capture the spatial and temporal features of the video at the same time, it is completely left to the model to learn the features of the video content, and the overall interpretability is poor.

[0064] This embodiment proposes a scheme for evaluating the similarity of video content by extracting and detecting key frames based on scene segmentation, representing the content of video clips using bipartite graphs, and evaluating the similarity of video content. The scheme divides the video into several short video clips according to the consistency of the scene, and obtains the object information in the frame and the corresponding relationship between the objects between two frames through parallel calculation and target tracking technology. By analyzing the movement of objects, changes in neighbor relationships, and the appearance or disappearance of objects in key frames, the key frames of each short video clip are extracted and constructed into corresponding bipartite graphs to represent the changing relationship between objects in the video clips, thereby capturing the "intrinsic pattern" of the short video clips. In addition, by calculating the similarity of the bipartite graphs corresponding to the short video clips, the weighted fusion of the object distribution similarity (i.e., the object distribution similarity, calculated based on the Hausdorff distance) and the node degree distribution similarity (i.e., the degree distribution similarity, calculated based on the EMD distance) is combined to accurately evaluate the similarity of the video content. Compared with the traditional method, this embodiment effectively captures the structural change information of the video clips through bipartite graph modeling, realizes the accurate quantification of the changes in video content in complex scenes, and has high robustness and adaptability. The technical scheme of this embodiment specifically includes the following contents: Reference Figure 2 The technical solution of this embodiment mainly includes processes such as video input, key frame extraction and detection, bipartite graph video content representation, similarity calculation, and video clip similarity determination. The technical solution of this embodiment mainly focuses on modules such as key frame extraction, bipartite graph video content representation, and similarity calculation. Specifically: 1. Key frame extraction and detection scheme based on scene segmentation.

[0065] In order to effectively improve the processing speed and obtain the key information of the video when processing video data, this embodiment proposes an innovative key frame extraction and detection scheme. The scheme realizes the key frame extraction and detection tasks through strategies of video scene segmentation technology, parallel computing and target tracking technology. Its core lies in using video scene segmentation technology to segment the video into a collection of short video clips to optimize the speed and effect of key frame extraction. And through technologies such as parallel computing and target tracking, the efficiency and accuracy of key frame extraction are further improved. This scheme not only significantly improves the accuracy and processing speed of key frame extraction, but also has a high degree of flexibility and scalability, and is suitable for video data processing needs of different scales and application scenarios.

[0066] 1. Introduction to video scene segmentation module.

[0067] The core of video scene segmentation is to divide the video to be detected into several short video clips, so as to achieve a trade-off between the speed and accuracy of video frame extraction. At the same time, it is necessary to understand the semantics of the video as a whole and the clips, and retain as much valuable content as possible. Figure 3,Video scene segmentation is mainly divided into two stages: video analysis and scene change detection.,First, video analysis is used to understand the overall content of the video,,providing basic information such as semantics and time points for scene change detection.,Then, through scene change detection, the video is segmented into a collection of,several short video clips.,This video scene segmentation design can segment the video into multiple short video clips,according to the semantics of the video and the changes of the scene,to achieve a balance between the speed and accuracy of video frame extraction,,thus improving the efficiency and performance of key frame extraction.

[0068] 1.1. Video analysis module.

[0069] Video analysis is the first step of the key frame extraction and detection scheme based on scene segmentation. It is mainly responsible for providing information and data basis for scene change detection, thereby screening out potential key frames. The video analysis module analyzes the video content by extracting information from the video, providing information and data basis for subsequent scene change detection, thereby potentially screening out more representative key frames. The characteristics of the video analysis module: high efficiency, wide applicability, strong practicality and good performance.

[0070] Specific process: (1) Video input (i.e., the video to be evaluated): Receive video data, including video file upload or video stream capture.

[0071] (2) Video preprocessing: Convert the input video data into the same and appropriate format, such as uniformly adjusting the resolution and frame rate, to eliminate the differences between different videos to ensure the consistency of video content and the accuracy of analysis.

[0072] (3) Text information extraction and integration: Automatic speech recognition (ASR) technology is used to convert the speech in the video into text format (i.e., obtain speech text). If the input video has corresponding subtitles, visual text recognition technology is used to detect and extract the visual text in the video frame. Then, they are integrated according to the time order in which they appear in the video to ensure the synchronization of the text and video content.

[0073] (4) Perform text preprocessing on the spoken and visual texts: remove irrelevant content in the text data, such as special symbols, numbers, etc. Then convert the text data into a standard format, including lowercase words and removing stop words. In addition, perform word segmentation on the text to split it into words or phrases to prepare for natural language processing (NLP) analysis.

[0074] (5) NLP analysis: Identify key entities and grammatical connections between words in preprocessed text data (i.e., processed speech text and processed visual text), and combine the analysis results with the video visual content to achieve understanding of the video scene.

[0075] (6) Data transfer: The results of NLP analysis (i.e., preprocessed video data) are transferred to the scene change detection module to provide information data support for video segmentation.

[0076] 1.2 Scene change detection module.

[0077] The scene change detection module is the second step of the key frame extraction and detection scheme based on scene segmentation. It is mainly responsible for segmenting the video into a series of short video clips, aiming to improve the efficiency and accuracy of key frame extraction. By receiving the information and data provided by the video analysis module (i.e., pre-processed video data), the scene change detection module can achieve efficient scene segmentation to accelerate video processing and optimize the representativeness of the extracted key frames to ensure that they can better reflect the core part of the video content.

[0078] Specific process: (1) Data extraction: receiving data provided by the video analysis module (i.e., preprocessed video data).

[0079] (2) Scene consistency analysis: Combine the provided data with the video content to roughly determine the time segmentation points of the video and ensure the scene consistency of each short video clip.

[0080] (3) Scene change detection: For adjacent time division points, the inter-frame difference is used to further define the consistency of scene changes and content. Specifically, this embodiment first extracts adjacent time division points in the video, and then sets a pixel difference threshold P. Next, the pixel difference at adjacent time division points is calculated respectively. The calculated pixel difference result is compared with the preset pixel difference threshold. After screening, the scene change point and time point (i.e., the target time segmentation point) of each video clip are further defined.

[0081] (4) Video segmentation: Segment the video into a collection of short video clips based on the detected scene change points and time points . Each fragment and snippets Represents a complete and continuous scene segment respectively.

[0082] (5) Dynamic adjustment: According to the video segmentation results and actual needs, the pixel difference threshold is dynamically adjusted to achieve the optimal effect. The short video clips are detected through the subsequent detection module.

[0083] 2. Introduction to key frame extraction and detection module.

[0084] Reference Figure 4, the key frame extraction and detection module is designed to extract and detect the key frames of each short video clip. Specifically, in order to improve the speed and efficiency of video detection, the key frames of multiple short video clips are extracted and detected simultaneously through parallel computing methods. For the detection of short video clips, this embodiment uses target tracking technology to obtain the motion trajectory and position information of objects in the video, and concentrates the detected objects to a central point through the central point representation of the detected objects to optimize the position recognition and representation of the objects, thereby improving the accuracy of key frame extraction.

[0085] Specific process: (1) Short video clip input: The short video clip obtained by the video scene segmentation module is used as input.

[0086] (2) Parallel computing: Short video clip data is distributed to different processors or computing nodes based on the number of available processors or computing cores, and key frame extraction and detection are performed simultaneously.

[0087] (3) Target tracking: Use target tracking technology to obtain the motion trajectory of the object. Perform target detection, recognition, and object labeling on each frame of the short video clip, and assign a unique tracking ID to each object. Predict the position of each object in the next frame, match objects between frames, and maintain tracking continuity. At the same time, obtain the relative position relationship and absolute position of the object in each frame image. Among them, the relative position relationship refers to the spatial relationship between objects, and the absolute position refers to the specific coordinate position of the object in the video frame.

[0088] (4) Representation of the center point of the detected object: Since the target tracking technology detects the bounding box of each object, for each detected object, the geometric center point of its bounding box (i.e., the centroid of the bounding box) is calculated. By using these geometric center points to represent the detected object, the positional relationship between objects can be better reflected.

[0089] (5) Key frame selection: The key frame selection is based on the following conditions: 1) New label objects appear. For example, if there are people and dogs in the previous frame and people, cats and dogs in the next frame, then the cat is a new label object and the next frame is marked as a key frame.

[0090] 2) Existing labeled objects disappear. For example, if there are people, cats, and dogs in the previous frame and people and dogs in the next frame, then the cat is an existing labeled object that disappears, and the next frame is marked as a key frame.

[0091] 3) When the change in relative position between labeled objects exceeds a threshold, the frame is marked as a key frame.

[0092] 4) Special case: If there is a short video clip in the short video clip set that does not meet any of the three conditions: 1) the appearance of a new label object, 2) the disappearance of an existing label object, and 3) the change in the relative position between label objects exceeds a threshold, then the first and last frames of the short video clip are taken as the key frames of the short video clip.

[0093] Through the above process, this embodiment divides the video into several short videos based on the changes in the video scene, and obtains the key frames in each short video segment based on parallel computing and target tracking methods, and classifies and sorts the key frames according to the scene and time sequence. This innovative key frame extraction and detection solution based on scene segmentation improves the accuracy, relevance and efficiency of key frame extraction.

[0094] 2. Video clip content representation scheme based on bipartite graph.

[0095] In order to confirm the similarity of the videos, this embodiment proposes an innovative video clip content representation scheme of converting key frames into bipartite graphs. This scheme realizes the task of converting key frames into bipartite graphs by converting each key frame in the same short video clip into a nearest neighbor graph and then converting the nearest neighbor graph into a bipartite graph.

[0096] 1. Introduction to keyframe conversion to nearest neighbor graph.

[0097] Reference Figure 5 , Figure 5 a, b, c and d in represent nodes. Converting the key frame into a nearest neighbor graph is the first step of the bipartite graph construction scheme. To simplify the problem, this embodiment only converts the key frame into an undirected nearest neighbor graph for characterization.

[0098] Specific process: (1) Graph definition: Convert keyframe images into graph structures.

[0099] (2) Node definition: In the graph structure, the center point of the detected object in the key frame is used as the node of the graph.

[0100] (3) Edge construction: Based on the absolute position Mn of the detected object and the Euclidean distance calculation formula: ; in, Represents the absolute position of an object (i.e. node). Represents the absolute position of another object, and for each pair of nodes calculates the Euclidean distance between it and other nodes , identify the nodes closest to each node and build undirected edges between these nodes, that is, build a nearest neighbor graph.

[0101] 2. Introduction to the conversion of nearest neighbor graph into bipartite graph.

[0102] Converting the nearest neighbor graph into a bipartite graph representation is the second step of the bipartite graph construction scheme.

[0103] Specific process: (1) Representation of subsets and nodes: In a short video clip, the detection objects of the first and second key frames of the clip are represented by non-overlapping sets. For example, the first subset , the second subset ,in, In Corresponding to In Nodes represent the states of the same object in consecutive frames, while Indicates the new object that appears in the second keyframe. And so on according to the sequence of frames.

[0104] (2) Representation of edges: and The edges are defined by the following properties: edge type, i.e., the absolute position coordinates of the object in the video frame, the direction in which the object moves, and the distance (Euclidean distance) that the object moves.

[0105] (3) Edge type representation: refer to Figure 6 , the edge type representation is based on the following conditions: 1) Edges between the same objects (i.e. the edges where the objects themselves move): ; Indicates the change (movement) of the object itself.

[0106] 2) Edges between neighbors (i.e. edges where neighbor objects move): ; Indicates the change (movement) relative to the neighbor. That is, if two objects in a set are neighbors, the movement change of one object relative to its neighbor object.

[0107] 3) Edges reflecting the appearance or disappearance of objects: ;express represents a newly appeared object, whereas a disappeared object.

[0108] 3. Video content similarity assessment scheme (similarity calculation) based on bipartite graph representation method.

[0109] After completing the bipartite graph representation of the short video clips, this embodiment evaluates the similarity of the video content by calculating the similarity of the bipartite graphs corresponding to the two short video clips.

[0110] Before calculation, the similarity is defined. The short video clip similarity considered in this embodiment can be defined as a comprehensive measure of the similarity of the initial distribution of objects in the scene and the similarity of the object motion trend. The similarity of the initial distribution of objects in the scene mainly refers to the consistency of the spatial position, quantity and arrangement pattern of different objects at the beginning of the video clip. This distribution reflects the overall structural characteristics at the beginning of the scene, which helps to capture the static characteristics at the beginning of the scene. The similarity of the object motion trend focuses on the dynamic changes of the object in the time dimension, including the motion patterns, relative position changes (such as approaching, moving away) and other features between objects, reflecting the dynamic characteristics of the scene. By combining these two dimensions, the similarity between video clips can be fully characterized, thereby more accurately supporting the analysis and processing of video content.

[0111] Process overview (this embodiment takes two key frames as an example): (1) Based on the Hausdorff distance, the object distribution similarity of the first key frame in two short video clips is calculated (i.e., the object distribution similarity is obtained) to capture the static characteristics of the initial scene in a short video clip.

[0112] (2) Based on the EMD distance, the degree distribution similarity of the nodes in the bipartite graph (i.e., degree distribution similarity) is calculated. The degree distribution reflects the topological structure characteristics of the bipartite graph to a certain extent and is used to capture the dynamic changes of objects in the time dimension.

[0113] (3) The similarities between the two parts are weighted and fused to obtain the final similarity score (i.e., the similarity between the two short video clips).

[0114] (4) Determine whether there is similarity between video clips based on the threshold method or top-k query method.

[0115] For ease of understanding, this embodiment takes the following example: Suppose there are two short video clips, each using and Short video clip The first keyframe (starting frame) in is , the second key frame is ; Short video clips The first keyframe (starting frame) in is , the second key frame is .

[0116] Keyframes To Keyframe The object position set in is , , , For example: Object Keyframe , ; Object Keyframe , . And the coordinates here are normalized. For example, The coordinates are .

[0117] Short video clips and The corresponding bipartite graph is and . Bipartite graph The left set of , the right set is , the edge set is ; Bipartite graph The left set of , the right set is , the edge set is . Node Collection , , , The degree distributions of , , , .

[0118] The distance calculation process is: 1. Calculate the object distribution similarity in the two start frames based on Hausdorff distance . That is, calculate the key frame and keyframes The Hausdorff distance of the object in, for The object position collection, for A collection of object positions in a keyframe. and keyframes The Hausdorff distance of the object in is Since the smaller the distance, the higher the similarity, the similarity Calculated as: ; 2. Calculate the degree distribution similarity of nodes in the bipartite graph based on EMD distance . Degree distribution calculation method: Calculate separately , , , The degree distribution of the midpoint nodes is obtained , , , For example, the point set Midpoint The degree is ,Will Normalized to between 0 and 1 (i.e. Divide by the point set The total degree of all nodes in the node distribution . Then calculate the left sets of the two bipartite graphs respectively and Similarity , and two right sets of bipartite graphs and Similarity Since the smaller the distance, the higher the similarity, the similarity Calculated as: ; 1) Calculate the node degree distribution. Suppose a bipartite graph is given ,in, is a node of the left set, is a node of the right set, is the set of edges between nodes.

[0119] 2) Calculate the EMD distance between two distributions. It should be noted that EMD distance calculation is a prior art and will not be described in detail in this embodiment.

[0120] 3) Calculate the degree distribution similarity of nodes in the bipartite graph based on EMD distance Calculation process. Calculated by the process in 1) and The node degree distribution of , , , , will get , , , Substitute into 2) and calculate separately and , calculated from this .

[0121] 3. Calculate the comprehensive similarity (i.e. the similarity between two short video clips) , ,in, and represents the weight coefficient, , can be made Or adjust according to different focuses.

[0122] 4. Based on the above calculation results, there are two solutions to determine whether video clips are similar: (1) Threshold method, setting the threshold ,when , the two video clips are considered similar, otherwise they are dissimilar.

[0123] (2) Top-k query: calculate the similarity between short video segment a and all short video segments in the selected video segment set one by one, sort the similarity results, and the short video segments with the top k similarity in the selected set are considered to be similar to short video segment a.

[0124] Summarize: The technical solution of this embodiment designs a method for representing the content of video clips based on video scene segmentation and bipartite graphs. By introducing the concepts of video scene segmentation and bipartite graphs, this embodiment can capture the movement of objects and changes in neighbor relationships in each short video clip, and fully consider the combination of time and space. This method can effectively make up for the defects of traditional frame-level methods that cannot capture dynamic behavior and timing information. Video scene segmentation divides the video into several video clips through video analysis and scene consistency. The bipartite graph clearly represents the relationship and changes between objects in the video clips, so that the changes in video content can be visualized through the structure of the graph. For example, by analyzing the changes in nodes and edges in the bipartite graph, the appearance, disappearance and changes in mutual relationships of objects can be directly observed. This structured representation method is easier to understand and explain than high-dimensional feature vectors in deep learning.

[0125] At the same time, the problem of computing resources and data requirements of the spatiotemporal feature method is solved to a certain extent. Compared with the deep learning method that requires a large amount of training data and computing resources, the method of this embodiment avoids the deep learning model's dependence on a large amount of labeled data by constructing a bipartite graph and combining target tracking technology. This not only reduces the demand for computing resources, but also improves processing efficiency.

[0126] The technical solution of this embodiment mainly includes the following three parts: (1) Key frame extraction and detection scheme based on scene segmentation: The video is segmented into several consecutive short video clips in the same scene, and these short video clips are processed simultaneously through parallel computing to extract and detect key frames respectively.

[0127] (2) Video clip content representation scheme based on bipartite graph. The content changes (such as object movement) in two frames with a temporal relationship are represented by a bipartite graph. The left set is all objects in the first frame, and the right set is all objects in the second frame. The changes in the objects themselves and the changes between objects are represented as the edges of the left and right sets in the bipartite graph.

[0128] (3) On the basis of the bipartite graph, the similarity of the video content is characterized by a comprehensive measure of the similarity of the initial distribution of objects in the scene and the similarity of the object motion trends. The similarity is calculated based on the Hausdorff distance and the EMD distance, respectively. Finally, the two similarities are weighted to obtain the similarity of the two bipartite graphs, that is, the similarity of the two video clips.

[0129] Compared with the prior art, the technical solution of this embodiment has the following advantages: This embodiment is based on the key frame extraction of scene segmentation, the representation of video clip content by bipartite graph, and the similarity calculation and evaluation method. By segmenting the scene and effectively capturing the movement of objects in the video clip, the change of neighbor relations, and the appearance and disappearance of objects, it successfully solves the problem that the traditional method cannot accurately quantify the changes in video content in dynamic scenes. This method not only improves the accuracy of similarity evaluation, but also significantly reduces the dependence on a large amount of labeled data and computing resources, and has strong robustness and adaptability. In addition, the structured representation based on the bipartite graph improves the interpretability of the model, making the similarity evaluation process of video content more transparent and intuitive, easy to understand and analyze, and has broad application prospects.

[0130] Reference Figure 7 The embodiment of the present application further provides a system for evaluating the similarity of video clip contents, the system comprising a data processing unit 100, a video segmentation unit 200, a key frame extraction unit 300, a nearest neighbor graph construction unit 400, a bipartite graph construction unit 500, a similarity calculation unit 600 and a similarity evaluation unit 700, wherein: The data processing unit 100 is used to obtain the video to be evaluated, and preprocess the video to be evaluated to obtain preprocessed video data; The video segmentation unit 200 is used to segment the video to be evaluated according to the preprocessed video data to obtain multiple short video clips; The key frame extraction unit 300 is used to extract the key frame in each short video segment to obtain the key frame corresponding to each short video segment, wherein each short video segment corresponds to at least one key frame; The nearest neighbor graph construction unit 400 is used to construct a nearest neighbor graph corresponding to each key frame in each short video segment according to the key frame corresponding to each short video segment; A bipartite graph construction unit 500, configured to construct a bipartite graph corresponding to each short video segment according to a nearest neighbor graph corresponding to each key frame in each short video segment; A similarity calculation unit 600, used to calculate the similarity between two short video clips based on a bipartite graph; The similarity evaluation unit 700 is used to evaluate the similarity between two short video clips according to the similarity.

[0131] It should be noted that since the system for evaluating the content similarity of video clips in this embodiment and the method for evaluating the content similarity of video clips described above are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.

[0132] An embodiment of the present application also provides an electronic device, comprising: at least one control processor and a memory for communicating with the at least one control processor.

[0133] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0134] The non-transient software program and instructions required to implement the method for evaluating the similarity of video clip content in the above embodiment are stored in the memory. When executed by the processor, the method for evaluating the similarity of video clip content in the above embodiment is executed, for example, the above-described Figure 1 The method comprises steps S100 to S700.

[0135] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] The present application also provides a computer-readable storage medium storing computer-executable instructions. The computer-executable instructions are executed by one or more control processors to enable the one or more control processors to execute a method for evaluating the similarity of video clip content in the above method embodiment, for example, to execute the above described Figure 1 The functions of method steps S100 to S700 in the method.

[0137] It will be appreciated by those skilled in the art that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0138] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation methods. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the embodiments of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the embodiments of the present application.

[0139] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A method for evaluating the similarity of video clip contents, characterized in that: The method comprises: Acquire a video to be evaluated, and preprocess the video to be evaluated to obtain preprocessed video data; According to the preprocessed video data, the video to be evaluated is segmented to obtain multiple short video clips; Extracting key frames from each short video clip to obtain key frames corresponding to each short video clip, wherein each short video clip corresponds to at least one key frame; According to the key frames corresponding to each short video segment, construct a nearest neighbor graph corresponding to each key frame in each short video segment; Constructing a bipartite graph corresponding to each short video segment according to the nearest neighbor graph corresponding to each key frame in each short video segment; Based on the bipartite graph, calculating the similarity between the two short video clips; The similarity between the two short video clips is evaluated according to the similarity.

2. The method for evaluating the similarity of video clip contents according to claim 1, characterized in that: The step of obtaining a video to be evaluated and preprocessing the video to be evaluated to obtain preprocessed video data includes: Converting the format of the video to be evaluated to obtain a video after the format conversion; Converting the speech in the video after the format conversion into speech text; Extracting visual text from the video converted into the format; Performing data processing of the text on the speech text and the visual text to obtain processed speech text and processed visual text; The processed speech text and the processed visual text are analyzed by natural language processing to obtain preprocessed video data.

3. The method for evaluating the similarity of video clip contents according to claim 1, characterized in that: The step of segmenting the video to be evaluated according to the preprocessed video data to obtain a plurality of short video clips includes: Determining the time segmentation points of the video to be evaluated based on the preprocessed video data; Calculating the pixel difference between two adjacent time division points, and determining the scene change point and the target time division point according to the pixel difference; The video to be evaluated is segmented according to the target time segmentation point and the scene change point to obtain a plurality of short video clips.

4. The method for evaluating the similarity of video clip contents according to claim 1, characterized in that: The step of extracting the key frame from each short video segment to obtain the key frame corresponding to each short video segment includes: Performing target detection on each frame in each short video clip to obtain multiple objects, and labeling each of the objects; Calculating the geometric center point of each of the objects to obtain the position coordinates of the object, and using the geometric center point to represent the object; Calculating a relative position change value between two objects according to the position coordinates of the objects; Compare all objects in the next frame with all objects in the previous frame. If any of the following conditions exists: a new label object appears, an existing label object disappears, or the relative position change between two objects exceeds a preset threshold, the next frame is taken as a key frame. If any of the following conditions is met in the short video clip: no new label object appears, an existing label object disappears, or the relative position change between two objects exceeds a preset threshold, the first and last two frames of the short video clip are taken as key frames to extract the key frames corresponding to each short video clip.

5. The method for evaluating the similarity of video clip contents according to claim 4, characterized in that: The step of constructing a nearest neighbor graph corresponding to each key frame in each short video segment according to the key frame corresponding to each short video segment includes: Obtain all objects in each key frame, and use the geometric center point of each object as a node of the graph; Calculate the nearest node of each object according to the geometric center point of each object; An undirected edge is constructed between each of the objects and the nearest node corresponding thereto, so as to construct a nearest neighbor graph corresponding to each key frame in each of the short video clips.

6. The method for evaluating the similarity of video clip contents according to claim 1, characterized in that: The step of constructing a bipartite graph corresponding to each short video segment according to the nearest neighbor graph corresponding to each key frame in each short video segment includes: Representing all objects in the nearest neighbor graph corresponding to each key frame by using a set to obtain a set corresponding to each key frame; According to the set corresponding to each key frame, construct the edge type between each object in two key frames in each short video clip, wherein the edge type includes the edge where the object itself moves, the edge where the neighboring object moves, and the edge where the object appears or disappears; Constructing a bipartite graph representation between two key frames according to the set corresponding to each key frame and the edge type; According to the bipartite graph representation, a bipartite graph corresponding to each short video clip is constructed, and the bipartite graph includes a degree distribution of nodes.

7. The method for evaluating the similarity of video clip contents according to claim 1, characterized in that: The calculating the similarity between the two short video clips based on the bipartite graph includes: Calculating the object distribution similarity between the two first key frames based on the first key frames respectively corresponding to the two short video clips, wherein the first key frame is the key frame that is arranged at the front in the short video clip; Based on the bipartite graphs corresponding to the two short video clips, respectively, calculating the similarity of the degree distribution of nodes in the two bipartite graphs; The degree distribution similarity and the object distribution similarity are weightedly summed to obtain the similarity between the two short video clips.

8. A system for evaluating the similarity of video clip content, characterized in that: The system comprises: A data processing unit, used to obtain a video to be evaluated, and preprocess the video to be evaluated to obtain preprocessed video data; A video segmentation unit, configured to segment the video to be evaluated according to the preprocessed video data to obtain a plurality of short video clips; A key frame extraction unit, used to extract key frames from each short video segment to obtain key frames corresponding to each short video segment, wherein each short video segment corresponds to at least one key frame; A nearest neighbor graph construction unit, configured to construct a nearest neighbor graph corresponding to each key frame in each short video segment according to the key frame corresponding to each short video segment; A bipartite graph construction unit, configured to construct a bipartite graph corresponding to each short video segment according to a nearest neighbor graph corresponding to each key frame in each short video segment; A similarity calculation unit, used for calculating the similarity between the two short video clips based on the bipartite graph; The similarity evaluation unit is used to evaluate the similarity between the two short video clips according to the similarity.

9. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the method for evaluating the similarity of video segment contents as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method for evaluating the similarity of video segment contents according to any one of claims 1 to 7.

Images