A method and system for video redundant segment detection

Abstract

The application provides a video redundant segment detection method and system. The method comprises the following steps: extracting audio features from an audio segment of a target video; clustering the audio segment according to a time sequence constraint condition based on the audio features to obtain a clustering result; dividing the target video into corresponding multiple video scenes based on the clustering result, determining a scene switching point of the video scene; obtaining a target video frame corresponding to the scene switching point, comparing a perceptual hash of the target video frame with a perceptual hash of a reference video to determine a redundant segment in the target video; and wherein the target video and the reference video have an association relationship. The video redundant segment detection method provided by the application can realize fast and accurate redundant segment detection, effectively improve the video redundant segment detection efficiency and precision, and thus improve the user experience.

Description

Technical Field

[0001] This invention relates to the field of computer multimedia technology, specifically to a method and system for detecting redundant video segments. It also relates to an electronic device and a processor-readable storage medium. Background Technology

[0002] Web video is an important multimedia application, and its transmission currently accounts for a major portion of internet data traffic. Interactive videos (such as TV series, documentaries, and animated episodes) are the most common type of content transmitted. For ordinary users, interactive videos often contain a large number of redundant segments, most of which are included in opening and ending themes, and commercials. For example, each episode of a TV series contains several minutes of redundant segments, such as opening or ending credits, sometimes even accounting for 10% of the episode's length. Viewers typically want to skip these redundant segments for a more continuous viewing experience, while simultaneously saving time and network bandwidth.

[0003] Driven by user demand, mainstream online video content providers today typically mark redundant segments in advance at the video playback bar and offer users an automatic skip function during playback. In fact, the most direct method for marking these redundant segments is manual annotation; however, this incurs huge labor costs for these video content providers and is therefore rarely used in practice. Furthermore, since video content providers often use different video encoding parameters for different videos to achieve an optimal balance between video quality and size, traditional redundancy detection algorithms based on file byte patterns are not suitable for this scenario. To address this, some video websites have attempted to use computer vision methods to detect these redundant segments; however, this method involves training computer vision models, which is extremely costly. Other video websites have reduced the required overhead by decreasing model complexity and only detecting the beginning and end of the video, but this reduces detection accuracy and produces many mislabeled or missed labels. In addition, other video websites have also tried to identify redundant segments using a lighter audio fingerprinting method, which compares the audio spectral features of the video. However, audio-based methods are affected by audio encoding, easily introducing noise and causing the loss of key information, making the actual detection accuracy unstable.

[0004] Therefore, existing methods for detecting redundant video segments through cross-correlation cannot achieve a good balance between detection accuracy and computational and cost requirements. This is the main reason why the automatic skipping of redundant segments offered by current video content providers is not satisfactory to users. To solve this problem, providing a more accurate and efficient solution for detecting redundant video segments has become an urgent challenge. Summary of the Invention

[0005] To address this, the present invention provides a method and system for detecting redundant video segments, thereby solving the shortcomings of existing video redundant segment detection and marking schemes, which have low efficiency and accuracy, resulting in a poor actual user experience.

[0006] In a first aspect, the present invention provides a method for detecting redundant video segments, comprising:

[0007] Extract audio features from audio segments of the target video;

[0008] Based on the audio features, the audio segments are clustered according to the time sequence constraint to obtain the clustering results; based on the clustering results, the target video is divided into multiple corresponding video scenes, and the scene switching points of the video scenes are determined.

[0009] Obtain the target video frame corresponding to the scene switching point, compare the perceptual hash of the target video frame with the perceptual hash of the reference video, and determine the redundant segments in the target video; wherein, there is a correlation between the target video and the reference video.

[0010] Furthermore, before extracting audio features from the audio segments of the target video, the process also includes: determining a reference video from the video data to be detected;

[0011] Determine the reference perceptual hash corresponding to the reference video frame in the reference video, and construct a perceptual hash table based on the reference perceptual hash corresponding to the reference video frame to complete the matching query operation in linear time complexity; wherein, the perceptual hash table contains the reference perceptual hash corresponding to the reference video frame, and the reference video frame contains video frames of redundant segments in the reference video.

[0012] Furthermore, the step of extracting audio features from the audio segments of the target video specifically includes: determining the target video from the video data to be detected;

[0013] The target video is divided into multiple video segments according to a preset time interval, and corresponding audio features are extracted based on the audio segments corresponding to the video segments; the audio features include frequency domain features and time domain features of the audio data.

[0014] Furthermore, the step of clustering the audio segments according to the temporal order constraint based on the audio features to obtain the clustering results specifically includes:

[0015] Based on the audio features, the Ward distance between each pair of adjacent audio segments in the audio segment is determined, and the two adjacent audio segments with the closest Ward distance are clustered and merged to obtain the clustering result.

[0016] Furthermore, before comparing the perceptual hash of the target video frame with the perceptual hash of the reference video to determine redundant segments in the target video, the process further includes:

[0017] Determine the perceptual hash of the target video frame; and model the video scene as a corresponding scene partitioning tree based on the scene switching points of the video scene;

[0018] The step of comparing the perceptual hash of the target video frame with the perceptual hash of the reference video to determine redundant segments in the target video specifically includes:

[0019] Using the perceptual hash of the target video frame as an index, and based on the scene partitioning tree, a reference perceptual hash matching the index is searched in the perceptual hash table of the reference video in a coarse-grained to fine-grained manner. Based on the reference perceptual hash matching the index, redundant segments in the target video are determined.

[0020] Furthermore, the step of dividing the target video into multiple corresponding video scenes based on the clustering results and determining the scene switching points of the video scenes specifically includes:

[0021] Based on the clustering points between two adjacent audio segments in the clustering results, the target video is divided into multiple corresponding video scenes, and the clustering points between two adjacent audio segments are determined as the scene switching points of the video scenes.

[0022] Furthermore, obtaining the target video frame corresponding to the scene switching point specifically includes: determining the number of video frames to be extracted corresponding to the scene switching point;

[0023] Based on the number of video frames extracted, a corresponding number of video frames are obtained from before and after the scene switching point as target video frames.

[0024] Secondly, the present invention also provides a video redundant segment detection system, comprising:

[0025] The audio feature extraction module is used to extract audio features from audio segments of the target video.

[0026] The scene segmentation module is used to cluster the audio segments according to the time sequence constraints based on the audio features to obtain the clustering results; and to divide the target video into multiple corresponding video scenes based on the clustering results to determine the scene switching points of the video scenes.

[0027] The video comparison module is used to obtain the target video frame corresponding to the scene switching point, compare the perceptual hash of the target video frame with the perceptual hash of the reference video, and determine the redundant segments in the target video; wherein, there is a correlation between the target video and the reference video.

[0028] Furthermore, before extracting audio features from the audio segments of the target video, the method further includes: a video frame hash calculation module, which is used to determine a reference video from the video data to be detected; determine the reference perceptual hash corresponding to the reference video frame in the reference video; and construct a perceptual hash table based on the reference perceptual hash corresponding to the reference video frame to complete the matching query operation in linear time complexity; wherein, the perceptual hash table contains the reference perceptual hash corresponding to the reference video frame, and the reference video frame contains video frames of redundant segments in the reference video.

[0029] Furthermore, the audio feature extraction module is specifically used for: determining the target video from the video data to be detected; dividing the target video into multiple video segments according to a preset time interval; and extracting corresponding audio features based on the audio segments corresponding to the video segments; the audio features include frequency domain features and time domain features of the audio data.

[0030] Furthermore, the scene segmentation module is specifically used for:

[0031] Based on the audio features, the Ward distance between each pair of adjacent audio segments in the audio segment is determined, and the two adjacent audio segments with the closest Ward distance are clustered and merged to obtain the clustering result.

[0032] Furthermore, the scene segmentation module is specifically used for:

[0033] Based on the clustering points between two adjacent audio segments in the clustering results, the target video is divided into multiple corresponding video scenes, and the clustering points between two adjacent audio segments are determined as the scene switching points of the video scenes.

[0034] Furthermore, the video comparison module is specifically used to: determine the number of video frames to be extracted corresponding to the scene switching point;

[0035] Based on the number of video frames extracted, a corresponding number of video frames are obtained from before and after the scene switching point as target video frames.

[0036] Furthermore, before comparing the perceptual hash of the target video frame with the perceptual hash of the reference video to determine redundant segments in the target video, the process further includes:

[0037] The video frame hash calculation module is also used to determine the perceptual hash of the target video frame; the scene segmentation module is also used to model the video scene as a corresponding scene segmentation tree according to the scene switching point of the video scene.

[0038] The video comparison module is specifically used for:

[0039] Using the perceptual hash of the target video frame as an index, and based on the scene partitioning tree, a reference perceptual hash matching the index is searched in the perceptual hash table of the reference video in a coarse-grained to fine-grained manner. Based on the reference perceptual hash matching the index, redundant segments in the target video are determined.

[0040] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the video redundant segment detection method as described in any of the above.

[0041] Fourthly, the present invention also provides a processor-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the video redundant segment detection method described in any of the above claims.

[0042] The video redundancy segment detection method provided by this invention extracts audio features from audio segments of a target video, clusters the audio segments according to temporal constraints based on these features to obtain clustering results, divides the target video into multiple corresponding video scenes based on the clustering results, and determines the scene transition points of the video scenes. The method then obtains the target video frames corresponding to the scene transition points and compares the perceptual hash of the target video frames with the perceptual hash of a reference video to identify redundant segments in the target video. This video redundancy segment detection method provides fast and accurate redundancy segment detection, effectively improving the efficiency and accuracy of video redundancy segment detection. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the video redundant segment detection method provided in an embodiment of the present invention;

[0045] Figure 2 This is a complete flowchart of the video redundant segment detection method provided in the embodiments of the present invention;

[0046] Figure 3 This is a time-constrained hierarchical clustering diagram provided in an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of the process of searching for redundant fragments provided in an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the video redundant segment detection system provided in an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] This invention proposes an audio-driven, high-performance method for detecting redundant segments in web-based cross-correlation videos. This method utilizes auxiliary audio information to achieve fast and accurate detection of redundant segments. Video scenes are a crucial concept in video creation, representing a series of logically related, sequentially presented plot events. In fact, redundant opening and closing credits and commercials are often inserted fully into the video content after its creation; therefore, video data consists of one or more complete scenes, rather than suddenly appearing or ending within a single scene. This invention starts with the video scene, using lightweight audio information to coarsely determine the approximate location of scene transitions, and compares only video frames near the scene transition point to quickly and accurately locate the start and end positions of redundant segments in the video data, thus avoiding inefficient processing of all video frames in a single video dataset.

[0052] The following section first describes in detail the embodiments of the video redundant segment detection method described in this invention. For example... Figure 1 The diagram shown is a flowchart of the video redundant segment detection method provided in an embodiment of the present invention. The specific implementation process includes the following steps:

[0053] Step 101: Extract audio features from the audio segments of the target video.

[0054] In this embodiment of the invention, before performing this step, it is necessary to pre-determine reference videos from the video data to be detected and calculate the reference perceptual hashes corresponding to reference video frames in the reference videos. Based on the reference perceptual hashes corresponding to the reference video frames, a perceptual hash table is constructed to complete the matching query operation in linear time complexity. The perceptual hash table contains the reference perceptual hashes corresponding to the reference video frames, and the reference video frames contain video frames of redundant segments in the reference videos. For example, before detecting redundant segments, it is first necessary to arbitrarily select one episode (e.g., the first episode of a TV series) from the entire cross-correlated video set as the reference video, and the remaining videos as target videos (i.e., query videos). The target videos can refer to video data other than the reference videos determined from the video data to be detected. When detecting redundant segments, it is convenient to compare the video frames of each target video and the reference video under the guidance of audio scene switching information, thereby detecting redundant segments. The cross-correlated videos are video data with a correlation relationship, such as a complete TV series or documentary. In this step, the target video can be divided into multiple video segments according to a preset time interval, and corresponding audio features can be extracted based on the audio segments corresponding to the video segments; the audio features include the frequency domain features and time domain features of the audio data.

[0055] In other words, in this embodiment of the invention, all video frames of the reference video are first decoded, and the perceptual hash (i.e., perceptual hash value or perceptual hash feature value) of the decoded video frames is calculated, ultimately outputting a 256-bit perceptual hash for each frame of the reference video. After the 256-bit perceptual hashes of all video frames are generated, a perceptual hash table that can be queried in sublinear time complexity is established using multi-segment index hashing technology. Then, audio features are extracted from the audio data of the target video to facilitate subsequent scene segmentation of the target video. For example, the target video can be divided into video segments with a time interval of 180 milliseconds, and then the frequency domain features and time domain features of the audio data are extracted from the audio segments of these video segments. The frequency domain features mainly include Mel-frequency cepstral coefficients (MFCC) and frequency centroid, while the time domain features mainly include zero-crossing rate and root mean square energy. Since acoustic feature data extracted from audio data is used, this process is very lightweight, yet it can provide a large amount of useful guidance information for the subsequent detection of redundant video segments. Using the acoustic features of video data to segment the scene of video data effectively improves the detection efficiency.

[0056] like Figure 2As shown, for a given cross-correlated video (e.g., a TV series or documentary), this invention first selects one episode as the reference video. The other episodes of this cross-correlated video are used as target videos. Based on the reference video, redundant video segments (i.e., redundant video segments) are searched. After selecting the reference video, this invention first decodes all video frames of the reference video and calculates a perceptual hash for all decoded video frames. Specifically, for a given frame, it is first reduced to a 16×16 grayscale image using fast bilinear interpolation; then, this 16×16 grayscale image undergoes a discrete cosine transform to extract visual features, ultimately obtaining a 16×16 frequency domain feature matrix. Unlike traditional image hashing processes, this invention does not discard high-frequency components in this step, but retains all frequency domain components. This is because the differences between adjacent frames of a video are very small, and this tiny difference will be reflected in the high-frequency components of the discrete cosine transform. If the high-frequency components are discarded, adjacent frames will be difficult to distinguish. Then, based on the following formula, this invention generates a perceptual hash H from the 16×16 frequency domain feature matrix F of the video frame:

[0057]

[0058]

[0059] Where i and j represent the index values ​​of the 16×16 frequency domain feature matrix F or perceptual hash H extracted from the video frame. For example, F(i,j) represents the value of the component in the i-th row and j-th column of the 16×16 frequency domain feature matrix, and H(i,j) represents the value of the component in the i-th row and j-th column of the 16×16 perceptual hash.

[0060] a represents the average value of all frequency domain components of the 16×16 frequency domain characteristic matrix.

[0061] Therefore, the perceptual hash H of a video frame is a 0-1 vector of length 256. After calculating the perceptual hash for each video frame of the reference video, this invention uses multi-segment indexed hashing technology to organize these hash values ​​and construct the perceptual hash, thereby providing sublinear time complexity for subsequent video frame comparison steps.

[0062] Step 102: Based on the audio features, cluster the audio segments according to the time sequence constraint to obtain the clustering results; based on the clustering results, divide the target video into multiple corresponding video scenes and determine the scene switching points of the video scenes.

[0063] In this step, the audio features (i.e., acoustic features) of the extracted audio segments are acquired, and hierarchical clustering constrained by time order is performed on the audio segments based on these audio features to divide the corresponding video scenes and detect possible scene transition points (i.e., scene transition time points) in the target video. Specifically, the Ward distance between any two adjacent audio segments in the audio segment can be determined based on the audio features, and the two adjacent audio segments with the closest Ward distance are clustered and merged to obtain the clustering result. Based on the clustering points between two adjacent audio segments in the clustering result, the target video is divided into multiple corresponding video scenes, and the clustering points between two adjacent audio segments are determined as the scene transition points of the video scenes.

[0064] In practical implementation, for each target video, audio features are first extracted from the video's audio data. Specifically, this step extracts Mel-spectral coefficients, frequency centroids, zero-crossing rates, and root-mean-square energy from the audio data within every 180-millisecond sliding window, with a sliding window step size of 90 milliseconds. That is, there is a 90-millisecond overlap between the 180-millisecond sliding windows, allowing for more granular extraction of acoustic features. This invention extracts the first 13 coefficients of the Mel-spectral coefficients, then combines the frequency centroids, zero-crossing rates, and root-mean-square energy, ultimately obtaining a 16-dimensional feature vector for each 180-millisecond audio segment.

[0065] Time-constrained hierarchical clustering is performed based on all audio features of the target video to obtain scene segmentation information. For example... Figure 3 As shown, this illustrates the process of performing time-constrained hierarchical clustering on each audio segment. At the beginning of this process, all audio segments are in an unclustered state. In each iteration of the algorithm, this step calculates the Ward distance between every two adjacent audio segments (or segments already clustered in previous iterations) and selects the two adjacent audio segments with the closest Ward distance for cluster merging. For two adjacent audio segment sets A and B, the Ward distance is calculated as follows:

[0066]

[0067] Here, μ represents the centroid of all audio feature vectors in the set; x represents a specific audio segment in the set. Therefore, each cluster point between adjacent audio segments can be considered a scene transition point. For example... Figure 3As shown, the transition between audio clips 6 and 7 can be considered a scene switch, dividing the video in two. Further subdivision allows for the division of these scenes into sub-scenes; for example, scenes 1-6 can be subdivided into sub-scenes 1-3 and 4-6, and scenes 7-16 can be subdivided into sub-scenes 7-11 and 12-16. This partitioning process models the video scenes as a scene partitioning tree.

[0068] Step 103: Obtain the target video frame corresponding to the scene switching point, compare the perceptual hash of the target video frame with the perceptual hash of the reference video, and determine the redundant segments in the target video; wherein, there is a correlation between the target video and the reference video.

[0069] In this step, the number of video frames to be extracted corresponding to the scene switching point is first determined. Based on the number of video frames to be extracted, a corresponding number of video frames are obtained before and after the scene switching point as target video frames. By traversing all target videos, for each target video, the video frame closest to the scene switching time point (obtained by audio feature clustering in the previous step) is extracted step by step according to the number of video frames extracted as the target video frame. The target video frame is then decoded and a 256-bit perceptual hash is calculated.

[0070] It should be noted that before comparing the perceptual hash of the target video frame with the perceptual hash of the reference video to determine redundant segments in the target video, the perceptual hash of the target video frame needs to be determined in advance, and the video scene needs to be modeled as a corresponding scene segmentation tree based on the scene switching points of the video scene. Then, using the perceptual hash of the target video frame as an index, and based on the scene segmentation tree, a reference perceptual hash matching the index is searched in the perceptual hash table of the reference video in a coarse-grained to fine-grained manner. Based on the reference perceptual hash matching the index, redundant segments in the target video are determined. That is, in a coarse-grained to fine-grained manner, video frames decoded near these scene switching time points are searched in the perceptual hash table of the reference video to ultimately determine the boundary position of redundant segments, thereby avoiding unnecessary comparisons of video frames within the video scene and greatly improving detection efficiency.

[0071] Based on the scene segmentation tree generated by hierarchical clustering of the target video, this step can search for redundant video segments in the target video relative to the reference video in a top-down, coarse-grained to fine-grained manner. For example... Figure 4As shown, starting from the root node of the scene segmentation tree, each node divides a video segment into two scenes. This step extracts video frames (generally the most recent n keyframes, as they can be decoded independently of other video frames, where n represents a specific number of video keyframes) from the scene transition points of the target video, calculates the perceptual hash value, and searches it in the multi-segment index hash table of the reference video to determine whether the reference video contains the same video frame (e.g., ...). Figure 4 If ① exists, it indicates that there may be redundant segments in the target video at this point. Therefore, this frame is marked, and a redundancy flag is set (indicating that the start or end of a redundant segment has been found, and another endpoint needs to be found to define the redundant segment). The search continues recursively into the subtrees of the scene partitioning tree (e.g., ...). Figure 4 ②); If the reference video does not contain a video frame at the target video scene transition point (e.g. Figure 4 In step ②), based on the subtree of the scene partitioning tree, the search and comparison are performed on video frames closer to the redundant marker in the target video, narrowing the search range until another identical frame is found, so as to finally determine the position of the other end of the redundant segment (e.g., ...). Figure 4 (③) Afterwards, the redundant marker bits are removed, and the current scene's tree nodes are no longer recursively searched downwards. From Figure 4 As can be seen, this process skips the vast majority of video frames, only decoding and comparing video frames at scene transition points, thus saving a significant portion of video data processing overhead. It should be noted that during the search process, if no redundant video frames are found, the algorithm will recursively search downwards until it converges to a certain granularity where scene segmentation can no longer be performed; that is, the duration of the video segment represented by the node of the current scene segmentation tree is less than a certain time interval or time threshold, such as 1 second. When no more scene segmentation tree nodes can be searched, the redundancy detection process for this set of target videos ends, and the start and end positions of the redundant segments are output. Then, step 101 is returned to perform the search and comparison for the next set, until all redundant segments of the target videos have been detected, and the execution ends.

[0072] This invention innovatively uses acoustically segmented scene information to guide the comparison of redundant segments in cross-correlation videos, and further accelerates the comparison process using multi-segment index hashing technology. Compared to traditional computer vision-based methods, this method significantly reduces computational overhead, and compared to traditional audio fingerprint-based methods, it greatly improves comparison accuracy. The actual accuracy for detecting redundant segments reached 98%, and the recall rate reached 93%, representing an average speed improvement of 9.3 times compared to computer vision methods. Specifically, it can be used in the backend of video websites to help video content providers pre-label redundant segments in videos before publishing. It can also be seamlessly integrated as a browser plugin, enabling users to detect redundant content in real time and display corresponding prompts when watching cross-correlation videos that have not yet been labeled with redundant segments.

[0073] The video redundant segment detection method described in this invention extracts audio features from audio segments of a target video, clusters the audio segments according to temporal constraints based on these features to obtain clustering results, and divides the target video into multiple corresponding video scenes based on the clustering results, determining the scene transition points of each video scene. The method then obtains the target video frames corresponding to the scene transition points and compares the perceptual hash of the target video frames with the perceptual hash of a reference video to identify redundant segments in the target video. The video redundant segment detection method provided by this invention can achieve fast and accurate redundant segment detection, effectively improving the efficiency and accuracy of video redundant segment detection.

[0074] Corresponding to the video redundancy segment detection method provided above, this invention also provides a video redundancy segment detection system. Since the embodiments of this system are similar to the method embodiments described above, the description is relatively simple. For relevant details, please refer to the description in the method embodiment section above. The embodiments of the video redundancy segment detection system described below are merely illustrative. Please refer to... Figure 5 As shown, it is a structural schematic diagram of a video redundant segment detection system provided in an embodiment of the present invention.

[0075] The video redundant segment detection system of the present invention specifically includes the following parts:

[0076] The audio feature extraction module 501 is used to extract audio features from audio segments of the target video.

[0077] The scene segmentation module 502 is used to cluster the audio segments according to the time sequence constraints based on the audio features to obtain the clustering results; and to divide the target video into multiple corresponding video scenes based on the clustering results to determine the scene switching points of the video scenes.

[0078] The video comparison module 502 is used to obtain the target video frame corresponding to the scene switching point, compare the perceptual hash of the target video frame with the perceptual hash of the reference video, and determine the redundant segments in the target video; wherein, there is a correlation between the target video and the reference video.

[0079] Furthermore, before extracting audio features from the audio segments of the target video, the method further includes: a video frame hash calculation module, which is used to determine a reference video from the video data to be detected; determine the reference perceptual hash corresponding to the reference video frame in the reference video; and construct a perceptual hash table based on the reference perceptual hash corresponding to the reference video frame to complete the matching query operation in linear time complexity; wherein, the perceptual hash table contains the reference perceptual hash corresponding to the reference video frame, and the reference video frame contains video frames of redundant segments in the reference video.

[0080] Furthermore, the audio feature extraction module is specifically used for: determining the target video from the video data to be detected; dividing the target video into multiple video segments according to a preset time interval; and extracting corresponding audio features based on the audio segments corresponding to the video segments; the audio features include frequency domain features and time domain features of the audio data.

[0081] Furthermore, the scene segmentation module is specifically used for:

[0082] Based on the audio features, the Ward distance between each pair of adjacent audio segments in the audio segment is determined, and the two adjacent audio segments with the closest Ward distance are clustered and merged to obtain the clustering result.

[0083] Furthermore, the scene segmentation module is specifically used for:

[0084] Based on the clustering points between two adjacent audio segments in the clustering results, the target video is divided into multiple corresponding video scenes, and the clustering points between two adjacent audio segments are determined as the scene switching points of the video scenes.

[0085] Furthermore, the video comparison module is specifically used to: determine the number of video frames to be extracted corresponding to the scene switching point;

[0086] Based on the number of video frames extracted, a corresponding number of video frames are obtained from before and after the scene switching point as target video frames.

[0087] Furthermore, before comparing the perceptual hash of the target video frame with the perceptual hash of the reference video to determine redundant segments in the target video, the process further includes:

[0088] The video frame hash calculation module is also used to determine the perceptual hash of the target video frame; the scene segmentation module is also used to model the video scene as a corresponding scene segmentation tree according to the scene switching point of the video scene.

[0089] The video comparison module is specifically used for:

[0090] Using the perceptual hash of the target video frame as an index, and based on the scene partitioning tree, a reference perceptual hash matching the index is searched in the perceptual hash table of the reference video in a coarse-grained to fine-grained manner. Based on the reference perceptual hash matching the index, redundant segments in the target video are determined.

[0091] The video redundant segment detection system described in this invention extracts audio features from audio segments of a target video, clusters the audio segments according to temporal constraints based on these features to obtain clustering results, and divides the target video into multiple corresponding video scenes based on the clustering results, determining the scene transition points of each video scene. The system then acquires the target video frames corresponding to the scene transition points and compares the perceptual hash of the target video frames with the perceptual hash of a reference video to identify redundant segments in the target video. The video redundant segment detection system provided by this invention can achieve fast and accurate redundant segment detection, effectively improving the efficiency and accuracy of video redundant segment detection.

[0092] Corresponding to the video redundancy segment detection method or video redundancy segment detection method provided above, the present invention also provides an electronic device. Since the embodiments of this electronic device are similar to the above method embodiments, the description is relatively simple. For relevant details, please refer to the description in the above method embodiment section. The electronic device described below is merely illustrative. Figure 6The diagram shows a physical structure of an electronic device disclosed in an embodiment of the present invention. The electronic device may include a processor 601, a memory 602, and a communication bus 603. The processor 601 and the memory 602 communicate with each other via the communication bus 603 and communicate with external systems via a communication interface 604. The processor 601 can call logical instructions in the memory 602 to execute a video redundancy segment detection method. This method includes: extracting audio features from audio segments of a target video; clustering the audio segments according to temporal constraints based on the audio features to obtain clustering results; dividing the target video into multiple corresponding video scenes based on the clustering results and determining scene switching points of the video scenes; obtaining the target video frame corresponding to the scene switching point; comparing the perceptual hash of the target video frame with the perceptual hash of a reference video to determine redundant segments in the target video; wherein there is a correlation between the target video and the reference video.

[0093] Furthermore, the logical instructions in the aforementioned memory 602 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as memory chips, USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] On the other hand, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a processor-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the video redundant segment detection method provided in the above-described method embodiments. The method includes: extracting audio features from audio segments of a target video; clustering the audio segments according to temporal order constraints based on the audio features to obtain clustering results; dividing the target video into multiple corresponding video scenes based on the clustering results, and determining scene switching points of the video scenes; obtaining target video frames corresponding to the scene switching points, and comparing the perceptual hash of the target video frame with the perceptual hash of a reference video to determine redundant segments in the target video; wherein, there is a correlation between the target video and the reference video.

[0095] In another aspect, embodiments of the present invention also provide a processor-readable storage medium storing a computer program that, when executed by a processor, implements the video redundancy segment detection method provided in the above embodiments. The method includes: extracting audio features from audio segments of a target video; clustering the audio segments according to temporal constraints based on the audio features to obtain clustering results; dividing the target video into multiple corresponding video scenes based on the clustering results, and determining scene switching points of the video scenes; obtaining target video frames corresponding to the scene switching points, and comparing the perceptual hash of the target video frames with the perceptual hash of a reference video to determine redundant segments in the target video; wherein there is a correlation between the target video and the reference video.

[0096] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0097] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting redundant segments in video, characterized in that, include: A reference video and a target video are determined from the video data to be detected; wherein the reference video is any set of the video data to be detected, and the target video is video data other than the reference video determined from the video data to be detected; Extract audio features from the audio segments of the target video; Based on the audio features, the audio segments are subjected to multi-round iterative hierarchical clustering according to the temporal order constraint to obtain the clustering results; wherein, the following operations are performed in each round of hierarchical clustering: Based on the audio features, the Ward distance between each pair of adjacent audio segments in the audio segment is determined, and the two adjacent audio segments with the closest Ward distance are clustered and merged to obtain the clustering result of the current round. Based on the clustering results, the target video is divided into multiple corresponding video scenes, and the scene switching points of the video scenes are determined, including: Based on the clustering points between two adjacent audio segments in the clustering results, the target video is divided into multiple corresponding video scenes, and the clustering points between two adjacent audio segments are determined as the scene switching points of the video scenes. Obtain the target video frame corresponding to the scene switching point; Determine the perceptual hash of the target video frame; and model the video scene as a corresponding scene partitioning tree based on the scene switching points of the video scene; By comparing the perceptual hash of the target video frame with the perceptual hash of the reference video, redundant segments in the target video are identified, including: Starting from the root node of the scene segmentation tree, each node divides a video segment into two scenes. Video frames are extracted from the scene switching point of the target video, a perceptual hash value is calculated, and the result is searched in the multi-segment index hash table of the reference video to determine if there are identical video frames in the reference video. If there are, a redundancy flag is set for the video frame, and the search continues recursively into the subtrees of the scene segmentation tree. If there are no video frames at the scene switching point of the target video in the reference video, the search and comparison are performed on the video frames in the target video closer to the redundancy flag based on the subtree of the scene segmentation tree, narrowing the search range until another identical video frame is found, thus finally determining the position of the other end of the redundant segment. After that, the redundancy flag is canceled, and the current scene segmentation tree node is no longer searched recursively downwards. When there are no more scene segmentation tree nodes to search, the redundancy detection process for this set of target videos ends, and the start and end positions of the redundant segment are output.

2. The video redundant segment detection method according to claim 1, characterized in that, Before extracting audio features from the audio segments of the target video, the process also includes: Determine the reference perceptual hash corresponding to the reference video frame in the reference video, and construct a perceptual hash table based on the reference perceptual hash corresponding to the reference video frame to complete the matching query operation in linear time complexity; wherein, the perceptual hash table contains the reference perceptual hash corresponding to the reference video frame, and the reference video frame contains video frames of redundant segments in the reference video.

3. The video redundant segment detection method according to claim 1, characterized in that, The extraction of audio features from audio segments of the target video specifically includes: The target video is divided into multiple video segments according to a preset time interval, and corresponding audio features are extracted based on the audio segments corresponding to the video segments; the audio features include frequency domain features and time domain features of the audio data.

4. The video redundant segment detection method according to claim 1, characterized in that, The step of obtaining the target video frame corresponding to the scene switching point specifically includes: Determine the number of video frames to be extracted corresponding to the scene switching point; Based on the number of video frames extracted, a corresponding number of video frames are obtained from before and after the scene switching point as target video frames.

5. A video redundant segment detection system, characterized in that, include: An audio feature extraction module is used to determine a reference video and a target video from the video data to be detected; wherein the reference video is any set of the video data to be detected, and the target video is video data other than the reference video determined from the video data to be detected; and Extract audio features from the audio segments of the target video; The scene segmentation module is used to perform multi-round iterative hierarchical clustering of the audio segments based on the audio features and according to the temporal order constraint to obtain the clustering results; wherein, the following operations are performed in each round of hierarchical clustering: Based on the audio features, the Ward distance between each pair of adjacent audio segments in the audio segment is determined, and the two adjacent audio segments with the closest Ward distance are clustered and merged to obtain the clustering result of the current round. Based on the clustering results, the target video is divided into multiple corresponding video scenes, and the scene switching points of the video scenes are determined, including: Based on the clustering points between two adjacent audio segments in the clustering results, the target video is divided into multiple corresponding video scenes, and the clustering points between two adjacent audio segments are determined as the scene switching points of the video scenes. The video comparison module is used to obtain the target video frame corresponding to the scene switching point; Determine the perceptual hash of the target video frame; and model the video scene as a corresponding scene partitioning tree based on the scene switching points of the video scene; By comparing the perceptual hash of the target video frame with the perceptual hash of the reference video, redundant segments in the target video are identified, including: Starting from the root node of the scene segmentation tree, each node divides a video segment into two scenes. Video frames are extracted from the scene switching point of the target video, a perceptual hash value is calculated, and the result is searched in the multi-segment index hash table of the reference video to determine if there are identical video frames in the reference video. If there are, a redundancy flag is set for the video frame, and the search continues recursively into the subtrees of the scene segmentation tree. If there are no video frames at the scene switching point of the target video in the reference video, the search and comparison are performed on the video frames in the target video closer to the redundancy flag based on the subtree of the scene segmentation tree, narrowing the search range until another identical video frame is found, thus finally determining the position of the other end of the redundant segment. After that, the redundancy flag is canceled, and the current scene segmentation tree node is no longer searched recursively downwards. When there are no more scene segmentation tree nodes to search, the redundancy detection process for this set of target videos ends, and the start and end positions of the redundant segment are output.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the video redundant segment detection method as described in any one of claims 1 to 4.

7. A processor-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the video redundant segment detection method as described in any one of claims 1 to 4.

Images