Video version comparison method and related device

Abstract

The invention discloses a video version comparison method and a related device, and the method comprises the steps: extracting a plurality of video frames from dynamic video segments of a first video and a second video for multiple times, and carrying out the calculation of the first similarity information of the dynamic video segments of the first video and the second video based on the extracted video frames, so as to carry out the rough comparison of the first video and the second video; calculating second similarity information between missing video segments in the first video and the second video based on starting point information and terminal point information in the first similarity information so as to carry out fine comparison on the first video and the second video; and determining the same video segment between the first video and the second video based on the first similarity information and the second similarity information. Compared with the prior art, the video version comparison is automatically carried out, that is, the same video segment between the first video and the second video is found, the comparison process is divided into two stages of coarse comparison and fine comparison, manual participation is not needed, the efficiency is high, and the comparison result is accurate and not prone to errors.

Description

Technical Field

[0001] The present invention relates to the field of videos, and in particular to a video version comparison method and related devices. Background Art

[0002] As technology continues to improve, people's entertainment methods are also changing. Currently, a large number of videos are uploaded to the Internet every day, and the content of the videos is rich and colorful. The production process of these videos is complicated and may involve different versions of videos. At present, different versions of videos are manually reviewed, which has the problems of low efficiency and duplication of work. Summary of the invention

[0003] In view of the above problems, the present invention provides a video version comparison method and related devices that overcome the above problems or at least partially solve the above problems.

[0004] In a first aspect, a video version comparison method includes:

[0005] Extracting multiple video frames from the dynamic video segments of the first video and the second video multiple times, and calculating first similarity information of the dynamic video segments of the first video and the second video based on the extracted video frames, so as to roughly compare the first video and the second video, wherein each time a video segment is respectively selected from the first video and the second video as the dynamic video segment of the corresponding video, and the first similarity information includes starting point information, end point information and a first similarity of the corresponding dynamic video segment, and the first similarity represents the similarity between the dynamic video segment of the first video and the dynamic video segment of the second video;

[0006] Based on the starting point information and the ending point information in the first similarity information, second similarity information between the missing video segments in the first video and the second video is calculated to perform a detailed comparison between the first video and the second video, wherein one piece of the second similarity information includes the starting point information, the ending point information and the second similarity of the corresponding missing video segment, the second similarity represents the similarity between the missing video segment of the first video and the missing video segment of the second video, and the missing video segment is a video segment that is not involved in the process of calculating the first similarity information multiple times;

[0007] Based on the first similarity information and the second similarity information, identical video segments between the first video and the second video are determined.

[0008] Optionally, in some optional implementations, extracting multiple video frames from the dynamic video segments of the first video and the second video for multiple times, and calculating first similarity information of the dynamic video segments of the first video and the second video based on the extracted video frames includes:

[0009] For any extraction of video frames, the following steps are performed: extracting multiple video frames from the dynamic video segment of the first video to obtain multiple first video frame groups, and extracting multiple video frames from the dynamic video segment of the second video to obtain multiple second video frame groups, wherein each extraction of video frames corresponds to one first video frame group and one second video frame group, and a video start time of a dynamic video segment extracted the second time is later than a video start time of a dynamic video segment extracted the previous time;

[0010] Corresponding first similarity information is calculated based on the extracted first video frame group and the second video frame group, wherein one video frame group corresponds to one first similarity information.

[0011] Optionally, in some optional implementations, the calculating corresponding first similarity information based on the extracted first video frame group and the second video frame group includes:

[0012] Using a pre-trained picture model, respectively extracting features from the first video frame group and the second video frame group to obtain corresponding first feature matrices and second feature matrices, wherein one first video frame group corresponds to one first feature matrix, and one second video frame group corresponds to one second feature matrix;

[0013] Multiply the transposed matrix of the first feature matrix by the transposed matrix of the second feature matrix to obtain a corresponding similarity matrix;

[0014] The similarity matrix is ​​analyzed to obtain video timestamps of the same video segments of the first video and the second video.

[0015] Optionally, in some optional implementations, analyzing the similarity matrix to obtain video timestamps of the same video segments of the first video and the second video includes:

[0016] Analyzing the similarity matrix to obtain start point information and end point information of the same video segment, wherein the same video segment is a video segment whose similarity is greater than a preset similarity threshold;

[0017] Based on the start point information and the end point information of the same video segment, a corresponding video timestamp is obtained as one of the first similarity information.

[0018] Optionally, in some optional implementations, the calculating second similarity information between the missing video segments in the first video and the second video based on the starting point information and the ending point information in the first similarity information to perform a detailed comparison on the first video and the second video includes:

[0019] Determine the starting point information and the ending point information of the missing video segments in the first video and the second video based on the starting point information and the ending point information in the first similarity information;

[0020] Based on the start point information and the end point information of the missing video segment, extracting the corresponding missing video segment from the first video and the second video;

[0021] A plurality of video frames are respectively extracted from the missing video segments in the first video and the second video, and second similarity information between the missing video segments is calculated based on the extracted video frames.

[0022] Optionally, in some optional implementations, extracting a plurality of video frames from the missing video segments in the first video and the second video respectively, and calculating the second similarity information between the missing video segments based on the extracted video frames, includes:

[0023] For any extraction of video frames, the following steps are performed: extracting multiple video frames from the missing video segment of the first video to obtain multiple third video frame groups, and extracting multiple video frames from the missing video segment of the second video to obtain multiple fourth video frame groups, wherein each extraction of video frames corresponds to one third video frame group and one fourth video frame group, and the video start time of the dynamic video segment extracted the latter time is later than the video start time of the missing video segment extracted the previous time;

[0024] Using a pre-trained picture model, respectively extracting features from the third video frame group and the fourth video frame group to obtain corresponding third feature matrices and fourth feature matrices, wherein one third video frame group corresponds to one third feature matrix, and one fourth video frame group corresponds to one fourth feature matrix;

[0025] Multiply the transposed matrix of the third characteristic matrix by the transposed matrix of the fourth characteristic matrix to obtain a corresponding similarity matrix;

[0026] The similarity matrix is ​​analyzed to obtain a video timestamp of a same video segment of the third video and the fourth video as one item of the second similarity information.

[0027] Optionally, in some optional implementations, determining the same video segment between the first video and the second video based on the first similarity information and the second similarity information includes:

[0028] Based on the start point information and the end point information in the first similarity information and the second similarity information, the corresponding video segments are merged to obtain the final identical video segments, wherein during the merging process, repeated time information is superimposed and covered, and non-repeated time information is connected to each other in chronological order.

[0029] Optionally, in some optional implementations, after determining the same video segments between the first video and the second video based on the first similarity information and the second similarity information, the method further includes:

[0030] Extend the end point information of the same video segment backward or move the start point information of the same video segment forward to obtain a corresponding new video segment;

[0031] Calculating the similarity of a new video segment between the first video and the second video;

[0032] If the similarity of the new video segment between the first video and the second video is greater than a preset threshold, the corresponding identical video segment is updated to be the new video segment.

[0033] In a second aspect, a video version comparison device includes: a coarse comparison unit, a fine comparison unit, and a result combing unit;

[0034] The rough comparison unit is used to extract multiple video frames from the dynamic video segments of the first video and the second video for multiple times, and calculate first similarity information of the dynamic video segments of the first video and the second video based on the extracted video frames, so as to perform a rough comparison on the first video and the second video, wherein each time a video segment is selected from the first video and the second video respectively as the dynamic video segment of the corresponding video, and the first similarity information includes the starting point information, the ending point information and the first similarity of the corresponding dynamic video segment, and the first similarity represents the similarity between the dynamic video segment of the first video and the dynamic video segment of the second video;

[0035] The detailed comparison unit is used to calculate second similarity information between missing video segments in the first video and the second video based on the starting point information and the ending point information in the first similarity information, so as to perform detailed comparison on the first video and the second video, wherein one piece of the second similarity information includes the starting point information, the ending point information and the second similarity of the corresponding missing video segment, the second similarity represents the similarity between the missing video segment of the first video and the missing video segment of the second video, and the missing video segment is a video segment that is not involved in the process of calculating the first similarity information multiple times;

[0036] The result combing unit is configured to determine the same video segments between the first video and the second video based on the first similarity information and the second similarity information.

[0037] In a third aspect, an electronic device comprises at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other via the bus; and the processor is used to call program instructions in the memory to execute any one of the video version comparison methods described.

[0038] By means of the above technical scheme, a video version comparison method and related devices provided by the present invention can extract multiple video frames from the dynamic video segments of a first video and a second video for multiple times, and calculate the first similarity information of the dynamic video segments of the first video and the second video based on the extracted video frames, so as to roughly compare the first video and the second video, wherein each time a video segment is selected from the first video and the second video respectively as the dynamic video segment of the corresponding video, and the first similarity information includes the starting point information, the end point information and the first similarity of the corresponding dynamic video segment, and the first similarity represents the similarity between the dynamic video segment of the first video and the dynamic video segment of the second video. similarity; based on the starting point information and the end point information in the first similarity information, calculate the second similarity information between the missing video segments in the first video and the second video, so as to make a detailed comparison between the first video and the second video, wherein the second similarity information includes the starting point information, the end point information and the second similarity of the corresponding missing video segment, and the second similarity represents the similarity between the missing video segment of the first video and the missing video segment of the second video, and the missing video segment is a video segment that is not involved in the process of calculating the first similarity information multiple times; based on the first similarity information and the second similarity information, determine the same video segment between the first video and the second video. It can be seen from this that the present invention automatically compares video versions, that is, finds the same video segment between the first video and the second video, and the comparison process is divided into two stages: coarse comparison and fine comparison. Not only does it not require manual participation, it is highly efficient and the comparison results are more accurate and less prone to errors.

[0039] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0041] Figure 1 A flow chart of a first video version comparison method provided by the present invention is shown;

[0042] Figure 2 A flow chart of a second video version comparison method provided by the present invention is shown;

[0043] Figure 3 A schematic structural diagram of a video version comparison device provided by the present invention is shown;

[0044] Figure 4 A structural schematic diagram of an electronic device provided by the present invention is shown. DETAILED DESCRIPTION

[0045] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0046] like Figure 1 As shown, the present invention provides a video version comparison method, including: S100, S200 and S300;

[0047] S100, extracting multiple video frames from the dynamic video segments of the first video and the second video multiple times, and calculating first similarity information of the dynamic video segments of the first video and the second video based on the extracted video frames, so as to roughly compare the first video and the second video, wherein each time a video segment is selected from the first video and the second video respectively as the dynamic video segment of the corresponding video, and the first similarity information includes starting point information, end point information and a first similarity of the corresponding dynamic video segment, and the first similarity represents the similarity between the dynamic video segment of the first video and the dynamic video segment of the second video;

[0048] Optionally, the first video and the second video mentioned in the present invention can be two different versions of videos, or can be the same version of videos. The present invention can compare and find the same segments between the two videos, so as to determine whether the two videos are the same version of videos. At the same time, the present invention can also determine the difference between the two videos through comparison, so as to specifically review the difference content, reduce the repeated review of repeated content, and improve the review efficiency, which is not limited by the present invention.

[0049] It should be noted that: when the similarity between two videos reaches 100% or reaches a certain preset threshold, it can be considered that the two videos are videos of the same version, otherwise, it means that the two videos are videos of different versions, and the present invention does not impose any limitation on this.

[0050] Optionally, S100 of the present invention is a repeatedly executed cyclic process. That is, the video frames are extracted multiple times, and the first video and the second video are extracted each time. The dynamic video segments extracted from the adjacent extracted video frames are in a step relationship from the time axis. For example, assuming that the duration of the first video and the second video is one hour and five minutes, and the video frames extracted by the present invention are extracted from the 10-minute dynamic video segment each time, then the present invention can execute S100 in the following manner:

[0051] First extraction: extract 600 frames of pictures from the video segment 00:00-00:10 of the first video (specifically, one frame can be extracted per second), and extract 600 frames of pictures from the video segment 00:00-00:10 of the second video (similarly, one frame can be extracted per second); then calculate a first similarity information based on the extracted 600 frames of pictures of the first video and the 600 frames of pictures of the second video; the similarity information is assumed to be 00:00-00:08 of the first video and 00:02-00:10 of the second video, and the similarity is 0.98.

[0052] Second extraction: extract 600 frames of images from the video segment 00:08-00:18 of the first video (specifically, one frame can be extracted per second), and extract 600 frames of images from the video segment 00:10-00:20 of the second video (similarly, one frame can be extracted per second); then calculate a first similarity information based on the extracted 600 frames of images of the first video and the 600 frames of images of the second video; the similarity information is assumed to be 00:08-00:17 of the first video and 00:11-00:20 of the second video, and the similarity is 0.97.

[0053] The third extraction: extract 600 frames of pictures from the video segment 00:17-00:27 of the first video (specifically, one frame can be extracted per second), and extract 600 frames of pictures from the video segment 00:20-00:30 of the second video (similarly, one frame can be extracted per second); then calculate a first similarity information based on the extracted 600 frames of pictures of the first video and the 600 frames of pictures of the second video; the similarity information is assumed to be 00:17-00:27 of the first video and 00:20-00:30 of the second video, and the similarity is 0.87.

[0054] This cycle repeats until the last extraction: 480 frames (8min) are extracted from the 00:27-00:35 video segment of the first video. 60) pictures (specifically, one frame can be extracted every second), and 300 frames (5min) are extracted from the 00:30-00:35 video segment of the second video. 60) pictures (similarly, one frame can also be extracted per second); then a first similarity information is calculated based on the extracted 480 frames of pictures of the first video and 300 frames of pictures of the second video. The similarity information is assumed to be 00:30-00:35 of the first video and 00:30-00:35 of the second video, and the similarity is 0.98.

[0055] That is, in some optional embodiments, the S100 includes: S110 and S120;

[0056] S110, for any extraction of video frames, executing: extracting multiple video frames from the dynamic video segment of the first video to obtain multiple first video frame groups, and extracting multiple video frames from the dynamic video segment of the second video to obtain multiple second video frame groups, one extraction of video frames corresponds to one first video frame group and one second video frame group, and the video start time of the dynamic video segment extracted the latter time is later than the video start time of the dynamic video segment extracted the previous time;

[0057] S120. Calculate corresponding first similarity information based on the extracted first video frame group and the second video frame group, wherein one video frame group corresponds to one first similarity information.

[0058] Optionally, the present invention does not impose any specific limitation on the method for calculating the first similarity information, and any feasible method falls within the protection scope of the present invention.

[0059] For example, in some optional embodiments, the S120 includes: S121, S122 and S123;

[0060] S121, using a pre-trained picture model to perform feature extraction on the first video frame group and the second video frame group respectively, to obtain corresponding first feature matrices and second feature matrices, wherein one first video frame group corresponds to one first feature matrix, and one second video frame group corresponds to one second feature matrix;

[0061] S122, performing a dot multiplication of the transposed matrix of the first feature matrix and the transposed matrix of the second feature matrix to obtain a corresponding similarity matrix;

[0062] S123: Analyze the similarity matrix to obtain video timestamps of the same video segments of the first video and the second video.

[0063] Optionally, in some optional embodiments, the S123 includes: step 1.1 and step 1.2;

[0064] Step 1.1, analyzing the similarity matrix to obtain the start point information and the end point information of the same video segment, wherein the same video segment is a video segment whose similarity is greater than a preset similarity threshold;

[0065] Step 1.2: Based on the start point information and the end point information of the same video segment, obtain a corresponding video timestamp as one of the first similarity information.

[0066] A complete execution example of S100 is provided below to further illustrate the solution of the present invention. In S100, the video frames are extracted and compared according to the rule of one frame per second. That is, one frame is extracted every 25 frames (the step size can be specified according to the situation) (generally, 25 frames per second is more common for videos). At the same time, for video segmentation processing, this stage is a cyclic iterative process, and only 10 minutes of video clips are taken from the first video (also recorded as videoA) and the second video (also recorded as videoB) each time. The cyclic process is described in detail below.

[0067] Loop initialization: let the start time of the current videoA be startA=0, the start time of the current videoB be startB=0, let the start time of the last videoA be last_startA=0, let the number of times the starting point moves backward be backward_time=0, let the final result list be match_result=[], and let the sampling step be stride=25.

[0068] Loop body:

[0069] F1. Determine whether startA is less than or equal to last_startA. If startA is less than or equal to last_startA, it means that this loop has not iterated forward, and backward_time increases by 1. If startA is greater than last_startA, it means that this loop has iterated forward normally, and backward_time is reset to 0. It should be noted that backward_time is a counter that has not stepped normally. If it has stepped normally, it will always be 0; if it has not stepped normally for more than 3 times, it can be forced to step later.

[0070] F2, first: videoA from startA to startA+600, take a frame every stride frame, a total of 10 60 = 600 frames of images. Use the image model to extract features from these 600 frames of images, and obtain a feature matrix matA with a shape of (600, 512). Do the same for videoB to obtain matB. It should be noted that if the video is less than 10 minutes, just take the end of the video.

[0071] Then: multiply the transposed matrices of matA and matB to obtain a similarity matrix sim_mat of shape (600, 600). (If the duration of a video segment is not enough, a similarity matrix sim_mat of shape (N, M) is obtained.

[0072] Then: Analyze sim_mat to get the comparison results of the two videos in the current loop (including two lists, bbox_result and scores). Each element in the bbox_result list means the time parameter of the same video segment, and the scores list represents the average score list of each pair of video segments. Assume that bbox_result is [(0, 0, 80, 80), (80, 100, 380, 400)], (0, 0, 80, 80) corresponds to the time parameters of the first pair of video clips (starting point in videoA, starting point in video B, end point in video A, end point in video B); (80, 100, 380, 400) corresponds to the time parameters of the second pair of video clips (starting point in videoA, starting point in video B, end point in video A, end point in video B); Assume that scores is [0.98, 0.976], which corresponds to (similarity between the first pair of video clips, similarity between the second pair of video clips). It should be noted that the similarity score range is 0-1. The larger the score, the more similar it is. A score of 1 indicates that they are exactly the same.

[0073] Finally: add the corresponding starting point to each element in bbox_result (for example, the current coarse comparison loop stage starts searching for the same segment from startA(100) and startB(120). The results found are [(0, 0, 80, 80), (80, 100, 380, 400)], then the actual result should be [(0+100, 0+120, 80+100, 80+120), (80+100, 100+120, 380+100, 400+120)]) to get the real timestamp of the same segment. For example, the video segment of the current loop starts from startA and startB.

[0074] F3. If bbox_result is empty, startA is assigned to startA+600, startB is assigned to startB+600, and the subsequent steps are skipped to start the next cycle. It should be noted that the current rough comparison time range is from startA to startA+600 and from startB to startB+600. If bbox_result is empty, it means that there are no identical segments in the current comparison time range, so you can step into the comparison range and compare the video segments corresponding to the next time period.

[0075] F4, traverse bbox_result and scores. Process the bbox and score of the same video segment in chronological order. If the score is greater than a threshold (for example, 0.95), add the bbox and score to the final result list match_result.

[0076] F5. If backward_time is greater than 2, startA and startB are assigned as the end points of the last pair of video clips in bbox_result, and this loop ends and the next loop starts.

[0077] F6. Traverse bbox_result and scores again. Set the latest starting point of videoA to new_startA=startA, and the latest starting point of videoB to new_startB=startB. Process the bbox and score of the same video segment in chronological order.

[0078] If the score is greater than the threshold, new_startA and new_startB are set as the end point of the current bbox. It should be noted that if the same video segment is not found, the search starts directly from the next segment. If the same video segment that meets the threshold is found here, the end point of the latest same video segment that meets the threshold is set as the starting point of the next match. This method can be more complete on the one hand, and on the other hand, if the current same segment found is (0, 500, 50, 550), the next comparison starts from startA=50 and startB=550, because if they all start from 600, many segments will be missed.

[0079] If the score is less than the threshold, the offset shfitB of the current same fragment is calculated. If the offset is not zero, startA is set to new_start and startB is set to new_startB+shfitB.

[0080] It should be noted that after calculating the offset, the result of the next rough comparison may still not be enough to meet the threshold. After repeatedly triggering F6, the program will fall into an infinite loop. Because only startB is assigned an offset, the present invention can determine whether startA remains unchanged for three consecutive times. If it remains unchanged for three consecutive times (backward_time), it is determined that the rough comparison of this video segment cannot be solved, and the rough comparison is continued. The video segment that cannot be solved by the rough comparison is left for subsequent fine comparison to be solved.

[0081] F7. If startA is greater than or equal to the duration of videoA, or startB is greater than or equal to the duration of videoB, then the loop ends.

[0082] After the above cycle, the two long videos are sampled and compared using the step-by-step relay method, and the match_result is obtained, which contains the starting point information and similarity score information of multiple segments (including the starting point information and similarity score information of the same video segment).

[0083] It should be noted that the above process of calculating the offset shfitB of the current same fragment is as follows:

[0084] (1) Input: After rough comparison, we get a pair of start and end points and a similarity matrix of the two videos. Since this is a correction of the rough comparison result, when the step length is 1, this step will be skipped and the original start and end points will be directly returned.

[0085] (2) Calculation process: Take a short segment near the start and end points of the comparison video (usually 2 seconds, if the segment is less than 2 seconds, take the entire segment). With a step size of 1, extract all frames, calculate features through the image model, and then calculate the similarity to obtain a similarity matrix.

[0086] (3) Calculate the unit matrix with half the length of the similarity matrix side, and start from the starting point in the x and y directions, move forward 1 second (25 steps) each, and calculate the average similarity score. Then take the largest route as the best route and return the corresponding offset.

[0087] It should be noted that the above analysis of sim_mat to obtain the comparison results of the two videos in the current loop (including the two lists of bbox_result and scores) is as follows:

[0088] (1) Input: Similarity matrix of two videos, a matrix of shape (M, N). The range of each element is 0~1. The larger the value, the higher the similarity. 0 means completely dissimilar and 1 means completely identical. Due to the influence of errors and the copying process, the threshold is taken as 0.98 here, and two video frames with a similarity higher than 0.98 are considered to be exactly the same.

[0089] (2) Calculation process:

[0090] Step 1. Binarize the similarity matrix using a threshold. Assign 1 to the elements in the similarity matrix that are greater than or equal to the threshold, and copy the others to 0.

[0091] Step 2. Use the findContours method in opencv to find the image contours in step 1 above, and obtain a series of rectangular frame ranges of image contours in the similarity binary matrix. And judge each rectangular frame, if its length or width is less than 2, it is considered to be a useless part and discarded.

[0092] Step 3. Find the best route (the diagonal of the square with the maximum value) in each rectangular box. For the same video clip, the duration of the two video clips must be consistent, and the similarity of each frame must be greater than the threshold. This is reflected in the similarity matrix, which means that the effective area of ​​this similarity clip must be a square with equal length and width, and its diagonal is each similar corresponding frame. First, calculate the unit matrix of the shorter side of the image contour, and along the starting point of the longer side-the longer side-the shorter side, calculate the sum of the product of the unit matrix and the corresponding area of ​​the image contour, and then divide it by the side length of the unit matrix to get the average score of the current path. Find the corresponding area with the largest average score to get the path with the maximum score. And record the coordinates of this square unit matrix.

[0093] Step 4. Use the dynamic programming method to find the best similar route. The specific process is shown in steps a and b below.

[0094] Step a: Sort the square boxes found in step 3 above, and calculate the Euclidean distance between the lower left corner of all square boxes and the lower left vertex (M, N) of the matrix. Then sort them from low to high according to this Euclidean distance. For each square box, instantiate it into a class, which contains the coordinate bbox and the score score (diagonal) of the current rectangular box. And preset two empty attributes, the maximum score route boxes=[] of the current coordinate box, and the maximum route score max_sum_score=None of the current coordinate.

[0095] Step b: traverse and calculate all the square boxes in order from the smallest distance to the largest distance. The specific process is shown in the following steps A and B.

[0096] Step A: Taking the lower left corner point of the current square box as the boundary, find all the rectangular boxes whose upper right corners have x and y coordinates greater than or equal to the coordinates of point. (Find all the same segments that can form a route with the current same segment).

[0097] Step B: Calculate the diagonal score of the current rectangle and the sum of the maximum score max_sum_score of the rectangles in step A, and find the largest score combination. Use max_sum_score+score as the maximum route score of the current rectangle, add the current coordinates to boxes, and use it as the maximum score route of the current rectangle. If no rectangle that meets the conditions is found in step A, add the coordinates of the current rectangle to boxes, and assign max_sum_score to score.

[0098] Note: If the boxes and max_sum_score of the rectangular boxes found in step A above are not None, it means that the best route has been calculated and can be used directly in B. If max_sum_score is None, it means that the best route has not been calculated. At this time, use the recursive method to calculate boxes and max_sum_score using steps A and B.

[0099] After the above iterative calculation is completed, find the route with the largest max_sum_score in all square boxes.

[0100] (3) Output: Through saliency analysis and dynamic programming calculation, find the list of start and end points and scores of the same sequence of the two videos matched in the similarity matrix and output them.

[0101] S200, based on the starting point information and the ending point information in the first similarity information, calculating the second similarity information between the missing video segments in the first video and the second video, so as to perform a detailed comparison between the first video and the second video, wherein the second similarity information includes the starting point information, the ending point information and the second similarity of the corresponding missing video segment, the second similarity represents the similarity between the missing video segment of the first video and the missing video segment of the second video, and the missing video segment is a video segment that is not involved in the process of calculating the first similarity information multiple times;

[0102] Optionally, as described above, in S100, a video segment is selected in a stepwise manner to extract video frames for rough comparison. In this manner, a portion of the long video will be missing on the time axis and will not be rough compared. Therefore, for the missing portion, the present invention can perform S200 to perform fine-grained comparison and find the remaining possible identical video segments.

[0103] That is, in some optional embodiments, the S200 includes: S210, S220 and S230;

[0104] S210: Determine the starting point information and the ending point information of the missing video segments in the first video and the second video based on the starting point information and the ending point information in the first similarity information;

[0105] S220: based on the starting point information and the ending point information of the missing video segment, extract the corresponding missing video segment from the first video and the second video;

[0106] S230: Extract multiple video frames from the missing video segments in the first video and the second video respectively, and calculate second similarity information between the missing video segments based on the extracted video frames.

[0107] Optionally, in some optional embodiments, the S230 includes: S231, S232, S233 and S234;

[0108] S231. For any extraction of video frames, the following steps are performed: extracting multiple video frames from the missing video segment of the first video to obtain multiple third video frame groups, and extracting multiple video frames from the missing video segment of the second video to obtain multiple fourth video frame groups, wherein each extraction of video frames corresponds to one third video frame group and one fourth video frame group, and a video start time of a dynamic video segment extracted in the latter extraction is later than a video start time of a missing video segment extracted in the previous extraction;

[0109] S232, using a pre-trained picture model to perform feature extraction on the third video frame group and the fourth video frame group respectively, to obtain corresponding third feature matrices and fourth feature matrices, wherein one third video frame group corresponds to one third feature matrix, and one fourth video frame group corresponds to one fourth feature matrix;

[0110] S233, performing a dot multiplication of the transposed matrix of the third characteristic matrix and the transposed matrix of the fourth characteristic matrix to obtain a corresponding similarity matrix;

[0111] S234: Analyze the similarity matrix to obtain a video timestamp of a same video segment of the third video and the fourth video as one item of the second similarity information.

[0112] Optionally, assuming that the duration of videoA is 300 seconds and the duration of videoB is 320 seconds, the comparison result in S100 is [(0, 0, 80, 80), (90, 110, 280, 300)], and the time axis range of the missing video segment is [(80, 80, 90, 110), (280, 300, 300, 320)].

[0113] For the missing video segment, the present invention performs the process of S200, which is also a cyclic process, which is described in detail below.

[0114] Loop initialization: add (0, 0, 0, 0) to the start time of match_result, add (durationA, durationA, durationB, durationB) to the end, and assign it to re_match_result (match_result remains unchanged), let the final result list of detailed comparison be new_result=[], and let the number of loops be the length of match_result.

[0115] Loop body: N = (0 to the length of match_result).

[0116] F1. Get the time starting point of the current loop. Let the starting point be the end point of the Nth element in re_match_result, let the end point be the starting point of the N+1th element in re_match_result, and intercept the corresponding segments of videoA and videoB.

[0117] F2. For the clips captured from videoA and videoB, use the comparison method of S100 and set the stride to 1, that is, all frames are sampled and compared to obtain the comparison results.

[0118] F3. Add the comparison results of the detailed comparison obtained in the loop of F2 to the new_result list.

[0119] It should be noted that: after each cycle of a coarse comparison, the present invention can then perform a corresponding fine comparison. Of course, the present invention can also cycle all coarse comparisons and then cycle all fine comparisons, and the present invention is not limited to this.

[0120] S300: Determine identical video segments between the first video and the second video based on the first similarity information and the second similarity information.

[0121] Optionally, in some optional implementations, the S300 includes: S310;

[0122] S310: Based on the start point information and the end point information in the first similarity information and the second similarity information, the corresponding video segments are merged to obtain a final identical video segment, wherein during the merging process, repeated time information is superimposed and overlapped, and non-repeated time information is connected to each other in chronological order.

[0123] Optionally, the present invention can merge the result match_result of S100 and the result new_result of S200 into the final comparison result. If the lengths of the comparison result of match_result and new_result are both greater than 0, a merge operation is performed, otherwise the comparison result with a length greater than 0 is returned.

[0124] Optionally, in the merging operation, the present invention can merge the parts of the time periods in the two results on the time axis, and add the non-overlapping parts into the list. The specific judgment process is as follows:

[0125] (1) For each fragment n_result (n_a1, n_b1, n_a2, n_b2) in new_result, a judgment is made in match_result.

[0126] (2) If there is a fragment m_result (m_a1, m_b1, m_a2, m_b2) in match_result, and the starting point m_a1 of the fragment in videoA is located between (n_a1 and n_a2), and the starting point mb1 of the fragment in videoB is located between (m_b1 and m_b2), then the starting point of this element of match_result is set to (n_a1, n_b1), that is, m_result = (n_a1, n_b1, m_a2, m_b2).

[0127] (3) If there is a fragment m_result (m_a1, m_b1, m_a2, m_b2) in match_result, and the end point m_a2 of the fragment in videoA is located between (n_a1 and n_a2), and the end point mb2 of the fragment in videoB is located between (m_b1 and m_b2), then the end point of this element of match_result is set to (n_a2, n_b2), that is, m_result = (m_a1, m_b1, n_a2, n_b2).

[0128] (4) If there is no element in match_result that satisfies (1) or (2), n_result is added to the final result.

[0129] Optional, such as Figure 2 As shown, in some optional embodiments, after S300, the method further includes: S400, S500 and S600;

[0130] S400, extending the end point information of the same video segment backward or moving the start point information of the same video segment forward to obtain a corresponding new video segment;

[0131] S500, calculating the similarity of a new video segment between the first video and the second video;

[0132] S600: If the similarity of the new video segment between the first video and the second video is greater than a preset threshold, update the corresponding identical video segment to the new video segment.

[0133] Optionally, for the result obtained from S300, the present invention can try to expand it at both ends of the same direction. The specific calculation process is as follows:

[0134] (1) For each final identical segment, try to expand it near its starting point and end point. For example, assuming (x, y) is the end point, calculate the similarity of (x+1, y+1). If it is greater than the threshold, assign the end point of this segment to (x+1, y+1). The same is true for the starting point (in the opposite direction). If it is not greater than the threshold, continue to judge in the current direction. If there is a segment greater than the threshold, add this segment to the result.

[0135] (2) If there are no segments with a similarity greater than the threshold for five consecutive steps, the judgment of the current direction is terminated.

[0136] After the above process, the present invention can obtain the starting points of multiple identical segments in two videos, and the present invention does not impose any limitation on this.

[0137] In summary, the technical solution of the present invention has the following beneficial effects:

[0138] 1. Significantly improved the efficiency of video review and reduced the workload of reviewers.

[0139] 2. Through a phased comparison strategy, fast video analysis is achieved while reducing memory usage.

[0140] 3. Improved the accuracy of video difference detection, especially the ability to capture subtle changes.

[0141] That is, the audit video comparison method provided by the present invention can greatly improve the auditing efficiency of auditors, and the comparison process occupies little memory and has a fast comparison speed.

[0142] like Figure 3 As shown, the present invention provides a video version comparison device, comprising: a coarse comparison unit 100, a fine comparison unit 200 and a result combing unit 300;

[0143] The rough comparison unit 100 is used to extract multiple video frames from the dynamic video segments of the first video and the second video for multiple times, and calculate the first similarity information of the dynamic video segments of the first video and the second video based on the extracted video frames, so as to perform a rough comparison on the first video and the second video, wherein each time a video segment is selected from the first video and the second video respectively as the dynamic video segment of the corresponding video, and the first similarity information includes the starting point information, the ending point information and the first similarity of the corresponding dynamic video segment, and the first similarity represents the similarity between the dynamic video segment of the first video and the dynamic video segment of the second video;

[0144] The detailed comparison unit 200 is used to calculate the second similarity information between the missing video segments in the first video and the second video based on the starting point information and the ending point information in the first similarity information, so as to perform detailed comparison on the first video and the second video, wherein one piece of the second similarity information includes the starting point information, the ending point information and the second similarity of the corresponding missing video segment, the second similarity represents the similarity between the missing video segment of the first video and the missing video segment of the second video, and the missing video segment is a video segment that is not involved in the process of calculating the first similarity information multiple times;

[0145] The result combing unit 300 is configured to determine the same video segments between the first video and the second video based on the first similarity information and the second similarity information.

[0146] Optionally, in some optional implementations, the coarse comparison unit 100 includes: a first extraction subunit and a first calculation subunit;

[0147] The first extraction subunit is used to perform, for any extraction of video frames, the following steps: extracting multiple video frames from the dynamic video segment of the first video to obtain multiple first video frame groups, and extracting multiple video frames from the dynamic video segment of the second video to obtain multiple second video frame groups, wherein each extraction of video frames corresponds to one first video frame group and one second video frame group, and a video start time of the dynamic video segment extracted the latter time is later than a video start time of the dynamic video segment extracted the previous time;

[0148] The first calculation subunit is used to calculate corresponding first similarity information based on the extracted first video frame group and the second video frame group, wherein one video frame group corresponds to one first similarity information.

[0149] Optionally, in some optional embodiments, the first calculation subunit includes: a first feature subunit, a first point multiplication subunit and a first analysis subunit;

[0150] The first feature subunit is used to use a pre-trained picture model to perform feature extraction on the first video frame group and the second video frame group respectively to obtain corresponding first feature matrices and second feature matrices, wherein one first video frame group corresponds to one first feature matrix, and one second video frame group corresponds to one second feature matrix;

[0151] The first point multiplication subunit is used to perform a point multiplication of the transposed matrix of the first feature matrix and the transposed matrix of the second feature matrix to obtain a corresponding similarity matrix;

[0152] The first analysis subunit is configured to analyze the similarity matrix to obtain video timestamps of the same video segments of the first video and the second video.

[0153] Optionally, in certain optional embodiments, the first analysis subunit includes: a first matrix analysis subunit and a first information determination subunit;

[0154] The first matrix analysis subunit is used to analyze the similarity matrix to obtain the start point information and the end point information of the same video segment, wherein the same video segment is a video segment whose similarity is greater than a preset similarity threshold;

[0155] The first information determination subunit is used to obtain a corresponding video timestamp as an item of the first similarity information based on the start point information and the end point information of the same video segment.

[0156] Optionally, in some optional implementations, the detailed comparison unit 200 includes: a missing segment information subunit, a missing segment truncation subunit and a missing segment extraction subunit;

[0157] The missing segment information subunit is used to determine the starting point information and the ending point information of the missing video segments in the first video and the second video based on the starting point information and the ending point information in the first similarity information;

[0158] The missing segment interception subunit is used to intercept the corresponding missing video segment from the first video and the second video based on the starting point information and the ending point information of the missing video segment;

[0159] The missing segment extraction subunit is used to extract multiple video frames from the missing video segments in the first video and the second video respectively, and calculate the second similarity information between the missing video segments based on the extracted video frames.

[0160] Optionally, in some optional embodiments, the missing segment extraction subunit includes: a second extraction subunit, a second feature subunit, a second point multiplication subunit and a second analysis subunit;

[0161] The second extraction subunit is used to perform, for any extraction of video frames, the following steps: extracting multiple video frames from the missing video segment of the first video to obtain multiple third video frame groups, and extracting multiple video frames from the missing video segment of the second video to obtain multiple fourth video frame groups, wherein each extraction of video frames corresponds to one third video frame group and one fourth video frame group, and a video start time of the dynamic video segment extracted the latter time is later than a video start time of the missing video segment extracted the previous time;

[0162] The second feature subunit is used to use a pre-trained picture model to perform feature extraction on the third video frame group and the fourth video frame group respectively to obtain corresponding third feature matrices and fourth feature matrices, wherein one third video frame group corresponds to one third feature matrix, and one fourth video frame group corresponds to one fourth feature matrix;

[0163] The second point multiplication subunit is used to perform a point multiplication of the transposed matrix of the third characteristic matrix and the transposed matrix of the fourth characteristic matrix to obtain a corresponding similarity matrix;

[0164] The second analysis subunit is used to analyze the similarity matrix to obtain a video timestamp of a same video segment of the third video and the fourth video as an item of information in the second similarity information.

[0165] Optionally, in some optional embodiments, the result combing unit 300 includes: a result combing subunit;

[0166] The result combing subunit is used to merge the corresponding video segments to obtain the final identical video segments based on the start point information and the end point information in the first similarity information and the second similarity information, wherein during the merging process, repeated time information is superimposed and covered together, and non-repeated time information is connected to each other in chronological order.

[0167] Optionally, in certain optional implementations, the apparatus further comprises: a starting point moving unit, a new similarity calculating unit, and a video segment updating unit;

[0168] The starting point moving unit is configured to, after determining the same video segment between the first video and the second video based on the first similarity information and the second similarity information, extend the end point information of the same video segment backward or move the start point information of the same video segment forward to obtain a corresponding new video segment;

[0169] The new similarity calculation unit is used to calculate the similarity of a new video segment between the first video and the second video;

[0170] The video segment updating unit is configured to update the corresponding identical video segments to the new video segments if the similarity of the new video segments between the first video and the second video is greater than a preset threshold.

[0171] The present invention provides a computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the video version comparison method described in any one of the above items is implemented.

[0172] like Figure 4 As shown, the present invention provides an electronic device 70, which includes at least one processor 701, and at least one memory 702 and a bus 703 connected to the processor 701; wherein the processor 701 and the memory 702 communicate with each other through the bus 703; the processor 701 is used to call the program instructions in the memory 702 to execute any of the video version comparison methods described above.

[0173] In the present invention, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0174] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0175] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined in the present invention may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in the present invention, but will conform to the widest scope consistent with the principles and novel features disclosed in the present invention.

[0176] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A video version comparison method, characterized in that: include: Extracting multiple video frames from the dynamic video segments of the first video and the second video multiple times, and calculating first similarity information of the dynamic video segments of the first video and the second video based on the extracted video frames, so as to roughly compare the first video and the second video, wherein each time a video segment is respectively selected from the first video and the second video as the dynamic video segment of the corresponding video, and the first similarity information includes starting point information, end point information and a first similarity of the corresponding dynamic video segment, and the first similarity represents the similarity between the dynamic video segment of the first video and the dynamic video segment of the second video; Based on the starting point information and the ending point information in the first similarity information, second similarity information between the missing video segments in the first video and the second video is calculated to perform a detailed comparison between the first video and the second video, wherein one piece of the second similarity information includes the starting point information, the ending point information and the second similarity of the corresponding missing video segment, the second similarity represents the similarity between the missing video segment of the first video and the missing video segment of the second video, and the missing video segment is a video segment that is not involved in the process of calculating the first similarity information multiple times; Based on the first similarity information and the second similarity information, identical video segments between the first video and the second video are determined.

2. The method according to claim 1, characterized in that The extracting multiple video frames from the dynamic video segments of the first video and the second video for multiple times, and calculating first similarity information of the dynamic video segments of the first video and the second video based on the extracted video frames, includes: For any extraction of video frames, the following steps are performed: extracting multiple video frames from the dynamic video segment of the first video to obtain multiple first video frame groups, and extracting multiple video frames from the dynamic video segment of the second video to obtain multiple second video frame groups, wherein each extraction of video frames corresponds to one first video frame group and one second video frame group, and a video start time of a dynamic video segment extracted the second time is later than a video start time of a dynamic video segment extracted the previous time; Corresponding first similarity information is calculated based on the extracted first video frame group and the second video frame group, wherein one video frame group corresponds to one first similarity information.

3. The method according to claim 2, characterized in that The step of calculating corresponding first similarity information based on the extracted first video frame group and the second video frame group includes: Using a pre-trained picture model, respectively extracting features from the first video frame group and the second video frame group to obtain corresponding first feature matrices and second feature matrices, wherein one first video frame group corresponds to one first feature matrix, and one second video frame group corresponds to one second feature matrix; Multiply the transposed matrix of the first feature matrix by the transposed matrix of the second feature matrix to obtain a corresponding similarity matrix; The similarity matrix is ​​analyzed to obtain video timestamps of the same video segments of the first video and the second video.

4. The method according to claim 3, characterized in that The analyzing the similarity matrix to obtain video timestamps of the same video segments of the first video and the second video includes: Analyzing the similarity matrix to obtain start point information and end point information of the same video segment, wherein the same video segment is a video segment whose similarity is greater than a preset similarity threshold; Based on the start point information and the end point information of the same video segment, a corresponding video timestamp is obtained as one of the first similarity information.

5. The method according to claim 1, characterized in that The calculating, based on the start point information and the end point information in the first similarity information, second similarity information between the missing video segments in the first video and the second video to perform a detailed comparison on the first video and the second video includes: Determine the starting point information and the ending point information of the missing video segments in the first video and the second video based on the starting point information and the ending point information in the first similarity information; Based on the start point information and the end point information of the missing video segment, extracting the corresponding missing video segment from the first video and the second video; A plurality of video frames are respectively extracted from the missing video segments in the first video and the second video, and second similarity information between the missing video segments is calculated based on the extracted video frames.

6. The method according to claim 5, characterized in that The extracting a plurality of video frames from the missing video segments in the first video and the second video respectively, and calculating the second similarity information between the missing video segments based on the extracted video frames, comprises: For any extraction of video frames, the following steps are performed: extracting multiple video frames from the missing video segment of the first video to obtain multiple third video frame groups, and extracting multiple video frames from the missing video segment of the second video to obtain multiple fourth video frame groups, wherein each extraction of video frames corresponds to one third video frame group and one fourth video frame group, and the video start time of the dynamic video segment extracted the latter time is later than the video start time of the missing video segment extracted the previous time; Using a pre-trained picture model, respectively extracting features from the third video frame group and the fourth video frame group to obtain corresponding third feature matrices and fourth feature matrices, wherein one third video frame group corresponds to one third feature matrix, and one fourth video frame group corresponds to one fourth feature matrix; Multiply the transposed matrix of the third characteristic matrix by the transposed matrix of the fourth characteristic matrix to obtain a corresponding similarity matrix; The similarity matrix is ​​analyzed to obtain a video timestamp of a same video segment of the third video and the fourth video as one item of the second similarity information.

7. The method according to claim 1, characterized in that The determining, based on the first similarity information and the second similarity information, the same video segments between the first video and the second video includes: Based on the start point information and the end point information in the first similarity information and the second similarity information, the corresponding video segments are merged to obtain the final identical video segments, wherein during the merging process, repeated time information is superimposed and covered, and non-repeated time information is connected to each other in chronological order.

8. The method according to claim 1, characterized in that After determining the same video segments between the first video and the second video based on the first similarity information and the second similarity information, the method further includes: Extend the end point information of the same video segment backward or move the start point information of the same video segment forward to obtain a corresponding new video segment; Calculating the similarity of a new video segment between the first video and the second video; If the similarity of the new video segment between the first video and the second video is greater than a preset threshold, the corresponding identical video segment is updated to be the new video segment.

9. A video version comparison device, characterized in that: include: Coarse comparison unit, fine comparison unit and result combing unit; The rough comparison unit is used to extract multiple video frames from the dynamic video segments of the first video and the second video for multiple times, and calculate first similarity information of the dynamic video segments of the first video and the second video based on the extracted video frames, so as to perform a rough comparison on the first video and the second video, wherein each time a video segment is selected from the first video and the second video respectively as the dynamic video segment of the corresponding video, and the first similarity information includes the starting point information, the ending point information and the first similarity of the corresponding dynamic video segment, and the first similarity represents the similarity between the dynamic video segment of the first video and the dynamic video segment of the second video; The detailed comparison unit is used to calculate second similarity information between missing video segments in the first video and the second video based on the starting point information and the ending point information in the first similarity information, so as to perform detailed comparison on the first video and the second video, wherein one piece of the second similarity information includes the starting point information, the ending point information and the second similarity of the corresponding missing video segment, the second similarity represents the similarity between the missing video segment of the first video and the missing video segment of the second video, and the missing video segment is a video segment that is not involved in the process of calculating the first similarity information multiple times; The result combing unit is configured to determine the same video segments between the first video and the second video based on the first similarity information and the second similarity information.

10. An electronic device, characterized in that: The electronic device includes at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the video version comparison method as described in any one of claims 1 to 8.

Images