Method and device for comparing video similarity and electronic device

Abstract

The present application relates to the field of video copyright protection, and specifically to a method for comparing video similarity, comprising: obtaining a first video and a second video; performing a frame extraction operation on the first video to obtain a first image, and performing a frame extraction operation on the second video to obtain a second image; performing a similarity comparison process on the first image and the second image to obtain a first comparison similarity; if it is determined that the first comparison similarity is not less than a preset similarity threshold, the first comparison similarity is used as the similarity between the first image and the second image; if it is determined that the first comparison similarity is less than the preset similarity threshold, feature point comparison process is performed on the first image and the second image to obtain a second comparison similarity; and the similarity between the first video and the second video is obtained based on the similarity between the first image and the second image.

Description

Technical Field

[0001] The present application relates to the field of video copyright protection, and in particular to a method and device for comparing video similarity and an electronic device. Background Art

[0002] At present, with the improvement of public copyright awareness, more and more content producers are seeking copyright protection for their original content. In the field of video content, original videos are often stolen and edited before being used. Copyright owners want to protect their rights and interests. Existing technologies directly use two complete videos to determine infringement, but the infringing video may only have captured part of the original video. This results in low efficiency, time wasting, and duplication of work when looking for infringing video clips. Summary of the invention

[0003] Based on this, the present application provides a method and device for comparing video similarity and an electronic device, which obtains images by performing frame extraction operations on videos, and then flexibly applies two image similarity comparison methods to find similar images extracted from two videos, and finds similar video clips in accordance with the time at which the corresponding images are located in the video, thereby obtaining their similarity.

[0004] According to one aspect of the present application, a method for comparing video similarity is proposed, comprising:

[0005] Get the first video and the second video;

[0006] Performing a frame extraction operation on the first video to obtain a first image, and performing a frame extraction operation on the second video to obtain a second image;

[0007] Performing a similarity comparison process on the first image and the second image to obtain a first comparison similarity;

[0008] If it is determined that the first comparison similarity is not less than a preset similarity threshold, the first comparison similarity is used as the similarity between the first image and the second image;

[0009] If it is determined that the first comparison similarity is less than a preset similarity threshold, feature point comparison processing is performed on the first image and the second image to obtain a second comparison similarity; if it is determined that the second comparison similarity is less than the first comparison similarity, the first comparison similarity is used as the similarity between the first image and the second image;

[0010] If it is determined that the second comparison similarity is not less than the first comparison similarity, the second similarity is used as the similarity between the first image and the second image; and the similarity between the first video and the second video is obtained based on the similarity between the first image and the second image.

[0011] According to some embodiments, a first image and a second image are subjected to similarity comparison processing to obtain a first comparative similarity, including: grayscale processing the first image and reducing it to a predetermined size to obtain a first converted image; grayscale processing the second image and reducing it to a predetermined size to obtain a second converted image; processing the first converted image using a mean hash algorithm to obtain a first hash fingerprint; processing the second converted image using a mean hash algorithm to obtain a second hash fingerprint; and comparing the first hash fingerprint and the second hash fingerprint to obtain a first comparative similarity.

[0012] According to some embodiments, feature point comparison processing is performed on the first image and the second image to obtain a second comparison similarity, including: feature point extraction processing is performed on the first image to obtain first feature points, where the first feature points are points on the first image; feature point extraction processing is performed on the second image to obtain second feature points, where the second feature points are points on the second image.

[0013] According to some embodiments, the aforementioned method further includes: determining the number of the first feature points or

[0014] Whether the number of the second feature points is less than the preset lower limit of the number of feature points; if so, the first comparison similarity is used as the similarity between the first image and the second image; if not, feature point matching processing is performed on the first feature points and the second feature points to obtain similar feature points, which are similar feature points between the first feature points and the second feature points.

[0015] According to some embodiments, after performing feature point matching processing on the first feature point and the second feature point to obtain similar feature points, it also includes: determining whether the number of similar feature points is less than a preset lower limit of the number of similar feature points; if less than, obtaining a second comparative similarity through the number of similar feature points, the number of first feature points, and the number of second feature points; if not less than, obtaining a first similar area in the first image and a second similar area in the second image based on the similar feature points.

[0016] According to some embodiments, after obtaining a first similar region in the first image based on similar feature points and obtaining a second similar region in the second image, it also includes: obtaining the similarity of the first feature points by comparing the area of ​​the first similar region with the area of ​​the first image; and obtaining the similarity of the second feature points by comparing the area of ​​the second similar region with the area of ​​the second image.

[0017] According to some embodiments, the aforementioned method further includes: comparing the first feature point similarity with the second feature point similarity; and taking the larger value as the second comparison similarity.

[0018] According to some embodiments, performing a frame extraction operation on a first video to obtain a first image and performing a frame extraction operation on a second video to obtain a second image include: recording a first time of the first image in the first video and recording a second time of the second image in the second video.

[0019] According to some embodiments, the similarity between the first video and the second video is obtained based on the similarity between the first image and the second image, including: obtaining a first similar video segment of the first video and a second similar video segment of the second video based on the similarity between the first image and the second image and the first time and the second time; obtaining the similarity between the first video and the second video based on the length of the first similar video segment and the first video and the length of the second similar video segment and the second video.

[0020] According to one aspect of the present application, a device for comparing video similarity is provided, characterized in that it includes:

[0021] An acquisition module is used to acquire the first video and the second video;

[0022] A frame extraction module, performing a frame extraction operation on the first video to obtain a first image, and performing a frame extraction operation on the second video to obtain a second image;

[0023] A first similarity comparison module performs a similarity comparison process on the first image and the second image to obtain a first comparison similarity;

[0024] A judgment module, which judges that if the first comparison similarity is not less than a preset similarity threshold, then uses the first comparison similarity as the similarity between the first image and the second image;

[0025] The judgment module further judges that if the first comparison similarity is less than a preset similarity threshold, then transfers to the second similarity comparison module;

[0026] A second similarity comparison module performs feature point comparison processing on the first image and the second image to obtain a second comparison similarity;

[0027] The judging module further judges that if the second comparison similarity is less than the first comparison similarity, the first comparison similarity is used as the similarity between the first image and the second image;

[0028] The judging module further judges that if the second comparison similarity is not less than the first comparison similarity, the second similarity is used as the similarity between the first image and the second image;

[0029] The video similarity calculation module obtains the similarity between the first video and the second video based on the similarity between the first image and the second image.

[0030] According to one aspect of the present application, an electronic device is provided, characterized in that it includes

[0031] one or more processors;

[0032] A storage device for storing one or more programs;

[0033] When one or more programs are executed by one or more processors, the one or more processors implement any of the aforementioned methods.

[0034] Beneficial effects of this application:

[0035] According to some embodiments, the method proposed in the present application compresses the amount of video data to be processed through a frame extraction operation.

[0036] According to some embodiments, the method proposed in the present application comprehensively utilizes two different image comparison processing methods to compare images extracted from videos, thereby improving the efficiency of image comparison similarity and at the same time improving the accuracy of image comparison similarity.

[0037] According to some embodiments, when the infringing video is obtained by partially intercepting the original video, the method proposed in the present application can still efficiently find similar video clips between the videos, thereby deriving the similarity between the videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without exceeding the scope of protection required by the present application.

[0039] Figure 1 A flow chart of a method for comparing video similarities according to an example embodiment is shown.

[0040] Figure 2 A flow chart of a method for comparing video similarities according to an example embodiment is shown.

[0041] Figure 3 A flow chart of a method for comparing video similarities according to an example embodiment is shown.

[0042] Figure 4 A flow chart of a method for comparing video similarities according to an example embodiment is shown.

[0043] Figure 5 A block diagram of a device for comparing video similarities according to an example embodiment is shown.

[0044] Figure 6 A schematic diagram showing a method for comparing video similarities according to an example embodiment.

[0045] Figure 7 A block diagram of an electronic device according to an exemplary embodiment is shown. DETAILED DESCRIPTION

[0046] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0047] The described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of these specific details, or other modes, components, materials, devices or the like may be adopted. In these cases, known structures, methods, devices, implementations, materials or operations will not be shown or described in detail.

[0048] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0049] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0050] Video similarity can be used as the basis for determining infringement in video copyright protection. However, since the infringer may have partially intercepted or spliced ​​the video to be protected, existing methods are unable to efficiently determine the similarity between videos and make infringement determinations in this case.

[0051] To this end, the present application proposes a solution. According to some embodiments of the present application, two image similarity comparison methods are flexibly applied when comparing the similarity of two images, so that the comparison accuracy is improved while the processing efficiency is improved.

[0052] Figure 1 A flow chart of a method for comparing video similarities according to an example embodiment is shown.

[0053] like Figure 1 In S100, a first video and a second video are obtained.

[0054] According to an example embodiment, the first video and the second video may be stored locally or in a cloud, etc.

[0055] According to the example embodiment, the first video and the second video are respectively a video to be protected and a video to be determined (suspected infringing video). For the convenience of description, in the description of the embodiment of the present application, the first video is taken as the video to be protected and the second video is taken as the video to be determined, and no further details are given hereinafter.

[0056] According to an embodiment, the first video and the second video, in particular, may also be, for example, GIF animations.

[0057] According to an exemplary embodiment, after obtaining the first video and the second video, the two videos can also be uniformly transcoded and converted into the same encoding format for subsequent processing. According to an embodiment, for example, FFmpeg software can be used to uniformly convert the first video and the second video into MP4 format.

[0058] In S101, a frame extraction operation is performed on the first video / the second video to obtain a first image / the second image.

[0059] According to an example embodiment, after the first video and the second video are uniformly transcoded, their video information, such as the video frame rate, is obtained, and a certain number of frame images are extracted per second according to the video frame rate as the first image and the second image for subsequent image similarity comparison processing.

[0060] According to one embodiment, for example, when the frame rate of the first video or the second video is 24 frames per second, an average of 3 images per second can be extracted as images to be compared, and the first extracted image sequence and the second extracted image sequence are stored separately according to the video. If the frame rate is increased, more images can be extracted. The principle is that the number of images extracted can make the subsequent calculation results of video similarity accurate and comprehensive without consuming too much processing resources and time costs, but this application does not limit the specific method and rules for extracting images.

[0061] According to an exemplary embodiment, the method further includes recording a first time / second time of the first image / second image in the first video / second video in the aforementioned first extracted image sequence / second extracted image sequence.

[0062] In S103, a similarity comparison process is performed on the first image and the second image to obtain a first comparison similarity.

[0063] According to an example embodiment, the first image is all images obtained by performing a frame extraction operation on the first video, and the second image is all images obtained by performing a frame extraction operation on the second video. The comparison method is: each first image is compared with each second image in turn in time series, which will not be repeated hereafter.

[0064] According to an example embodiment, the processing method uses a hash function to process two images to obtain the similarity between the first image and the second image. There are many well-known technologies that can perform this processing, such as the "Average Hash Algorithm". Here, the "Average Hash Algorithm" is used as an example to briefly introduce its process: first, the two images are grayed and reduced, and then the hash value of each image is calculated to obtain its hash fingerprint, and then the fingerprints are compared to obtain a first comparative similarity. Its advantages are fast processing speed and resource saving. Its disadvantages are that it only has good performance when the two images are basically consistent. If the second image is the first image that has been processed by distortion, interception, splicing, etc., it is difficult to accurately distinguish. For specific methods, see Figure 2 Description of embodiments.

[0065] In S105 , it is determined whether the first comparison similarity is less than a preset similarity threshold.

[0066] According to an example embodiment, the preset similarity threshold is a similarity coefficient that is pre-set manually, which reflects the degree of similarity between two images and is used to determine: when the degree of similarity between two images reaches this threshold, it can be considered that the second image is substantially similar to the first image, and the second image can be determined to be infringing in the field of infringement determination.

[0067] According to an embodiment, the preset similarity threshold may be set to, for example, 0.8, which means that when 80% of the two images are similar, it is determined that the second image is substantially similar to the first image.

[0068] According to an exemplary embodiment, if the judgment result of S105 is "No", it means that the similarity between the two images obtained after the aforementioned similarity comparison process reaches the preset similarity threshold, and then the process goes to S107.

[0069] According to the example embodiment, if the judgment result of S105 is "yes", it means that the similarity between the two images obtained after the aforementioned similarity comparison processing does not reach the preset similarity threshold, but this does not mean that the second image is substantially dissimilar to the first image. As mentioned above, it is possible that the second image is obtained by deforming, cutting and splicing the first image. In this case, go to S109.

[0070] In S107, the first comparison similarity is used as the similarity between the first image and the second image.

[0071] According to the example embodiment, the similarity of the two images obtained after the aforementioned similarity comparison processing reaches a preset similarity threshold, that is, as mentioned above, it means that the two images are basically the same, and it can be determined that the second image is substantially similar to the first image. Then, the first comparison similarity can be used as the similarity between the first image and the second image, and the comparison of the first image and the second image can be ended.

[0072] In S109, feature point comparison processing is performed on the first image and the second image to obtain a second comparison similarity.

[0073] According to the example embodiment, the method uses the SIFT (Scale-invariant feature transform) algorithm or the SURF (Speeded Up Robust Features) algorithm to extract feature points from the two images respectively, and matches the extracted feature points to obtain similar feature points in the two images, and then calculates the second comparison similarity of the two images. This method has good robustness and can distinguish images that have been deformed, cut and spliced, partially covered, liquefied, etc. from the original image and find similar points and similar areas, and then calculate the second comparison similarity. For details, see Figure 3 Description of embodiments.

[0074] In S111 , it is determined whether the second comparison similarity is smaller than the first comparison similarity.

[0075] According to an example embodiment, for the obtained first comparison similarity and second comparison similarity, the larger value is taken as the final similarity between the first image and the second image.

[0076] Therefore, when S111 is judged as "yes", the process goes to S107 and the first comparison similarity is used as the similarity between the first image and the second image;

[0077] When S111 is judged as “No”, the process goes to S113 and uses the second comparison similarity as the similarity between the first image and the second image.

[0078] In S115 , the similarity between the first video and the second video is obtained based on the similarity between the first image and the second image.

[0079] According to an exemplary embodiment, based on the similarity between the first image and the second image obtained in S107 or S113, and the first time and the second time, the similarity between the first video and the second video is calculated. For details, see Figure 4 Description of embodiments.

[0080] Figure 2 A flow chart of a method for comparing video similarities according to an example embodiment is shown.

[0081] According to the example embodiment, as mentioned above, there are a variety of processing methods using hash functions to implement this processing process and obtain the similarity between the first image and the second image. The embodiment of this application uses the "hash mean algorithm" as an example for explanation.

[0082] like Figure 2 As shown, in S201, the first image / the second image is gray-scaled and reduced to a predetermined size to obtain a first converted image / a second converted image.

[0083] According to the exemplary embodiment, the first image and the second image are firstly gray-scaled and converted into gray-scale images, and then reduced to a predetermined size to obtain a first conversion image and a second conversion image. This step is intended to compress the information volume of the image and optimize the processing efficiency. The conversion into a gray-scale image reduces the information of each pixel from the values ​​of the three dimensions of red, green and blue to the value of one dimension, namely the gray-scale. Compression means compressing the number of pixels of the image to a level that is easy to handle.

[0084] According to one embodiment, the image can be converted into 64 grayscales, that is, 64 grayscales are divided from absolute white to absolute black, and the color value of each pixel is between 0 and 63. Then the image is compressed to 8×8 pixels, that is, it contains 64 pixels in total. The number of pixels can also be set by yourself. This value determines the number of bits of the hash value, that is, the hash fingerprint generated later. 64 pixels correspond to 64-bit hash fingerprints. For example, if it is set to 16×16 pixels, a 256-bit hash fingerprint will be generated. Similarly, the higher this value, the higher the accuracy, but it will also increase the complexity of processing.

[0085] In S203 , the first converted image / the second converted image is processed by using a mean hash algorithm to obtain a first hash fingerprint / a second hash fingerprint.

[0086] According to an example embodiment, the specific process of processing using the mean hash algorithm is:

[0087] First, the first conversion image / second conversion image generated in the above steps is converted to black and white respectively: for a single image, the average grayscale values ​​of all its pixels is calculated, and then a determination is made as to whether the grayscale value of each pixel is greater than the average grayscale value. If the determination result is true, the pixel is marked as "black" and assigned a value of 1; if the determination result is false, the pixel is marked as "white" and assigned a value of 0.

[0088] Then, the values ​​of each pixel of the converted black and white image are arranged in sequence from left to right and from top to bottom to obtain a 64-bit hash value composed of 0s and 1s, which is the hash fingerprint of the corresponding image.

[0089] In S205, the first hash fingerprint and the second hash fingerprint are compared to obtain a first comparison similarity.

[0090] According to an example embodiment, this step compares and calculates the similarity between the first hash fingerprint obtained in the previous step and the second hash fingerprint, that is, the first comparison similarity, and the specific steps are:

[0091] First, the Hamming distance between the first hash fingerprint and the second hash fingerprint is calculated. The process is to compare the two hash fingerprints bit by bit to find out the similarities and differences, and obtain the number of bits that are the same in the comparison result. For example, a Hamming distance of 58 means that the hash fingerprints of the two images have 58 bits in common among the 64 bits.

[0092] The first comparison similarity is then obtained by using the Hamming distance / total number of hash fingerprint bits. For example, in the above example, the first comparison similarity obtained is 58 / 64=0.90625, which means that the similarity between the first image and the second image is 90.625% as determined by this method.

[0093] According to the example embodiments, in summary, this method greatly compresses the information volume of the image during the process, making the processing efficiency very high, but at the same time, it is easy to know that only when the two images are basically the same, this method can effectively draw the conclusion that the two images are substantially similar based on their similarity. When the images are processed by steps such as distortion and interception, this method is difficult to effectively determine the substantial similarity of the two images, and is suitable for the preliminary screening of a large number of images to be compared.

[0094] Figure 3 A flow chart of a method for comparing video similarities according to an example embodiment is shown.

[0095] When the first image and the second image are Figure 2 If the method described in the embodiment fails to determine that the two images are similar, it does not mean that the two images are not substantially similar. Therefore, it is necessary to use another method that can still identify the similarities between the first image and the second image when the second image is generated by changing the first image. This method has good robustness, but the processing process is relatively cumbersome.

[0096] In S301, feature point extraction is performed on the first image / the second image to obtain first feature points / the second feature points.

[0097] According to an embodiment, the aforementioned SURF algorithm may be used to process the first image and the second image respectively to extract their feature points.

[0098] According to the exemplary embodiment, this step uses integral images, square filters, Hessian matrix and other methods well known to those skilled in the art to extract feature points from the image, and the specific implementation process will not be repeated here.

[0099] According to one embodiment, a library function in the OpenCV function library may be used to perform feature point extraction processing on the first image and the second image.

[0100] According to an example embodiment, the first feature point and the second feature point are both sets of points on the image, that is, after feature point extraction of the first image, a plurality of first feature points can be obtained, and after feature point extraction of the second image, a plurality of second feature points can also be obtained.

[0101] In S303, it is determined whether the number of the first feature points / the second feature points is less than a preset lower limit of the number of feature points.

[0102] According to an example embodiment, the preset lower limit of the number of feature points is artificially set, and its setting principle is that when the number of extracted feature points is not less than the preset lower limit of the number of feature points, similar feature points can be obtained more accurately through matching in subsequent steps, that is, when the number of extracted feature points is less than the preset lower limit of the number of feature points, similar feature points cannot usually be accurately obtained. According to an embodiment, the inventor of the present application sets the preset lower limit of the number of feature points to 20 through long-term practical research, but the present application does not limit this.

[0103] According to the example embodiment, for the first image or the second image, when the number of extracted feature points is insufficient, it means that the subsequent processing steps are difficult to continue to implement and thus obtain reliable comparison results. Therefore, the first comparison similarity obtained in the aforementioned steps can be directly used as the similarity between the first image and the second image, that is, when the judgment result of S303 is "yes", go to S305 and use the first comparison similarity as the similarity between the first image and the second image.

[0104] According to an exemplary embodiment, when the judgment result of S303 is “No”, it means that the number of feature points extracted can be used for the implementation of subsequent steps, and therefore, the process goes to S307 .

[0105] In S307, feature point matching processing is performed on the first feature point and the second feature point to obtain similar feature points.

[0106] According to the exemplary embodiment, the first feature point and the second feature point in the above step are matched to find similar feature points that can establish similar associations. This method is well known to those skilled in the art and will not be described in detail here.

[0107] According to an embodiment, a library function in the OpenCV function library may be used to perform feature point matching processing on the first feature point and the second feature point.

[0108] According to an example embodiment, this similar feature point is used to characterize a feature point that is similar to a second feature point in the first feature point, and is actually a mapping relationship established between the first feature point and the second feature point, that is, through a certain similar feature point, one point can be found in each of the first feature points and the second feature point, and these two points are similar.

[0109] In S309 , it is determined whether the number of similar feature points is less than a preset lower limit of the number of similar feature points.

[0110] According to an example embodiment, the preset lower limit of the number of similar feature points is artificially set, and its setting principle is that when the number of similar feature points matched is not less than the preset lower limit of the number of similar feature points, the similar area can be accurately obtained, that is, when the number of similar feature points matched is less than the preset lower limit of the number of similar feature points, the similar area cannot usually be accurately obtained. According to an embodiment, the inventor of the present application sets the preset lower limit of the number of similar feature points to 20 through long-term practical research, but the present application does not limit this.

[0111] When the judgment result of S309 is "yes", go to S311; when the judgment result of S309 is "no", go to S313.

[0112] In S311 , a second comparison similarity is obtained by the number of similar feature points and the number of first feature points / second feature points.

[0113] According to the example embodiment, as mentioned above, in this case, since it is impossible to accurately obtain similar areas in the first image and the second image, it is possible to calculate: the ratio of the number of similar feature points to the number of first feature points, and the ratio of the number of similar feature points to the number of second feature points, and take the maximum value of the above two ratios as the second comparison similarity.

[0114] In S312, based on the similar feature points, an association is established between the first feature point and the second feature point.

[0115] According to an example embodiment, a new image is generated, which includes a first image and a second image, and points on the two images that are associated by the same similar feature point are connected by a straight line to represent their association relationship, so that the similar feature points of the two images are visualized and associated.

[0116] In S313, a first similar region / a second similar region is obtained in the first image / the second image based on the similar feature points.

[0117] According to the exemplary embodiment, as mentioned above, in this case, the first similar area and the second similar area can be drawn in the first image and the second image respectively through the similar feature points obtained by matching. The effect can be referred to as Figure 5 Schematic diagram of .

[0118] According to the exemplary embodiment, the method is, for example, in the first image, the coordinates of all points corresponding to the similar feature points are obtained, and the point with the largest x coordinate is found, and a straight line parallel to the y axis is drawn through the point; the point with the smallest x coordinate is found, and a straight line parallel to the y axis is drawn through the point; the point with the largest y coordinate is found, and a straight line parallel to the x axis is drawn through the point; the point with the smallest y coordinate is found, and a straight line parallel to the x axis is drawn through the point. In this way, the four straight lines drawn above frame a rectangle, and the rectangular area contains all the similar feature points, and the rectangular area is the first similar area.

[0119] According to an example embodiment, the second similar region is obtained using the same method as the first similar region.

[0120] In S315 , the first feature point similarity / the second feature point similarity is obtained by comparing the area of ​​the first similar region / the second similar region with the area of ​​the first image / the second image.

[0121] According to an example embodiment, the first feature point similarity is obtained by dividing the area of ​​the first similar region obtained above by the area of ​​the first image, and the second feature point similarity is obtained by dividing the area of ​​the second similar region by the area of ​​the second image.

[0122] In S317, the first feature point similarity is compared with the second feature point similarity, and the larger value is taken as the second comparison similarity.

[0123] According to the example embodiment, the reason for taking the maximum value between the first feature point similarity and the second feature point similarity as the second similarity is that the first feature point similarity and the second feature point similarity are not the same. When, for example, a part of the second image is the entire content of the first image, it can be determined that the second image completely infringes the copyright of the first image. At this time, the first feature point similarity can be expected to be close to 100%, and using it as the second similarity can fully express the infringement determination result; and the second feature point similarity at this time depends on the size of the area occupied by the first image in the picture. If it is relatively small, the second feature point similarity cannot reflect the fact that the second image and the first image are substantially similar.

[0124] In S319 , based on the similar feature points, an association is established between the first feature point and the second feature point, and the first similar area and the second similar area are marked and displayed.

[0125] According to the example embodiment, a new image is generated, which includes the first image and the second image. The points on the two images that are associated with the same similar feature point are connected with a straight line to represent their association relationship, so that the similar feature points of the two images are visualized and associated. The effect can be referred to Figure 5 Schematic diagram of .

[0126] According to an exemplary embodiment, on the aforementioned newly generated image, a first similarity region and a second similarity region are drawn on the first image and the second image respectively, so as to further visualize and display the similarity regions of the two images.

[0127] Figure 4 A flow chart of a method for comparing video similarities according to an example embodiment is shown.

[0128] like Figure 4 As shown, in S401, based on the similarity between the first image and the second image and the first time and the second time, similar video segments of the first video and the second video are obtained.

[0129] According to an exemplary embodiment, according to the above, in the process of comparing the similarity between the first image and the second image, when the similarity between the two images obtained by comparison is greater than the above similarity threshold, for example, 0.8, the two images are determined to be similar. The two images are respectively stored in the first similar image sequence and the second similar image sequence and the time at which they are located in their respective videos is recorded respectively, and the two images are respectively removed from the extracted image sequence in which they are located.

[0130] According to an exemplary embodiment, the above process is repeated until the comparison of the first extracted image sequence and the second extracted image sequence is completed.

[0131] According to an example embodiment, the first similar image sequence is analyzed, and if the time interval between adjacent first images in the sequence is less than a preset scene segmentation time interval, they are divided into the same scene; if the time interval between adjacent first images in the sequence is not less than the preset scene segmentation time interval, the first image in the previous time is used as the end image of the previous scene, and the first image in the later time is used as the start image of the next scene. This process is repeated until the first similar image sequence is divided into scenes. Based on the time of the starting first image and the ending first image of each scene, similar scene video segments are divided in the first video, and the collection of all similar scene video segments is the first similar video segment of the first video.

[0132] According to an exemplary embodiment, the second similar image sequence is processed by the steps consistent with the above to obtain a second similar video segment.

[0133] In S403, based on the duration of the similar video segments and the duration of the first video and the second video, the similarity between the first video and the second video is obtained.

[0134] According to the example embodiment, following the previous steps, calculate: the ratio of the length of the first similar video segment to the length of the first video, and the ratio of the second similar video segment to the length of the second video, and take the maximum value of the above two ratios as the similarity between the first video and the second video.

[0135] According to an example embodiment, as long as similar video clips are found in the first video and the second video, it can be determined that the second video infringes upon the first video. The similarity between the first video and the second video is calculated in order to quantify the degree of infringement.

[0136] Figure 5 A block diagram of a device for comparing video similarities according to an example embodiment is shown.

[0137] like Figure 5 As shown, the apparatus for comparing video similarity includes: an acquisition module 501, a frame extraction module 502, a first similarity comparison module 503, a judgment module 505, a second similarity comparison module 507, and a video similarity calculation module 509, wherein:

[0138] The acquisition module 501 is used to acquire the first video and the second video.

[0139] The frame extraction module 502 is used to perform a frame extraction operation on the first video to obtain a first image, and to perform a frame extraction operation on the second video to obtain a second image.

[0140] The first similarity comparison module 503 is used to perform a similarity comparison process on the first image and the second image to obtain a first comparison similarity.

[0141] The judgment module 505 is used to judge whether the first comparison similarity is not less than a preset similarity threshold, and then use the first comparison similarity as the similarity between the first image and the second image; and to judge whether the first comparison similarity is less than the preset similarity threshold, and then switch to the second similarity comparison module; and to judge whether the second comparison similarity is less than the first comparison similarity, and then use the first comparison similarity as the similarity between the first image and the second image; and to judge whether the second comparison similarity is not less than the first comparison similarity, and then use the second similarity as the similarity between the first image and the second image.

[0142] The second similarity comparison module 507 is used to perform feature point comparison processing on the first image and the second image to obtain a second comparison similarity.

[0143] The video similarity calculation module 509 obtains the similarity between the first video and the second video based on the similarity between the first image and the second image.

[0144] The device performs functions similar to the method provided above. For other functions, please refer to the previous description and will not be repeated here.

[0145] Figure 7 A block diagram of an electronic device according to an exemplary embodiment is shown.

[0146] Refer to the following Figure 7 hereinafter describes an electronic device 700 according to this embodiment of the present application. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0147] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710), a display unit 740, etc.

[0148] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 executes the methods described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 710 can execute the methods described above.

[0149] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .

[0150] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0151] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0152] The electronic device 700 may also communicate with one or more external devices 7001 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. The network adapter 760 may communicate with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0153] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. The technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation method of the present application.

[0154] The software product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0155] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0156] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0157] The computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the computer-readable medium implements the aforementioned functions.

[0158] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0159] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present application.

[0160] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, changes or deformations made by those skilled in the art based on the ideas of the present application, the specific implementation methods and the scope of application of the present application, all belong to the scope of protection of the present application. In summary, the content of this specification should not be construed as a limitation on the present application.

Claims

1. A method for comparing video similarity, characterized in that: include: Get the first video and the second video; Performing a frame extraction operation on the first video to obtain a first image, and performing a frame extraction operation on the second video to obtain a second image; Performing a similarity comparison process on the first image and the second image to obtain a first comparison similarity; If it is determined that the first comparison similarity is not less than a preset similarity threshold, the first comparison similarity is used as the similarity between the first image and the second image; If the first comparison similarity is less than the preset similarity threshold, then Performing feature point comparison processing on the first image and the second image to obtain a second comparison similarity; If it is determined that the second comparison similarity is less than the first comparison similarity, the first comparison similarity is used as the similarity between the first image and the second image; If it is determined that the second comparison similarity is not less than the first comparison similarity, the second comparison similarity is used as the similarity between the first image and the second image; The similarity between the first video and the second video is obtained based on the similarity between the first image and the second image.

2. The method according to claim 1, characterized in that The performing similarity comparison processing on the first image and the second image to obtain a first comparison similarity includes: grayscale the first image and reduce it to a predetermined size to obtain a first converted image; grayscale the second image and reduce it to the predetermined size to obtain a second converted image; Processing the first converted image using a mean hash algorithm to obtain a first hash fingerprint; Processing the second transformed image using a mean hash algorithm to obtain a second hash fingerprint; The first hash fingerprint and the second hash fingerprint are compared to obtain the first comparison similarity.

3. The method according to claim 1, characterized in that The performing feature point comparison processing on the first image and the second image to obtain a second comparison similarity includes: Performing feature point extraction processing on the first image to obtain first feature points, where the first feature points are points on the first image; Perform feature point extraction processing on the second image to obtain second feature points, where the second feature points are points on the second image.

4. The method according to claim 3, characterized in that Also includes: Determine the number of the first feature points or Whether the number of the second feature points is less than a preset lower limit of the number of feature points; If it is less than, taking the first comparison similarity as the similarity between the first image and the second image; If it is not less than, feature point matching processing is performed on the first feature point and the second feature point to obtain similar feature points, where the similar feature points are similar feature points between the first feature point and the second feature point.

5. The method according to claim 4, characterized in that After performing feature point matching processing on the first feature point and the second feature point to obtain similar feature points, the method further includes: Determine whether the number of similar feature points is less than a preset lower limit of the number of similar feature points; If it is less than, obtaining the second comparison similarity by the number of the similar feature points, the number of the first feature points, and the number of the second feature points; If it is not less than, then a first similar region is obtained in the first image based on the similar feature points, and A second similar area is obtained in the second image.

6. The method according to claim 5, characterized in that After obtaining a first similar region in the first image based on the similar feature points and obtaining a second similar region in the second image, the method further includes: Obtaining a first feature point similarity by using an area of ​​the first similar region and an area of ​​the first image; The second feature point similarity is obtained by comparing the area of ​​the second similar region with the area of ​​the second image.

7. The method according to claim 6, characterized in that Also includes: Comparing the similarity of the first feature point with the similarity of the second feature point; The larger value is taken as the second comparison similarity.

8. The method according to claim 1, characterized in that The step of performing a frame extraction operation on the first video to obtain a first image and performing a frame extraction operation on the second video to obtain a second image includes: A first time of the first image in the first video is recorded, and a second time of the second image in the second video is recorded.

9. The method according to claim 8, characterized in that The obtaining the similarity between the first video and the second video based on the similarity between the first image and the second image includes: Based on the similarity between the first image and the second image and the first time and the second time, obtaining a first similar video segment of the first video and a second similar video segment of the second video; Based on the duration between the first similar video segment and the first video and the duration between the second similar video segment and the second video, the similarity between the first video and the second video is obtained.

10. A device for comparing video similarity, characterized in that: include: An acquisition module is used to acquire the first video and the second video; a frame extraction module, performing a frame extraction operation on the first video to obtain a first image, and performing a frame extraction operation on the second video to obtain a second image; A first similarity comparison module performs a similarity comparison process on the first image and the second image to obtain a first comparison similarity; A judgment module, which judges that if the first comparison similarity is not less than a preset similarity threshold, then uses the first comparison similarity as the similarity between the first image and the second image; The judging module further judges that if the first comparison similarity is less than the preset similarity threshold, then transfers to the second similarity comparison module; The second similarity comparison module performs feature point comparison processing on the first image and the second image to obtain a second comparison similarity; The judging module further judges that if the second comparison similarity is less than the first comparison similarity, then the first comparison similarity is used as the similarity between the first image and the second image; The judging module further judges that if the second comparison similarity is not less than the first comparison similarity, then the second comparison similarity is used as the similarity between the first image and the second image; The video similarity calculation module obtains the similarity between the first video and the second video based on the similarity between the first image and the second image.

11. An electronic device, characterized in that it comprises one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

Images