Video classification method and device

Abstract

The present application provides a method and device for video classification, the method comprising: receiving a video from a video source; obtaining M video frames P from the video; 1 , P 2 ,…,P M and M adjacent frames Q 1 , Q 2 , …, Q M , where the adjacent frame Q 1 is the video frame P 1 The adjacent video frames, the adjacent frames Q 2 is the video frame P 2 adjacent video frames, ..., the adjacent frames Q M is the video frame P M Adjacent video frames; Determine the distance D of M videos 1 , D 2 , …, D M , where the video distance D 1 is the video frame P 1 and adjacent frame Q 1 The Hamming distance between them, video distance D 2 is the video frame P 2 and adjacent frame Q 2 The Hamming distance between, ..., video distance D M is the video frame P M and adjacent frame Q M The Hamming distance between the M videos is D 1 , D 2 , …, D M are compared with the threshold values ​​to obtain the comparison result {H 1 , H 2 , ..., H M}; According to the comparison result {H 1 , H 2 , ..., H M}Determine the classification result to which the video belongs; compress the video according to the classification result and send it to the content distribution system.

Description

Technical Field

[0001] The present application relates to the field of computer multimedia, and in particular to a method and device for video classification. Background Art

[0002] Currently, live streaming and on-demand video are generally achieved by uploading video sources or live streaming sources to the live streaming / on-demand platform, and then distributing them to viewers who need to watch these video sources or live streaming sources through the content delivery network (CDN) of major operators. Among them, the operator fee for using CDN for data distribution is the main expenditure of the live streaming / on-demand platform. The larger the file size of the video source or live streaming source and the video stream bit rate, the higher the operating cost of the live streaming / on-demand platform. Therefore, the live streaming / on-demand platform generally classifies the video source or live streaming source into low-dynamic video and high-dynamic video through a video classification algorithm, and compresses different types of videos at different bit rates, so as to compress the size of the video file and the bit rate of the video stream as much as possible without affecting the user's viewing experience, thereby reducing the operating cost of the live streaming / on-demand platform.

[0003] Video classification algorithms have developed rapidly in recent years. For example, content classification algorithms based on support vector machines (SVM) have been applied in multiple scenarios. However, AI video classification algorithms require a large amount of computing resources for training and predicting neural network models, occupying a large amount of system resources on live / on-demand platforms, resulting in freezes in live or on-demand video, and reducing the user's viewing experience. Summary of the invention

[0004] The present application provides a method, apparatus and related equipment for video classification, which are used to solve the problems of consuming a large amount of computing resources when performing video classification on a live / on-demand platform, resulting in video live or on-demand freezes and high platform operating costs.

[0005] In a first aspect, a method for video classification is provided, the method comprising:

[0006] receiving video from a video source;

[0007] Get M video frames P1, P2, ..., P from the video M and M adjacent frames Q1,Q2,…,Q M , wherein the value of M is a positive integer, the adjacent frame Q1 is the adjacent video frame of the video frame P1, the adjacent frame Q2 is the adjacent video frame of the video frame P2, ..., the adjacent frame Q M is the video frame P M Adjacent video frames;

[0008] Determine the distances D1, D2, ..., D of M videos M , where video distance D1 is the Hamming distance between video frame P1 and adjacent frame Q1, video distance D2 is the Hamming distance between video frame P2 and adjacent frame Q2, ..., video distance D M is the video frame P M and adjacent frame Q M The Hamming distance between

[0009] Set the M video distances D1, D2, ..., D M are compared with the thresholds respectively, so as to obtain the comparison results {H1,H2,...,H M};

[0010] According to the comparison result {H1, H2, ..., H M}Determine the classification result to which the video belongs;

[0011] The video is compressed according to the classification result and sent to a content distribution system.

[0012] In one embodiment, according to the comparison result {H1, H2, ..., H M}Determine the classification result to which the video belongs, including: comparing the comparison results {H1, H2, ..., H M} to obtain the comparison results {H1,H2,...,H M}, and the comparison results {H1, H2, ..., H M}a second number n2 in which the video distance is not less than the threshold; and determining the classification result to which the video belongs according to the first number n1 and the second number n2.

[0013] In one embodiment, the classification result of the video is determined based on the first number n1 and the second number n2, including: when the ratio a1 between the n1 and the M is not less than the first ratio, the video is classified as a low-motion video; when the ratio a2 between the n2 and the M is not less than the second ratio, the video is classified as a high-motion video; when the ratio a1 between the n1 and the M is less than the first ratio, and the ratio a2 between the n2 and the M is less than the second ratio, the video is classified as a mixed video, wherein the picture change degree of the mixed video is higher than that of the low-motion video and lower than that of the high-motion video.

[0014] In one implementation, the threshold is a value determined according to a video distance between two dissimilar frames.

[0015] In one embodiment, the first ratio is determined based on the ratio of the number of video distances less than the threshold in the low-dynamic video samples to the total number of frames M; the second ratio is determined based on the ratio of the number of video distances not less than the threshold in the high-dynamic video samples to the total number of frames.

[0016] In one embodiment, the time interval between the adjacent frame Q1 and the video frame P1 is t, the time interval between the adjacent frame Q2 and the video frame P2 is t, ..., the adjacent frames Q M With the video frame P M The time interval is t, wherein the time interval t is determined according to the total duration of the video.

[0017] Implementing the method described in the first aspect only requires occupying a small amount of system resources of the live / on-demand platform server to complete video classification and compress different types of videos at different bit rates, thereby compressing the size of the signal source video file and the bit rate of the video stream as much as possible without affecting the user's viewing experience, thereby reducing the operating costs of the live / on-demand platform.

[0018] In a second aspect, a device for video classification is provided, the device comprising:

[0019] A receiving unit, the receiving unit is used to receive a video from a video source;

[0020] An acquisition unit, the acquisition unit is used to acquire M video frames P1, P2, ..., P from the video M and M adjacent frames Q1,Q2,…,Q M , wherein the value of M is a positive integer, the adjacent frame Q1 is the adjacent video frame of the video frame P1, the adjacent frame Q2 is the adjacent video frame of the video frame P2, ..., the adjacent frame Q M is the video frame P M Adjacent video frames;

[0021] A determination unit, the determination unit is used to determine M video distances D1, D2, ..., D M , where video distance D1 is the Hamming distance between video frame P1 and adjacent frame Q1, video distance D2 is the Hamming distance between video frame P2 and adjacent frame Q2, ..., video distance D M is the video frame P M and adjacent frame Q M The Hamming distance between

[0022] A comparison unit, the comparison unit is used to compare the M video distances D1, D2, ..., D M Compare with the threshold respectively to obtain the comparison result {H1,H2 , ...,HM};

[0023] A classification unit, the classification unit is used to classify the M}Determine the classification result to which the video belongs;

[0024] A compression unit is used to compress the video according to the classification result and send it to a content distribution system.

[0025] In one embodiment, the classification unit is used to: classify the comparison results {H1, H2, ..., H M} to perform statistics, thus obtaining the comparison results {H1,2,..., M}, and the comparison results {H1, H2, ..., H M}a second number n2 in which the video distance is not less than the threshold; and determining the classification result to which the video belongs according to the first number n1 and the second number n2.

[0026] In one embodiment, the classification unit is used to: classify the video as a low-motion video when the ratio a1 between the n1 and the M is not less than a first ratio; classify the video as the high-motion video when the ratio a2 between the n2 and the M is not less than a second ratio; classify the video as a mixed video when the ratio a1 between the n1 and the M is less than the first ratio and the ratio a2 between the n2 and the M is less than the second ratio, wherein the degree of picture change of the mixed video is higher than that of the low-motion video and lower than that of the high-motion video.

[0027] In one implementation, the threshold is a value determined according to a video distance between two dissimilar frames.

[0028] In one embodiment, the first ratio is determined based on the ratio between the number of video distances less than the threshold in the low-dynamic video sample and M; the second ratio is determined based on the ratio between the number of video distances not less than the threshold in the high-dynamic video sample and M.

[0029] In one embodiment, the time interval between the adjacent frame Q1 and the video frame P1 is t, the time interval between the adjacent frame Q2 and the video frame P2 is t, ..., the adjacent frames Q M With the video frame P M The time interval is t, wherein the time interval t is determined according to the total duration of the video.

[0030] According to a third aspect, a computer program product is provided. When the computer program product is read and executed by a computing device, the method described in the first aspect will be executed.

[0031] In a fourth aspect, a computer non-volatile storage medium is provided, comprising instructions, which, when executed on a computing device, cause the computing device to execute the method described in the first aspect.

[0032] In a fifth aspect, an electronic device is provided, comprising a processor and a memory, wherein the processor executes the code in the memory to execute the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0034] Figure 1 It is a schematic diagram of the system architecture of live video and on-demand video;

[0035] Figure 2 This is a flow chart of a method for video classification provided by this application;

[0036] Figure 3 This is a flow chart of a video classification method provided by this application;

[0037] Figure 4 It is a flowchart of a video classification method in an application scenario provided by the present application;

[0038] Figure 5 It is a verification process diagram of a video classification method provided by this application;

[0039] Figure 6 It is a structural schematic diagram of a video classification device provided by the present application;

[0040] Figure 7 It is a structural schematic diagram of an electronic device provided by this application. DETAILED DESCRIPTION

[0041] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0042] In order to facilitate understanding of the embodiments of the present application, some terms involved in the present application are first explained.

[0043] A Bernoulli Experiment is a random experiment that is repeated and mutually independent under the same conditions. Its characteristic is that there are only two possible outcomes for this random experiment: occurrence or non-occurrence. Suppose this experiment is independently repeated n times, then this series of repeated and independent random experiments is called an n-fold Bernoulli experiment, or a Bernoulli probability model.

[0044] Simply put, for any repeated experiment under the same conditions, the possibility of each result occurring is not affected by the occurrence of the results of other experiments, then it can be a Bernoulli experiment. For example, a Bernoulli experiment can be taking out a card from a deck of cards, which is either black or red; it can also be delivering a baby, which is either a boy or a girl; it can also be encountering a meteor or not encountering a meteor in a 24-hour day, etc. Among them, in a single Bernoulli experiment, the probability of event A occurring is p (0 < p < 1), and the probability of not occurring is (1 - p). Then, in an n-fold Bernoulli experiment, if X represents the number of times event A occurs, the probability distribution of X is a binomial distribution, denoted as X ~ (n, p), and the probability mass function of X can be shown as in Formula (1) and Formula (2):

[0045]

[0046]

[0047] That is to say, for an n-fold Bernoulli experiment, when knowing that the probability of event A occurring is p, the probability of event A occurring k times can be calculated according to Formula (1). The probability of event A occurring at least k times can be calculated according to Formula (2).

[0048] Secondly, a brief description is given to the system architecture of video live broadcast and on-demand involved in this application.

[0049] As Figure 1 shown, the system architecture of the video live broadcast and on-demand includes a signal source 110, a live broadcast platform server 120, an on-demand platform server 130, a CDN 140, a player 150, and a viewer 160. Among them, Figure 1 Taking the application scenario where multiple viewers in four cities of Beijing, Shanghai, Guangzhou, and Shenzhen simultaneously watch the signal source 110 as an example for illustration. In specific implementation, the system architecture of the video live broadcast and on-demand may include more signal sources 110, more live broadcast platform servers 120, more on-demand platform servers 140, more players 150, and more viewers 160, which are not specifically limited in this application.

[0050] The signal source 110 may be a video source or a live broadcast source published by an uploader, wherein the video source may be a self-media video uploaded by a self-media, a live broadcast recorded video uploaded by a host, user generated content (UGC), and a copyrighted video uploaded by a copyright holder, etc.; the live broadcast source may be a live broadcast source published by a live broadcast content provider such as a host, a teacher, or a sports event organizer. It should be understood that the above examples are for illustration only and are not specifically limited in this application.

[0051] The live broadcast platform server 120 and the on-demand platform server 130 can perform video transcoding on the signal source 110 to obtain a media stream that can be identified by the CDN 140. In a specific implementation, the live broadcast platform server 120 and the on-demand platform server 130 can also perform content sorting, copyright encryption, video pornography detection and other auditing on the signal source 110, and then perform video transcoding on the audited signal source 110. Among them, video transcoding can be simply understood as converting signal sources of various formats into media stream formats supported by CDN network general protocols through encoding methods, such as Real Time Messaging Protocol (RTMP), dynamic bit rate adaptive technology (HTTP Live Streaming, HLS), streaming media protocol (HTTP-Flash Video, HTTP-FLV) and other commonly used streaming media transmission protocols of CDN, thereby increasing the available transmission channels for data transmission, rather than being limited to the specified interface channels, which can greatly improve work efficiency.

[0052] CDN 140 can send the streaming media segment files sent by the live platform server 120 and the on-demand platform server 130 to the corresponding player 150. CDN is a network service model, usually provided by network operators. The cache server (Cache) in the CDN network can cache many media streams sent by the live platform server 120 and the on-demand platform server 130. When a viewer requests a certain content, the CDN network can select the cache that can be sent to the viewer the fastest, just like a website server close to the user responding to the access request of the local user, thereby ensuring the end-to-end service quality to a great extent.

[0053] The player 150 can process the received streaming media segment files and then present the signal source 110 to the viewer 160 again, thus achieving the following Figure 1As shown, audiences in cities such as Beijing, Shanghai, Guangzhou, and Shenzhen can simultaneously watch the video content of the same signal source 110 through mobile phone applications (Application, App), computer video software, live web pages, and other players 150. In a specific implementation, the player 150 may include the following: Figure 1 The App, computer video software, live web page, etc. shown may also include PC client, TV client, etc., which are not specifically limited in this application.

[0054] It is understandable that, since CDN 140 is usually provided by a network operator, the network distribution fee of CDN will become one of the main costs of the live platform and the on-demand platform for the live platform server 120 and the on-demand platform server 140. CDN is charged by traffic or bandwidth, that is, the larger the file size of the signal source 110 and the video bit rate (the number of data bits transmitted per unit time during data transmission), the greater the platform expenditure. Therefore, in the process of video transcoding the signal source 110, the live platform server 120 and the on-demand platform server 130 generally classify the signal source 110 according to the video content of the signal source 110 through a video classification algorithm, and compress different videos at different bit rates, so as to compress the size of the video file of the signal source 110 and the bit rate of the video stream as much as possible without affecting the user's viewing experience, thereby reducing the traffic cost of the subsequent CDN transmission and reducing the operating cost of the live / on-demand platform.

[0055] Video classification algorithms have developed rapidly in recent years, and most of them are based on artificial intelligence (AI). For example, content classification algorithms based on support vector machines have been applied in multiple scenarios. However, AI video classification algorithms require a large amount of computing resources for training and prediction of neural network models, occupying a large amount of system resources on live / on-demand platforms, resulting in freezes in live or on-demand video, and reducing the user's viewing experience.

[0056] In order to solve the problem that the above-mentioned video classification algorithm causes video live broadcast or on-demand freezes and poor user viewing experience, the present application proposes a video classification method that only needs to occupy a small amount of system resources of the live / on-demand platform server to complete video classification and compress different types of videos at different bit rates. In this way, the size of the signal source video file and the bit rate of the video stream can be compressed as much as possible without affecting the user's viewing experience, thereby reducing the operating cost of the live / on-demand platform.

[0057] Figure 2 This application provides a method for video classification. Figure 2 It can be seen that the specific process of the video classification method provided by this application can be as follows:

[0058] S201: Receive video A from a video source, where the video source can be Figure 1 The signal source in the video may be video A which needs to be compressed by the live / on-demand platform and then sent to CDN 140 for distribution.

[0059] S202: Obtain M video frames P1, P2, ..., P from video A M and M adjacent frames Q1,Q2,…,Q M .

[0060] Wherein, the value of M is a positive integer, the adjacent frame Q1 is the adjacent video frame of the video frame P1, the adjacent frame Q2 is the adjacent video frame of the video frame P2, ..., the adjacent frame Q M is the video frame P M adjacent video frames.

[0061] S203: Determine M video distances D1, D2, ..., D M .

[0062] Wherein, video distance D1 is the Hamming distance between video frame P1 and adjacent frame Q1, video distance D2 is the Hamming distance between video frame P2 and adjacent frame Q2, ..., video distance D M is the video frame P M and adjacent frame Q M The Hamming distance between .

[0063] S204: Set the M video distances D1, D2, ..., D M are compared with the thresholds respectively, so as to obtain the comparison results {H1,H2,...,H M}.

[0064] S205: Based on the comparison result {H1, H2, ..., H M}Determine the classification result to which the video A belongs.

[0065] S206: compress the video according to the classification result and send it to a content distribution system. The content distribution system may be Figure 1 That is, after the videos are classified, different types of videos are compressed at different bit rates, and then the compressed videos are sent to CDN 140, wherein the bit rate of the low dynamic video after compression is lower than the bit rate of the high dynamic video or the mixed video after compression, and the bit rate of the mixed video after compression is between the low dynamic video and the high dynamic video. Thus, the size of the signal source video file and the bit rate of the video stream are compressed as much as possible without affecting the user's viewing experience, thereby reducing the operating cost of the direct / on-demand platform.

[0066] In an embodiment of the present application, the classification result to which a video belongs can be determined based on the inter-frame similarity between the M video frames and adjacent frames in the video. It is understandable that a slide video has many identical frames within the same second, while an animation video has many different frames within the same second. In other words, the inter-frame similarity of a low-dynamic video (such as a slide video) is much lower than the inter-frame similarity of a high-dynamic video (such as an animation video). Therefore, by calculating the inter-frame similarity between the M video frames and adjacent frames of the video, the video can be divided into multiple categories.

[0067] In a specific implementation, the number of categories of the classification result of the video A can be determined according to the video content features. For example, taking educational videos as an example, educational videos can be divided into low-dynamic videos, high-dynamic videos and mixed videos, among which low-dynamic videos can be videos of slide content, and the screen of such videos is almost entirely slides, and the teacher only teaches in the form of voice-over. Such videos have many identical frames in the same second, and the inter-frame similarity is very high; high-dynamic videos can be videos of teaching content, and the screen of such videos is almost entirely teachers teaching in front of the blackboard, so the frames of such videos in the same second are different, and the inter-frame similarity is very low; part of the video screen in the mixed video is a slide video, and part is a teaching video, so the inter-frame similarity of such videos is between low-dynamic videos and high-dynamic videos. It should be understood that other types of videos, such as game videos, advertising videos, etc., all have their own characteristics, and the number of categories for classification can also be determined according to the video content features in a specific implementation, and this application does not make specific limitations on this. However, in order to make the present application better understood, the video classification method provided by the present application will be explained below by taking video A as an educational video as an example. That is to say, the video A will be divided into low-motion video, high-motion video and mixed video as an example for explanation.

[0068] It should be noted that the inter-frame similarity of video A can be obtained by calculating the M video frames P1, P2, ..., P M and M adjacent frames Q1,Q2,…,Q M The Hamming distance can be obtained by converting each video frame and its adjacent frames into a Differential Hash (dHash) value, and then comparing the dHash values ​​of each video frame and its adjacent frames to see if they are consistent.

[0069] In a specific implementation, the steps of calculating the inter-frame similarity of the video A may be as follows:

[0070] First, the M video frames and M adjacent frames are resized. For example, the image size of the video frame and its adjacent frames is reduced to 3×3, so that the details of the image are hidden and only the basic features such as structure and brightness are retained to facilitate the subsequent calculation of the dHash value.

[0071] Secondly, the reduced M video frames and their M adjacent frames are subjected to simplified color processing to obtain M grayscale video frames and their M grayscale adjacent frames. That is to say, the images of each video frame and its adjacent frames are converted into grayscale images, which greatly reduces the amount of calculation and thus improves the speed of calculating the similarity between frames.

[0072] Next, the difference value of each grayscale image is calculated to obtain M difference values ​​of M video frames and M difference values ​​of M adjacent frames. Specifically, the color intensity of each pixel in each grayscale image can be compared with the color intensity of the previous pixel. If the color intensity of the previous pixel is greater than that of the second pixel, the difference value is set to "1", and if the color intensity of the previous pixel is not greater than that of the second pixel, the difference value is set to "0", thereby obtaining M difference values ​​of M video frames and M difference values ​​of M adjacent frames.

[0073] Fourth, each value in the difference value between each video frame and its adjacent frames is regarded as a bit, and each 8 bits are grouped into a hexadecimal value. The hexadecimal values ​​are concatenated and converted into a string, so as to obtain the dHash value of each video frame and its adjacent frames.

[0074] Finally, compare each character of the dHash value of each video frame and its adjacent frames, and record the number of different characters in the corresponding positions of the two dHash values ​​as the Hamming distance D between the video frame and its adjacent frames. For example, the dHash value of video frame P1 is 11001101, and the dHash value of its adjacent frame Q1 is 11000011. At this time, only the fifth, sixth and seventh bits of the dHash values ​​of video frame P1 and adjacent frame Q1 are different, so the Hamming distance D between video frame P1 and adjacent frame Q1 is 3. It should be understood that the above examples are only for illustration and do not constitute specific limitations. The image size of the video frame and its adjacent frames can also be reduced to other sizes, such as 8×8, and the dHash value can also be 64 bits. This application does not make specific limitations on this.

[0075] The M video frames may be randomly selected, and the time interval t between each video frame and its adjacent frame is determined according to the total duration of the video A. That is, the time interval between the adjacent frame Q1 and the video frame P1 is t, the time interval between the adjacent frame Q2 and the video frame P2 is t, ..., the adjacent frames Q M With the video frame P MThe time interval t is determined according to the total duration of the video. In a specific implementation, the time interval t is a relatively small value relative to the total duration of the video A, and can be tens or hundreds of the total duration. For example, if the total duration of an educational video A is 1800 seconds, then the time interval t can be 2 seconds. The specific process of extracting 20 video frames and 20 adjacent frames can be as follows: First, randomly extract 20 time points t on the timeline of video A. i , where i = 1, 2, ..., 20, then 20 video frames can be at time point t i 1 frame of image extracted at time t, and 20 adjacent frames can be extracted at time point t i +t, or at time point t i It should be understood that the above examples are only for illustration and do not constitute a specific limitation.

[0076] It is understandable that the inter-frame similarity of video A can also be obtained by other calculation methods for calculating inter-frame similarity, such as: Peak Signal to Noise Ratio (PSNR), histogram calculation, structural similarity (SSIM), mean hash algorithm, etc., and this application does not specifically limit this. However, since the video classification algorithm provided in this application is applied to the live / on-demand platform, and is specifically applied to the application scenario where the live / on-demand platform receives the video file sent by the signal source and sends the video to the CDN network, the video classification algorithm provided in this application needs to consume very low computing resources in order to reduce the impact on the operation of the live and on-demand platforms. Compared with other methods for calculating the similarity between frames, the calculation of the Hamming distance between video frames and adjacent frames has the advantages of low computing resource consumption, faster speed and higher accuracy.

[0077] In the embodiment of the present application, video A is classified according to its inter-frame similarity. It is necessary to determine the numerical range of the inter-frame similarity of each type of video. If the inter-frame similarity of video A belongs to the numerical range of the inter-frame similarity of low-dynamic video, video A can be classified as low-dynamic video. Similarly, if the inter-frame similarity of video A belongs to the numerical range of the inter-frame similarity of high-dynamic video or mixed video, video A can be classified as high-dynamic video or mixed video. In order to avoid accidental phenomena and reflect the extensiveness of the results, the inter-frame similarity of video A is calculated by extracting M video frames and M adjacent frames and calculating their Hamming distances. The inter-frame similarity of video A is a sequence D1, D2, ..., D containing M Hamming distances. MTherefore, in the actual processing, when classifying video A according to its inter-frame similarity, if most of the M Hamming distances of video A are within the range of low-dynamic video, then video A can be classified as low-dynamic video. Similarly, if most of the M Hamming distances of video A are within the numerical range of high-dynamic video or mixed video inter-frame similarity, then video A can be classified as high-dynamic video or mixed video.

[0078] Therefore, according to the comparison result {H1, H2, ..., H M}Determine the classification result to which the video A belongs, including: M} to obtain the comparison results {H1,H2,...,H M}, and the comparison results {H1, H2, ..., H M} a second number n2 in which the video distance is not less than the threshold; and determining the classification result to which the video belongs according to the first number n1 and the second number n2. The classification result to which the video belongs determined according to the first number n1 and the second number n2 includes: when the ratio a1 between the n1 and the M is not less than the first ratio, classifying the video as a low-dynamic video; when the ratio a2 between the n2 and the M is not less than the second ratio, classifying the video as a high-dynamic video; when the ratio a1 between the n1 and the M is less than the first ratio, and the ratio a2 between the n2 and the M is less than the second ratio, classifying the video as a mixed video.

[0079] In a specific implementation, according to the comparison result {H1, H2, ..., H M The process of determining the classification result to which the video A belongs can be as follows: Figure 3 As shown, after calculating the M video distances D1, D2, ..., D M After that, firstly, according to the threshold D0, determine the number n1 of Hamming distances less than the threshold and the second number n2 not less than the threshold, and then calculate the ratio a1 between n1 and M, and the ratio a2 between n2 and M. If a1 is not less than the first ratio, video A is classified as a low-motion video. If a2 is not less than the first ratio, video A can be classified as a high-motion video. If a1 is less than the first ratio, or a2 is less than the first ratio, video A can be classified as a mixed video.

[0080] For example, if Figure 4As shown, assuming that the threshold D0=5, the first ratio k1=0.8, the second ratio k2=0.8, and assuming that video A has 20 frames, the specific process of the video classification method of the present application can be as follows:

[0081] Step 1: Video A randomly selects 5 video frames P1, P2, ..., P5 and 5 adjacent frames Q1, Q2, ..., Q5, where the time interval between each video frame and its adjacent frames is t = 2s.

[0082] Step 2: Calculate the Hamming distance D between each video frame and its adjacent frames, and obtain five Hamming distances D1, D2, …, D5. In the specific implementation, Figure 3 As shown, the fingerprint codes of each video frame and adjacent frames can be calculated first. According to the fingerprint code of video frame P1 being 1000111 and the fingerprint code of adjacent frame Q1 being 1110111, the Hamming distance D1=2 between video frame P1 and its adjacent frame Q1 can be obtained. Similarly, the Hamming distances D2 to D5 between other video frames and adjacent frames can be obtained. The specific process of obtaining the fingerprint code can be referred to the above content, which will not be repeated here.

[0083] Step 3: Compare the five Hamming distances with the threshold value D0 respectively, the number of Hamming distances n1=4 that are less than the threshold value D0=5, and the number of Hamming distances n2=1 that are not less than the threshold value D0=5, so according to n1 / M=4 / 5=0.8, which is not less than the first ratio k1=0.8, the video A can be determined as a low-motion video, and the video is compressed according to the classification result and sent to the CDN 140. The bit rate of the low-motion video after compression is less than the bit rate of the high-motion video or the mixed video after compression, and the bit rate of the mixed video after compression is between the low-motion video and the high-motion video.

[0084] It can be understood that if the number of videos n1 less than the threshold D0 is 1, the number of videos n2 not less than the threshold D0 is 4, the ratio a2=4 / 5=0.8 between n2 and the M is not less than the second ratio k2=0.8, so video A can be classified as a high-dynamic video at this time; if the number of videos n1 less than the threshold D0 is 2, the number of videos n2 not less than the threshold D0 is 3, the ratio a1=2 / 5=0.4 between n1 and the M is less than the second ratio k2=0.8, and the ratio a2=3 / 5=0.6 between n2 and the M is less than the second ratio k2=0.8, video A can be classified as a mixed video at this time.

[0085] In the embodiment of the present application, the threshold D0 may be an empirical value. Specifically, the threshold D0 is a value determined according to the video distance between two dissimilar frames. By calculating the Hamming distances of multiple similar frames and dissimilar frames, it can be known that, generally speaking, the Hamming distance D of two similar frames does not exceed 5, and the Hamming distance of two dissimilar or completely different frames exceeds 5, so the threshold D0 may be 5. It should be understood that the above examples are for illustration only and do not constitute specific limitations.

[0086] In an embodiment of the present application, the first ratio k1 and the second ratio k2 may be empirical values. The first ratio is determined based on the ratio j1 between the number of video distances less than the threshold value and M in the low-dynamic video sample; the second ratio is determined based on the ratio j2 between the number of video distances not less than the threshold value and M in the high-dynamic video sample. That is, by calculating the video distances between M video frames and M adjacent frames in the low-dynamic video sample, and counting the ratio j1 between the number of video distances less than the threshold value (for example, D0=5) and M, the value of the first ratio k1 can be determined; by calculating the video distances between M video frames and M adjacent frames in multiple high-dynamic video samples, and counting the ratio j2 between the number of video distances not less than the threshold value (for example, D0=5) and M, the value of the second ratio k2 can be determined. For example, if the 10 video frames in the low-dynamic video sample X1 and the 10 video distances between the 10 adjacent frames are {1, 3, 2, 2, 1, 5, 8, 2, 1, 3}, wherein the ratio between the number of video distances less than the threshold value D0 = 5 and M is j1 = 8 / 10 = 0.8, based on this, the value of the first ratio k1 can be determined to be 0.8. Similarly, if the 10 video frames in the high-dynamic video sample Y1 and the 10 video distances between the 10 adjacent frames are {5, 8, 9, 6, 5, 4, 7, 3, 9, 6}, wherein the ratio between the number of video distances not less than the threshold value D0 = 5 and M is j2 = 8 / 10 = 0.8, based on this, the value of the second ratio k2 can be determined to be 0.8. It is understandable that in order to eliminate the contingency of the data set to a certain extent, the value of the first ratio k1 can also be determined to be 0.15, 0.25, etc., and in order to avoid the contingency of a single sample, multiple low-motion video samples X1, X2, ..., X x The ratio j of each low-motion video sample in 1x , the first ratio k1 is determined by calculating the average, which is not specifically limited in this application.

[0087] It can be understood that the implementation of the above-mentioned video classification method only requires a small amount of system resources of the live / on-demand platform server to complete the video classification and compress different types of videos at different bit rates, thereby compressing the size of the signal source video file and the bit rate of the video stream as much as possible without affecting the user's viewing experience, thereby reducing the operating costs of the live / on-demand platform.

[0088] In order to prove the accuracy of the classification results of the video classification method provided by this application, the following Figure 5 , the video classification method provided by the present application is verified. Wherein, the threshold D0=5, the first ratio k1=0.8, and the second ratio k2=0.8, then the specific steps for verifying the classification method provided by the present application can be as follows:

[0089] 1. Obtain a sample set, which includes 53 low-motion videos with low-motion video labels, 21 high-motion videos with high-motion video labels, and 26 mixed videos with mixed video labels.

[0090] 2. Extract N video frames and N adjacent frames from each video in the sample set. Specifically, 53 low-dynamic videos, 21 high-dynamic videos, and 26 mixed videos can be extracted by the ffmpeg program. In order to avoid the particularity brought by random extraction, the frame extraction here refers to extracting a frame every 2 seconds from the beginning to the end of the video, and two adjacent frames of images are a group of video frames and adjacent frames.

[0091] 3. Calculate the video distances between the N video frames and the N adjacent frames of each video in the sample set. In a specific implementation, the N Hamming distances D of the first video can be obtained by executing step S202 in the above content. 11 ,D 12 ,...,D 1n , the N Hamming distances of the second video calculate the Hamming distance D 21 ,D 22 ,...,D 2n And so on, wherein the specific method of calculating the Hamming distance can refer to the aforementioned embodiment, which will not be described in detail here.

[0092] 4. Compare the Hamming distance of each low-motion video in the 53 low-motion videos with the threshold D0=5, and obtain the ratio c between the number of Hamming distances of the x-th low-motion video that are less than the threshold 5 and N. x , where x = 1, 2, ..., 53, and c x The minimum value is 0.84 and the average value is 0.94.

[0093] 5. Compare the Hamming distance of each high-dynamic video in the 21 high-dynamic videos with the threshold D0=5, and obtain the ratio d between the number of Hamming distances of the yth high-dynamic video that are less than the threshold 5 and N. y , where y = 1, 2, ..., 21, and d y The minimum value is 0.87 and the average value is 0.93.

[0094] 6. The calculation of the Hamming distance of the low-motion video or the high-motion video is recorded as a Bernoulli trial, wherein the Hamming distance of the low-motion video is less than 5 or the Hamming distance of the high-motion video is not less than 5 is recorded as the occurrence of event A in the Bernoulli trial. Then the probability of event A occurring can be p = (0.94 + 0.93) / 2 ≈ 0.9, and event A obeys the binomial distribution B ~ (n, p).

[0095] 7. Use steps S201-S205 to classify the 100 videos in the sample set, where the threshold D0=5, the first ratio k1=0.8, the second ratio k2=0.8, and make 10 predictions for each video and record the accuracy of each prediction, and calculate the average. The measured results show that the measured accuracy of 53 low-dynamic videos is 97.4%, and the measured accuracy of 21 high-dynamic videos is 93.3%.

[0096] It should be understood that since the mixed video is not used for calculation in steps 2 to 6, only step 7 calculates the mixed video, so Figure 5 The 26 mixed video labels are not plotted, but in practice, the measured accuracy obtained in step 7 is the accuracy obtained after 10 predictions for all samples.

[0097] 8. According to the probability formula of binomial distribution, i.e., formula (2) in the above content, the probability value of event A being repeated 10 times is calculated, and the theoretical accuracy of correctly classifying the video sample set using the threshold D0=5, the first ratio k1=0.8, and the second ratio k2=0.8 is obtained:

[0098]

[0099] The theoretical accuracy of 0.93 is basically consistent with the measured result in step 7. Therefore, the video classification method provided by the present application has a high accuracy rate.

[0100] It is understandable that, since the number of extracted frames and the calculated Hamming distance are only very small compared to the entire video (accounting for only tens or hundreds of one percent), the use of the binomial distribution probability calculation method can eliminate the randomness of the data set to a certain extent, making the calculation results closer to the actual results and greatly reducing the error. In addition, the first ratio k1 and the second ratio k2 can also be confirmed and adjusted based on the theoretical accuracy and measured accuracy after classifying the sample set with classification labels. Specifically, the threshold can be adjusted when the measured accuracy or theoretical accuracy is lower than the first expected value, or the threshold can be adjusted when the difference between the measured accuracy and the theoretical accuracy is greater than the second expected value.

[0101] Combined with the above Figure 1-Figure 5 The following describes the related devices involved in the embodiments of the present application. Figure 6 , Figure 6 6 is a schematic diagram of the structure of a video classification device 600 provided in the present application. The device 600 includes a receiving unit 610, an acquiring unit 620, a determining unit 630, a comparing unit 640, a classification unit 650 and a compression unit 660, wherein:

[0102] The receiving unit 610 is used to receive a video from a video source;

[0103] The acquisition unit 620 is used to acquire M video frames P1, P2, ..., P from the video. M and M adjacent frames Q1,Q2,…,Q M , wherein the value of M is a positive integer, the adjacent frame Q1 is the adjacent video frame of the video frame P1, the adjacent frame Q2 is the adjacent video frame of the video frame P2, ..., the adjacent frame Q M is the video frame P M Adjacent video frames;

[0104] The determining unit 630 is used to determine M video distances D1, D2, ..., D M , where video distance D1 is the Hamming distance between video frame P1 and adjacent frame Q1, video distance D2 is the Hamming distance between video frame P2 and adjacent frame Q2, ..., video distance D M is the video frame P M and adjacent frame Q M The Hamming distance between

[0105] The comparison unit 640 is used to compare the M video distances D1, D2, ..., D M are compared with the first threshold respectively, so as to obtain the comparison results {H1, H2, ..., H M};

[0106] The classification unit 650 is used to classify the comparison results {H1, H2, ..., H M}Determine the classification result to which the video belongs;

[0107] The compression unit 660 is used to compress the video according to the classification result and send it to the content distribution system.

[0108] Optionally, the classification unit 650 is used to: classify the comparison results {H1, H2, ..., H M} to obtain the comparison results {H1,H2,...,H M}, and the comparison results {H1, H2, ..., H M}a second number n2 in which the video distance is not less than the threshold; and determining the classification result to which the video belongs according to the first number n1 and the second number.

[0109] Optionally, the classification unit 650 is used to: classify the video as a low-motion video when the ratio a1 between the n1 and the M is not less than a first ratio; classify the video as the high-motion video when the ratio a2 between the n2 and the M is not less than a second ratio; classify the video as a mixed video when the ratio a1 between the n1 and the M is less than the first ratio and the ratio a2 between the n2 and the M is less than the second ratio, wherein the degree of picture change of the mixed video is higher than that of the low-motion video and lower than that of the high-motion video.

[0110] Optionally, the threshold is a value determined according to a video distance between two dissimilar frames.

[0111] Optionally, the first ratio is determined based on the ratio between the number of video distances less than the threshold in the low-dynamic video sample and M; the second ratio is determined based on the ratio between the number of video distances not less than the threshold in the high-dynamic video sample and M.

[0112] Optionally, the time interval between the adjacent frame Q1 and the video frame P1 is t, the time interval between the adjacent frame Q2 and the video frame P2 is t, ..., the adjacent frames Q M With the video frame P M The time interval is t, wherein the time interval t is determined according to the total duration of the video.

[0113] In practical applications, the device 600 of the embodiment of the present application can be implemented by an application-specific integrated circuit (ASIC), or a programmable logic device (PLD), and the PLD can be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. It can also be implemented by software. When implemented by software, the functions of the device 600 and each module in the device can also be software modules.

[0114] The above-mentioned video classification device only needs to occupy a small amount of system resources of the live / on-demand platform server to complete video classification and compress different types of videos at different bit rates, thereby compressing the size of the signal source video file and the bit rate of the video stream as much as possible without affecting the user's viewing experience, thereby reducing the operating cost of the live / on-demand platform.

[0115] See also Figure 7 , Figure 7 700 is a schematic diagram of the structure of an electronic device 700 provided in an embodiment of the present application. The electronic device 700 may be the video classification device 600 or the live / on-demand platform in the aforementioned content, such as Figure 7 As shown, the electronic device 700 includes one or more processors 703, a communication interface 702 and a memory 701. The processor 703, the communication interface 702 and the memory 701 can be connected via a bus, or can communicate via other means such as wireless transmission. The embodiment of the present application takes the connection via bus 704 as an example, wherein the bus 704 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 704 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0116] The memory 701 may include a volatile memory (Volatile Memory), such as a random access memory (Random Access Memory, RAM); the memory 701 may also include a non-volatile memory (NVM), such as a read-only memory (Read-Only Memory, ROM), a flash memory (Flash Memory, FM), a hard disk (Hard Disk Drive, HDD) or a solid-state drive (SSD). The memory 701 may also include a combination of the above-mentioned types of memories. Among them, the memory 701 may store program codes and program data. For example, program codes for calculating video distances, program codes for determining classification results, program codes for compressing classified videos, etc. are stored, and thresholds, first ratios, and second ratios, etc. are also stored.

[0117] The communication interface 702 may be a wired interface (e.g., an Ethernet interface), an internal interface (e.g., a high-speed serial computer expansion bus (Peripheral Component Interconnect express, PCIe) bus interface), a wired interface (e.g., an Ethernet interface), or a wireless interface (e.g., a cellular network interface or a wireless local area network interface), and is used to communicate with other devices or modules. For example, the communication interface 702 is used to support communication between the electronic device 700 and a content distribution network, and / or to perform other steps of the technology described herein, which will not be described in detail here.

[0118] The processor 703 includes one or more general-purpose processors, wherein the general-purpose processor can be any type of device capable of processing electronic instructions, including a central processing unit (CPU), a microprocessor, a microcontroller, a main processor, a controller, and an ASIC, etc. The processor 703 is used to execute various types of digital storage instructions, such as software or firmware programs stored in the memory 701, which enables the computing device to provide a wide variety of services.

[0119] Specifically, the processor 703 may include a classification module and a compression module, wherein the classification module may implement the function of video classification by calling the program code in the memory 701, and may include Figure 6 The functions described by the receiving unit 610, the acquiring unit 620, the determining unit 630, the comparing unit 640 and the classifying unit 650 in the above embodiment, such as extracting M video frames and M adjacent frames, calculating M video distances of the M video frames and the M adjacent frames, etc., can also be used to perform Figure 2-Figure 5The other steps described in the embodiment will not be repeated here; the compression module may include Figure 6 The functions described by the compression unit 660 in the above example, such as compressing the video according to the classification result, sending the compressed video to the CDN, etc., can also be used to perform Figure 2-Figure 5 The other steps described in the embodiment will not be described in detail here. Figure 2-Figure 5 The relevant explanations in the embodiments are not repeated here.

[0120] Need to explain, Figure 7 This is only a possible implementation of the embodiment of the present application. In practical applications, the electronic device may also include more or fewer components, which is not limited here. Figure 2-Figure 5 The relevant explanations in the embodiments are not repeated here.

[0121] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a processor, Figure 2-Figure 5 The method flow shown is realized.

[0122] The present application also provides a computer program product. When the computer program product is run on a processor, Figure 2-Figure 5 The method flow shown is realized.

[0123] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center containing one or more available media sets. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium. The semiconductor medium may be an SSD.

[0124] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for video classification, characterized in that: The method comprises: receiving video from a video source; Get M video frames P1, P2, ..., P from the video M and M adjacent frames Q1,Q2,…,Q M , wherein the value of M is a positive integer, the adjacent frame Q1 is the adjacent video frame of the video frame P1, the adjacent frame Q2 is the adjacent video frame of the video frame P2, ..., the adjacent frame Q M is the video frame P M Adjacent video frames; Determine the distances D1, D2, ..., D of M videos M , where video distance D1 is the Hamming distance between video frame P1 and adjacent frame Q1, video distance D2 is the Hamming distance between video frame P2 and adjacent frame Q2, ..., video distance D M is the video frame P M and adjacent frame Q M The Hamming distance between Set the M video distances D1, D2, ..., D M are compared with the thresholds respectively, so as to obtain the comparison results {H1,H2,...,H M }; According to the comparison result {H1, H2, ..., H M Determine the classification result to which the video belongs, the classification result including low-motion video, high-motion video and mixed video, the picture change degree of the mixed video is higher than that of the low-motion video and lower than that of the high-motion video; The video is compressed according to the classification result and sent to a content distribution system.

2. The method according to claim 1, characterized in that According to the comparison result {H1, H2, ..., H M }Determine the classification result to which the video belongs, including: For the comparison results {H1, H2, ..., H M } to perform statistics, thereby obtaining the comparison result {H1, H2, ..., H M }, and the comparison result {H1, H2, ..., H M } a second number n2 of video distances not less than the threshold; The classification result to which the video belongs is determined according to the first number n1 and the second number n2.

3. The method according to claim 2, characterized in that The step of determining the classification result to which the video belongs according to the first number n1 and the second number n2 includes: When a ratio a1 between n1 and M is not less than a first ratio, classifying the video as a low-motion video; When a ratio a2 between n2 and M is not less than a second ratio, classifying the video as a high dynamic video; When the ratio a1 between the n1 and the M is smaller than the first ratio, and the ratio a2 between the n2 and the M is smaller than the second ratio, the video is classified as a mixed video, wherein the degree of picture change of the mixed video is higher than that of the low-motion video and lower than that of the high-motion video.

4. The method according to claim 3, characterized in that The threshold is a value determined according to the video distance between two dissimilar frames.

5. The method according to claim 4, characterized in that The first ratio is determined according to the ratio of the number of video distances less than the threshold in the low-dynamic video samples to the total number of frames M; The second ratio is determined according to the ratio of the number of video distances not less than the threshold in the high dynamic video samples to the total number of frames.

6. The method according to any one of claims 1 to 5, characterized in that: The time interval between the adjacent frame Q1 and the video frame P1 is t, the time interval between the adjacent frame Q2 and the video frame P2 is t, ..., the adjacent frames Q M With the video frame P M The time interval is t, wherein the time interval t is determined according to the total duration of the video.

7. A video classification device, characterized in that: The device comprises: A receiving unit, the receiving unit is used to receive a video from a video source; An acquisition unit, the acquisition unit is used to acquire M video frames P1, P2, ..., P from the video M and M adjacent frames Q1,Q2,…,Q M , wherein the value of M is a positive integer, the adjacent frame Q1 is the adjacent video frame of the video frame P1, the adjacent frame Q2 is the adjacent video frame of the video frame P2, ..., the adjacent frame Q M is the video frame P M Adjacent video frames; A determination unit, the determination unit is used to determine M video distances D1, D2, ..., D M , where video distance D1 is the Hamming distance between video frame P1 and adjacent frame Q1, video distance D2 is the Hamming distance between video frame P2 and adjacent frame Q2, ..., video distance D M is the video frame P M and adjacent frame Q M The Hamming distance between A comparison unit, the comparison unit is used to compare the M video distances D1, D2, ..., D M are compared with the thresholds respectively, so as to obtain the comparison results {H1,H2,...,H M }; A classification unit, the classification unit is used to classify the M Determine the classification result to which the video belongs, the classification result including low-motion video, high-motion video and mixed video, the picture change degree of the mixed video is higher than that of the low-motion video and lower than that of the high-motion video; A compression unit is used to compress the video according to the classification result and send it to a content distribution system.

8. The device according to claim 7, characterized in that The taxonomic units are used for: For the comparison results {H1, H2, ..., H M } to perform statistics, thereby obtaining the comparison result {H1, H2, ..., H M }, and the comparison result {H1, H2, ..., H M } a second number n2 of video distances not less than the threshold; The classification result to which the video belongs is determined according to the first number n1 and the second number n2.

9. The device according to claim 8, characterized in that The taxonomic units are used for: When a ratio a1 between n1 and M is not less than a first ratio, classifying the video as a low-motion video; When a ratio a2 between n2 and M is not less than a second ratio, classifying the video as a high dynamic video; When the ratio a1 between the n1 and the M is smaller than the first ratio, and the ratio a2 between the n2 and the M is smaller than the second ratio, the video is classified as a mixed video, wherein the degree of picture change of the mixed video is higher than that of the low-motion video and lower than that of the high-motion video.

10. The device according to claim 9, characterized in that The threshold is a value determined according to the video distance between two dissimilar frames.

11. The device according to claim 10, characterized in that The first ratio is determined according to the ratio between the number of video distances less than the threshold value and M in the low-motion video samples; The second ratio is determined based on the ratio between the number of video distances not less than the threshold in the high dynamic video samples and M.

12. The device according to any one of claims 7 to 11, characterized in that The time interval between the adjacent frame Q1 and the video frame P1 is t, the time interval between the adjacent frame Q2 and the video frame P2 is t, ..., the adjacent frames Q M With the video frame P M The time interval is t, wherein the time interval t is determined according to the total duration of the video.

13. A computer program product, characterized in that When the computer program product is read and executed by a computing device, the method according to any one of claims 1 to 6 will be performed.

14. A computer non-transitory storage medium, characterized in that: The method comprises instructions, which, when executed on a computing device, cause the computing device to execute the method according to any one of claims 1 to 6.

15. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor executes the code in the memory to perform the method according to any one of claims 1 to 6.

Images