Video frame rate detection method and device based on visual perception

Abstract

The disclosure provides a visual perception-based video frame rate detection method and device, the method comprising: extracting the timestamp of each frame of the video, determining the adjacent frames of the video according to the timestamp; converting the RGB three channels of each frame of the video into a gray single channel, and calculating the gray pixel point value of the gray single channel; calculating the adjacent frame change rate of the video according to the proportion of the number of different gray pixel point values; calculating the adjacent frame content change rate of the video according to the proportion of the sum of the change difference of the gray pixel point value; detecting the relationship between the adjacent frame change rate of the video and the first stall threshold; detecting the relationship between the adjacent frame content change rate of the video and the second stall threshold; when the adjacent frame change rate is greater than or equal to the first stall threshold and the adjacent frame content change rate is greater than or equal to the second stall threshold, it is determined that the adjacent video frame is not stalled. By calculating the visual difference between the adjacent frames of the video to determine whether the video frame is a stall frame, the video stall can be accurately monitored.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of video detection, and particularly relates to a video frame rate detection method and device based on visual perception, an electronic device and a storage medium. BACKGROUND

[0002] Nowadays, video media has become a way of life or entertainment for everyone. When we watch a video, we often encounter video stuttering, frequent buffering signs or a situation of playing picture stuttering, which particularly affects the experience of watching. Most of us may think that the video stuttering is only caused by poor network speed. In fact, it is not entirely true. The common causes of video stuttering are as follows: the software and hardware configurations of the device are too low or the version of the playing software is too low. In the process of video playing, the video is encoded and decoded. High-definition video often brings decoding pressure to hardware. The stuttering caused by decoding is particularly obvious. The encoding and decoding speed will be reduced, which may cause the video playing to stutter.

[0003] Another situation is that the stuttering is caused by the change of adjacent video frames. The video frame rate is the number of frames displayed per second of a video, which directly determines the smoothness of video playing. When the video frame is slow and uneven, the human eye will perceive stuttering. When the video is higher than a certain threshold (generally 24fps) and the frame is uniform, the human eye generally cannot perceive stuttering. The higher the frame rate is above 24fps, the smoother the video is. However, in the process of adding filters, watermarks and the like to the video by using current video editing software, frame rate transcoding and copying frame mode are specified, which leads to a high video frame rate. Although the frame rate is high, the human eye still feels stuttering. The existing stuttering detection technology cannot monitor this stuttering situation and cannot accurately measure whether the video is stuttering. SUMMARY

[0004] In order to at least solve the above technical problems and improve the accuracy and efficiency of detecting video stuttering, the purpose of the embodiments of the present application is to provide a video frame rate detection method and device based on visual perception, an electronic device and a storage medium.

[0005] According to a first aspect of the present application, the embodiments of the present application provide a video frame rate detection method based on visual perception, comprising:

[0006] extracting a timestamp of each frame of a video, and determining adjacent frames of the video according to the timestamp;

[0007] converting the RGB three channels of each frame of the video into a gray single channel, and calculating a gray pixel point value of the gray single channel;

[0008] calculating a change rate of adjacent frames of the video according to a proportion of different quantities of the gray pixel point value;

[0009] According to the proportion of the sum of the change difference of the gray pixel point value, the adjacent frame content change rate of the video is calculated;

[0010] The relationship between the adjacent frame change rate of the video and the first stall threshold is detected;

[0011] The relationship between the adjacent frame content change rate of the video and the second stall threshold is detected; and

[0012] When the adjacent frame change rate is greater than or equal to the first stall threshold, and the adjacent frame content change rate is greater than or equal to the second stall threshold, it is determined that the adjacent video frame is not stalled.

[0013] Further, the timestamp of each frame of the video is the time point of each frame of the video after decoding, and the user calculates the playing time of each frame of the video, and determines the adjacent frame of the video according to the time point and the playing time of each frame.

[0014] Further, the RGB three channels of each frame of the video are converted into a gray single channel, and the gray pixel point value of the gray single channel is calculated, and the conversion formula is:

[0015] GRAY=R*0.299 + G*0.587 + B*0.114;

[0016] Wherein, GRAY is the gray pixel point value of the gray single channel, R is the red channel component, G is the green channel component, and B is the blue channel component.

[0017] Further, the adjacent frame change rate is the proportion of the number of corresponding gray pixel point values of the adjacent frame of the video which are not equal to the total number of pixel points, and the specific calculation formula is as follows:

[0018] ;

[0019] Wherein, H and W represent the height and width of the video frame, p ij and p ’ ij respectively represent the gray pixel point values of the adjacent frame image coordinates i , j , and the adjacent frame change rate ranges from 0 to 1.

[0020] Further, the adjacent frame content change rate refers to the proportion of the sum of the change difference of the corresponding gray pixel point value of the adjacent frame of the video and the number of corresponding gray pixel point values which are not equal, and the specific calculation formula is as follows:

[0021] ;

[0022] Wherein, H and W represent the height and width of the video frame, p ij and p’ ij Representing the coordinates of adjacent frames respectively ( i , j The grayscale pixel value is multiplied by 256 because there are 256 grayscale pixel values ​​in total, and the content change rate between adjacent frames is between 0 and 1.

[0023] Furthermore, the first lag threshold is set to 0.01 to 0.1, and the second lag threshold is set to 0.1 to 0.5.

[0024] Furthermore, the method also includes:

[0025] When the rate of change between adjacent frames of the video is less than a first stuttering threshold, and / or when the rate of change between adjacent frames of the video is less than a second stuttering threshold, the adjacent video frames are determined to be stuttering.

[0026] If an adjacent video frame is determined to be choppy, that video frame is discarded.

[0027] Furthermore, the method also includes:

[0028] Calculate the video-sensory frame rate of the video;

[0029] The video perception frame rate is the number of smooth, uninterrupted frames divided by the playback time of the last smooth, uninterrupted frame.

[0030] Furthermore, the method also includes:

[0031] The video is recorded using a timing recording method, and the perceived frame rate of the recorded video is compared with the actual frame rate to verify the perceived frame rate of the recorded video.

[0032] According to a second aspect of the present invention, embodiments of this disclosure provide a video frame rate detection device based on visual perception, comprising:

[0033] The extraction module is used to extract the timestamp of each frame of the video and determine the adjacent frames of the video based on the timestamp.

[0034] The conversion module is used to convert the RGB three channels of each frame of the video into a grayscale single channel, and to calculate the grayscale pixel value of the grayscale single channel.

[0035] The first calculation module is used to calculate the rate of change between adjacent frames of the video based on the proportion of different grayscale pixel values.

[0036] The second calculation module is used to calculate the content change rate of adjacent frames of the video based on the sum of the differences in the grayscale pixel values.

[0037] The first detection module is configured to detect a relationship between a change rate of adjacent frames of the video and a first freezing threshold.

[0038] The second detection module is configured to detect a relationship between a content change rate of adjacent frames of the video and a second freezing threshold.

[0039] The determination module is configured to determine that adjacent video frames are not frozen when the change rate of adjacent frames is greater than or equal to the first freezing threshold, and the content change rate of adjacent frames is greater than or equal to the second freezing threshold.

[0040] According to a third aspect of the present application, an electronic device is provided, comprising:

[0041] a memory configured to store computer readable instructions; and

[0042] a processor configured to execute the computer readable instructions, so that the electronic device implements the method according to any one of the first aspect.

[0043] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a program, and the program is executed to implement the method according to any one of the first aspect.

[0044] The present disclosure discloses a video frame rate detection method and device based on visual perception, an electronic device and a computer readable storage medium. The method comprises: extracting a timestamp of each frame of a video, determining adjacent frames of the video according to the timestamp; converting an RGB three-channel of each frame of the video into a gray single-channel, and calculating a gray pixel point value of the gray single-channel; calculating a change rate of adjacent frames of the video according to a proportion of different quantities of the gray pixel point value; calculating a content change rate of adjacent frames of the video according to a proportion of a sum of change differences of the gray pixel point value; detecting a relationship between the change rate of adjacent frames of the video and a first freezing threshold; detecting a relationship between the content change rate of adjacent frames of the video and a second freezing threshold; and determining that adjacent video frames are not frozen when the change rate of adjacent frames is greater than or equal to the first freezing threshold, and the content change rate of adjacent frames is greater than or equal to the second freezing threshold. The video frame rate detection method of the present disclosure can determine whether a video frame is a frozen frame by calculating the visual difference between adjacent frames of the video, divide the number of non-frozen frames by the total duration of the video to obtain a video perception frame rate, accurately monitor the video freezing, and provide a perception frame rate algorithm verification scheme. Experiments show that the perception frame rate is more consistent with the smoothness of human eyes watching the video, and provides a clear and reliable measurement index for the degree of video freezing.

[0045] The above description is only a summary of the technical solutions of the present disclosure. In order to make the technical means of the present disclosure more clearly understood, the technical solutions can be implemented according to the content of the description, and in order to make the above and other objects, characteristics and advantages of the present disclosure more apparent and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 A flowchart of a video frame rate detection method based on visual perception provided by an embodiment of the present disclosure is shown in the figure.

[0047] Figure 2 A schematic diagram of an original picture of a frame in a video provided by an embodiment of the present disclosure is shown in the figure.

[0048] Figure 3 A schematic diagram of an R channel component of a frame in a video provided by an embodiment of the present disclosure is shown in the figure.

[0049] Figure 4 A schematic diagram of a G channel component of a frame in a video provided by an embodiment of the present disclosure is shown in the figure.

[0050] Figure 5 A schematic diagram of a B channel component of a frame in a video provided by an embodiment of the present disclosure is shown in the figure.

[0051] Figure 6 A schematic diagram of a video frame rate detection device based on visual perception provided by another embodiment of the present disclosure is shown in the figure.

[0052] Figure 7 A schematic diagram of the structure of an electronic device provided by another embodiment of the present disclosure is shown in the figure. DETAILED DESCRIPTION

[0053] In order to make the technical content of the present disclosure more clearly understood, the embodiments of the present disclosure will be further described below in combination with specific embodiments.

[0054] The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0055] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather, these embodiments are provided to make the present disclosure more thoroughly and completely understood. It should be understood that the accompanying drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the scope of protection of the present disclosure.

[0056] It should be understood that each of the steps recited in the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit performing the steps shown. The scope of the present disclosure is not limited in this regard.

[0057] The term "comprises" and variations thereof used in the present disclosure are open-ended, that is, "comprising but not limited to." The term "based on" is "based, at least in part, on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." Related definitions will be given in the description below.

[0058] It should be noted that the "first", "second", and the like concepts mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0059] It should be noted that the modification of "one", "multiple" mentioned in the present disclosure is illustrative and not limiting, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0060] The terms used in the present disclosure are merely for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the present disclosure and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. The embodiments of the disclosure are described in detail below with reference to the accompanying drawings.

[0061] Figure 1 The flowchart of the video frame rate detection method based on visual perception provided by the embodiments of the present disclosure, the video frame rate detection method based on visual perception provided by the embodiments of the present disclosure can be executed by a video frame rate detection device based on visual perception. The device can be implemented as software, or as a combination of software and hardware. The device can be integrated into a certain device in a video frame rate detection system based on visual perception, such as a terminal device. As shown in the figure, the method comprises the following steps: Figure 1

[0062] Step S101: Extract the timestamp of each frame of the video, and determine the adjacent frames of the video according to the timestamp.

[0063] ​In step S101, when analyzing the video file, we often label a certain frame data and extract the presentation time stamp (PTS) of each frame of the video. The video presentation time stamp is the time point of each frame display after decoding the video frame. The user calculates the playing time of the video frame. The display time of each frame of the video is not constant, but dynamic. Therefore, it is more accurate to calculate the time of the video frame by the presentation time stamp. According to the time point and the playing time of each frame, the adjacent frames of the video are determined. The embodiment of the disclosure uses Opencv to read the frame data in the video. The video has the time stamp of the video frame data. The time stamp obtained by OpenCV allows the user to know the position of the frame data in the video file. The adjacent frames of the video can be determined according to the time stamp. The order of the adjacent frames and the playing time of each frame are determined. Through OpenCV, the frame rate of each video can be obtained:

[0064] import cv2

[0065] video = cv2.VideoCapture(videoFile)

[0066] fps = video.get(cv2.CAP_PROP_FPS)

[0067] OpenCV can add 1000 / fps (since milliseconds) to each frame in the video, but this assumes that the frame rate remains stable throughout the recording process.

[0068] Step S102: converting the RGB three channels of each frame of the video into a gray single channel, and calculating the gray pixel point value of the gray single channel.

[0069] In step S102, the 24-bit RGB image is also called a full-color image. It has three channels, which are R (red), G (green), and B (blue). In combination with the attached Figures 2-5 , Figures 2-5 An example of converting the RGB three channels of the video to the GRAY single channel is shown.

[0070] As Figure 2 shown in FIG. 1, a certain frame of the video is shown. First, read an image (orange in a blue background, which is shown as a gray background in the disclosure), which is a color image containing R, B, and G (which is shown as a gray image in the disclosure), and draw a point (red × in the figure, which is shown as gray or black in the disclosure):

[0071] get_grayval (Image, Row, Column, RGBGrayval) Get the gray pixel point value of the drawn point, the result is:

[0072] RGBGrayval [19,22,37].

[0073] Secondly, as Figures 3-5 , RGB three-channel separation, and get the gray pixel point value:

[0074] Three-channel separation is realized by decompose, and the gray pixel point values of the three channels are obtained respectively:

[0075] get_grayval (R, Row, Column, RGrayval)

[0076] RGBGrayval 19 (reference Figure 3 )

[0077] get_grayval (G, Row, Column, GGrayval)

[0078] RGBGrayval 22 (reference Figure 4 )

[0079] Reference Figure 5 , get_grayval (B, Row, Column, BGrayval)

[0080] RGBGrayval 37 (reference Figure 5 )

[0081] It can be concluded that the gray pixel point value of the three-channel image is the combination of the gray pixel point values of the three single channels. The gray pixel point value is 0-255, and each channel is 0-255. The larger the value, the brighter the image looks, and the smaller the value, the darker the image. The part of which color is darker on the three-channel image, which proves that the color component of that part is larger, and the brighter it is reflected on the single channel. For example, from the original image, it can be seen that the background is mostly blue, so the background part is brighter on the blue channel, and the orange part is very dark.

[0082] In addition, the single-channel gray image is a single-channel image calculated by R G B three components in a certain proportion, and the calculation method is not single.

[0083] In this embodiment, the components of each single channel of the RGB have corresponding weight values, the RGB three channels of each frame of the video are converted into a gray single channel, and the gray pixel point value of the gray single channel is calculated, and the conversion mode formula is:

[0084] GRAY = R * 0.299 + G * 0.587 + B * 0.114;

[0085] wherein, GRAY is the gray pixel point value of the gray single channel, R is the red channel component, G is the green channel component, and B is the blue channel component.

[0086] Step S103: calculating the adjacent frame change rate of the video according to the proportion of the number of different gray pixel point values.

[0087] In step S103, if the adjacent frames in the video are continuous, no stuttering will be felt, and if there is frame skipping or too large difference between the adjacent frames, the phenomenon of stuttering will be felt. Here, the concept of adjacent frame change rate is introduced, which has a certain correlation with the phenomenon of stuttering of the video.

[0088] The adjacent frame change rate is the proportion of the number of unequal pixel point values corresponding to the video adjacent frames to the total number of pixel points, and the specific calculation formula is as follows:

[0089] ;

[0090] wherein, H and W represent the height and width of the video frame, p ij and p ’ ij respectively represent the pixel point values of the adjacent frame image coordinates i , j , and the adjacent frame change rate ranges between 0 and 1.

[0091] The greater the adjacent frame change rate, the more the number of pixels of the adjacent frame image changes, and the human eye is more likely to feel the change of the video frame.

[0092] Step S104: calculating the adjacent frame content change rate of the video according to the proportion of the sum of the change differences of the gray pixel point values.

[0093] In step S104, the adjacent frame content change rate is different from the adjacent frame change rate in step S103. The adjacent frame content change rate is the proportion of the sum of the change differences of the pixel point values corresponding to the video adjacent frames to the number of unequal pixel point values. It can be seen that the denominator of the adjacent frame content change rate is a certain multiple of the denominator of the adjacent frame change rate.

[0094] Specifically, the adjacent frame content change rate refers to the proportion of the sum of the change differences of the pixel point values corresponding to the video adjacent frames to the number of unequal pixel point values. The specific calculation formula is as follows:

[0095] ;

[0096] where H and W represent the height and width of the video frame, p ij and p ’ ij respectively represent the pixel point values of the adjacent frame image coordinates i , j , where multiplying 256 in the denominator is because there are 256 gray image pixel values, so the adjacent frame content change rate range is between 0-1.

[0097] where multiplying 256 in the denominator is because there are 256 gray image pixel values, so the adjacent frame content change rate range is between 0-1, and the greater the change rate indicates that the adjacent frame image content changes more, and the human eye is more likely to feel the change of the video frame.

[0098] Step S105: detecting the relationship between the adjacent frame change rate of the video and the first stall threshold.

[0099] In step S105, the condition for the adjacent video frame to be non-stall frame is that the adjacent frame change rate and the adjacent frame content change rate must meet a certain threshold at the same time, otherwise the frame is discarded. Here, first, the relationship between the adjacent frame change rate of the video and the first stall threshold is detected, and the first stall threshold is set to be between 0.01-0.1, for example, 0.01, 0.02 or 0.05, taking 0.02 as an example, it needs to meet that the adjacent frame change rate is greater than 0.02, and based on this, further meet that the adjacent frame content change rate is greater than the second stall threshold to determine that the adjacent video frame is non-stall; otherwise, as long as at least one of the adjacent frame change rate or the adjacent frame content change rate is less than the corresponding first stall threshold or second stall threshold, it is determined that the adjacent video frame is stall, and the stall frame is discarded.

[0100] Step S106: detecting the relationship between the adjacent frame content change rate of the video and the second stall threshold.

[0101] In step S106, after detecting the relationship between the adjacent frame change rate of the video in step S105 and the first freeze threshold, the relationship between the adjacent frame content change rate of the video and the second freeze threshold is further detected. Because the condition for the adjacent video frames to be non-freezing frames is that the adjacent frame change rate and the adjacent frame content change rate must simultaneously satisfy a certain threshold, and only then can it be determined that the adjacent frames constitute freezing. When detecting the relationship between the adjacent frame content change rate of the video and the second freeze threshold, the second freeze threshold is set to be between 0.1 and 0.5, for example, 0.1, 0.15, or 0.3. Taking 0.15 as an example, the adjacent frame content change rate needs to exceed 0.15. However, only satisfying this condition cannot determine that the adjacent frames are non-freezing. It must also simultaneously satisfy that the adjacent frame change rate exceeds the first freeze threshold, and only then can it be determined that the adjacent frames are non-freezing. Otherwise, as long as at least one of the adjacent frame change rate or the adjacent frame content change rate is less than the corresponding first freeze threshold or second freeze threshold, it is determined that the adjacent video frames are freezing, and the freezing frame is discarded.

[0102] Step S107: When the adjacent frame change rate is greater than or equal to the first freeze threshold, and the adjacent frame content change rate is greater than or equal to the second freeze threshold, it is determined that the adjacent video frames are non-freezing.

[0103] In step S107, the condition for the adjacent video frames to be non-freezing frames is that the adjacent frame change rate and the adjacent frame content change rate must simultaneously satisfy a certain threshold, and only then can it be determined that the adjacent frames constitute non-freezing. That is, when the adjacent frame change rate is greater than or equal to the first freeze threshold, and the adjacent frame content change rate is greater than or equal to the second freeze threshold, it is determined that the adjacent video frames are non-freezing. When detecting the relationship between the adjacent frame change rate of the video and the first freeze threshold, the first freeze threshold is set to be between 0.01 and 0.1, for example, 0.01, 0.02, or 0.05. Taking 0.02 as an example, the adjacent frame change rate needs to exceed 0.02. At the same time, when detecting the relationship between the adjacent frame content change rate of the video and the second freeze threshold, the second freeze threshold is set to be between 0.1 and 0.5, for example, 0.1, 0.15, or 0.3. Taking 0.15 as an example of the present disclosure, only when the adjacent frame change rate exceeds 0.02 and the adjacent frame content change rate exceeds 0.15 can it be determined that the adjacent frames are non-freezing. Otherwise, as long as at least one of the adjacent frame change rate or the adjacent frame content change rate is less than the corresponding first freeze threshold or second freeze threshold, it is determined that the adjacent video frames are freezing, and the freezing frame is discarded. Specifically, when the adjacent frame change rate of the video is less than 0.02 and / or the adjacent frame content change rate of the video is less than 0.15, it is determined that the adjacent video frames are freezing. When it is determined that the adjacent video frames are freezing, the video frame is discarded.

[0104] In addition, the embodiments of the present disclosure further determine the freezing effect of the video frame through the video perceived frame rate. The time sensitivity and resolution of human vision vary according to the type and characteristics of visual stimuli, and are different between individuals. The video frame rate is the number of frames displayed per second of a video, which directly determines the smoothness of the video playback, and the video perceived frame rate refers to the video frame rate perceived by the human eye. The video freezing referred to in the embodiments of the present disclosure is perceived and verified by the human eye. The greater the perceived frame rate, the smoother the video, and vice versa. The human eye will perceive freezing when the video frame rate is slow and uneven. Generally, the freezing cannot be perceived when the video frame rate is higher than a certain threshold (usually 24 fps) and the frame rate is uniform. The higher the frame rate, the smoother the video. The human visual system can process 10 to 12 images per second and perceive them individually, while a higher rate is considered as motion. When the rate is higher than 50 Hz to 90 Hz, most research participants consider the modulated light (such as a computer display) to be stable. This feeling of stability of the modulated light is called the flicker fusion threshold. However, when the modulated light is uneven and contains images, the flicker fusion threshold can be much higher, hundreds of hertz. Regarding image recognition, it has been found that people recognize a specific image in an uninterrupted series of different images, each lasting as little as 13 milliseconds. The persistence of vision sometimes causes very short single-millisecond visual stimuli, with a perceived duration of 100 milliseconds to 400 milliseconds. Very short multiple stimuli are sometimes considered a single stimulus, for example, a 10-millisecond green flash followed by a 10-millisecond red flash is perceived as a single yellow flash.

[0105] Since there may be a black screen and other still images at the end of the video, in order to accurately calculate the video perceived frame rate, the final perceived frame rate algorithm result is the number of non-freezing frames divided by the playback time of the last non-freezing frame.

[0106] The embodiments of the present disclosure propose two perceived frame rate verification schemes:

[0107] 1. Calculate the video perceived frame rate of the video, which is the number of non-freezing frames divided by the playback time of the last non-freezing frame. Record the video in a recording timing manner, and compare the perceived frame rate and the actual frame rate of the recorded video to verify the perceived frame rate of the recorded video. Specifically, to verify that the perceived frame rate algorithm does not affect when the video content changes continuously, a recording timing stopwatch is used, and the recorded video is 60 fps. For example, the actual frame rate of an MP4 video is 60 fps, and the perceived frame rate is 60 fps.

[0108] 2. To verify that the perceived frame rate algorithm can accurately calculate the perceived frame rate when the video content changes little, a 10fps video recorded at 30fps is used, and the video is recorded as 1080p_10fps_20s_h264_0.mp4. The actual frame rate of the video is 30fps, and the perceived frame rate is 10.1fps.

[0109] Figure 6 The visual perception-based video frame rate detection device provided for another embodiment of the present disclosure is shown in the schematic diagram. The visual perception-based video frame rate detection device comprises:

[0110] The extraction module 601 is configured to extract the timestamp of each frame of the video, and determine the adjacent frames of the video according to the timestamp.

[0111] The conversion module 602 is configured to convert the RGB three channels of each frame of the video into a gray single channel, and calculate the gray pixel point value of the gray single channel.

[0112] The first calculation module 603 is configured to calculate the adjacent frame change rate of the video according to the proportion of the number of different gray pixel point values.

[0113] The second calculation module 604 is configured to calculate the adjacent frame content change rate of the video according to the proportion of the sum of the change differences of the gray pixel point values.

[0114] The first detection module 605 is configured to detect the relationship between the adjacent frame change rate of the video and the first stall threshold.

[0115] The second detection module 606 is configured to detect the relationship between the adjacent frame content change rate of the video and the second stall threshold.

[0116] The determination module 607 is configured to determine that the adjacent video frames are not stalled when the adjacent frame change rate is greater than or equal to the first stall threshold, and the adjacent frame content change rate is greater than or equal to the second stall threshold.

[0117] Wherein:

[0118] The extraction module 601 is configured to extract the timestamp of each frame of the video, and determine the adjacent frames of the video according to the timestamp.

[0119] A video playback timestamp is the time point at which each frame is displayed after video frame decoding. Users calculate the playback time of video frames. Since the display time of each frame is not constant but dynamically changes, calculating the video frame time using playback timestamps is more accurate. Adjacent frames of the video are determined based on the time point of each frame and the playback time. This embodiment uses OpenCV to read frame data from the video. The video contains timestamps for the video frame data. The timestamps obtained by OpenCV allow the user to know the location of this frame data within the video file. Through OpenCV, the frame rate of each video can be obtained.

[0120] The conversion module 602 is used to convert the RGB three channels of each frame of the video into a grayscale single channel, and calculate the grayscale pixel value of the grayscale single channel.

[0121] Color video has three channels: R (red), G (green), and B (blue). (See attached diagram.) Figures 2-5 , Figures 2-5 This example illustrates the conversion of a video's RGB three-channel format to a GRAY single-channel format. First, an image is read (oranges on a blue background, the text displayed on the gray background), which is a color image containing R, B, and G (the text displayed on the grayscale image), and a dot is drawn (a red × in the image, the text displayed on the gray or black background). Next, as... Figures 2-5 The RGB three channels are separated, and grayscale pixel values ​​are obtained. The grayscale pixel values ​​of the three-channel image are a combination of the grayscale pixel values ​​of the three single channels. Grayscale pixel values ​​range from 0 to 255, with each channel having a value of 0-255. Higher values ​​result in a brighter image, while lower values ​​result in a darker image. The area in the three-channel image where a particular color appears darker indicates the strength of that color component, which is reflected as brighter in that single channel. In this embodiment, each component of the RGB single channel has a corresponding weighting value. The conversion of the RGB three channels of each frame of the video to a single grayscale channel and the calculation of the grayscale pixel values ​​of that single channel are performed using the following formula:

[0122] GRAY=R*0.299 + G*0.587 + B*0.114;

[0123] Where GRAY is the grayscale pixel value of a single grayscale channel, R is the red channel component, G is the green channel component, and B is the blue channel component.

[0124] The first calculation module 603 is used to calculate the rate of change of adjacent frames of the video based on the proportion of different grayscale pixel values.

[0125] If the adjacent frames in the video are continuous, no stuttering is felt, if there is frame skipping or too large difference between adjacent frames, stuttering phenomenon is felt, here the concept of adjacent frame change rate is introduced, which has a certain correlation with the stuttering phenomenon of the video. The adjacent frame change rate is the proportion of the number of adjacent frame corresponding bit pixel points not equal to the total number of pixel points, the range of the adjacent frame change rate is between 0-1. The greater the adjacent frame change rate indicates that the number of pixel changes of adjacent frames is more, and the human eye is more likely to feel the change of video frames.

[0126] The second calculation module 604 is configured to calculate the adjacent frame content change rate of the video according to the proportion of the sum of the change difference of the gray pixel point value.

[0127] The adjacent frame content change rate is the proportion of the sum of the change difference of the adjacent frame corresponding bit pixel point value and the number of corresponding bit pixel points not equal to each other, the range of the adjacent frame content change rate is between 0-1. The greater the change rate indicates that the image content change of adjacent frames is greater, and the human eye is more likely to feel the change of video frames.

[0128] The first detection module 605 is configured to detect the relationship between the adjacent frame change rate of the video and the first stuttering threshold.

[0129] Here, first, the relationship between the adjacent frame change rate of the video and the first stuttering threshold is detected, the first stuttering threshold is set to 0.01-0.1, for example, 0.01, 0.02 or 0.05, taking 0.02 as an example, it needs to meet that the adjacent frame change rate is greater than 0.02, based on this, further meeting that the adjacent frame content change rate is greater than the second stuttering threshold determines that the adjacent video frame is not stuttering; otherwise, as long as at least one of the adjacent frame change rate or the adjacent frame content change rate is less than the corresponding first stuttering threshold or second stuttering threshold, it is determined that the adjacent video frame is stuttering, and the stuttering frame is discarded.

[0130] The second detection module 606 is configured to detect the relationship between the adjacent frame content change rate of the video and the second stuttering threshold.

[0131] After detecting the relationship between the adjacent frame change rate and the first freeze threshold, the relationship between the adjacent frame content change rate of the video and the second freeze threshold is further detected. Because the condition for the adjacent video frames to be non-freezing frames is that the adjacent frame change rate and the adjacent frame content change rate must simultaneously satisfy a certain threshold, so as to determine that the adjacent frames constitute freezing. When detecting the relationship between the adjacent frame content change rate of the video and the second freeze threshold, the second freeze threshold is set to be between 0.1 and 0.5, for example, 0.1, 0.15 or 0.3. Taking 0.15 as an example, the adjacent frame content change rate needs to exceed 0.15. However, only satisfying this condition cannot determine that the adjacent frames are non-freezing. It is necessary to simultaneously satisfy that the adjacent frame change rate exceeds the first freeze threshold, so as to determine that the adjacent frames are non-freezing. Otherwise, as long as at least one of the adjacent frame change rate or the adjacent frame content change rate is less than the corresponding first freeze threshold or second freeze threshold, it is determined that the adjacent video frames are freezing, and the freezing frame is discarded.

[0132] The determination module 607 is configured to determine that the adjacent video frames are non-freezing when the adjacent frame change rate is greater than or equal to the first freeze threshold, and the adjacent frame content change rate is greater than or equal to the second freeze threshold.

[0133] The condition for the adjacent video frames to be non-freezing frames is that the adjacent frame change rate and the adjacent frame content change rate must simultaneously satisfy a certain threshold, so as to determine that the adjacent frames constitute freezing. That is, when the adjacent frame change rate is greater than or equal to the first freeze threshold, and the adjacent frame content change rate is greater than or equal to the second freeze threshold, it is determined that the adjacent video frames are non-freezing. When detecting the relationship between the adjacent frame change rate of the video and the first freeze threshold, the first freeze threshold is set to be between 0.01 and 0.1, for example, 0.01, 0.02 or 0.05. Taking 0.02 as an example, the adjacent frame change rate needs to exceed 0.02. At the same time, when detecting the relationship between the adjacent frame content change rate of the video and the second freeze threshold, the second freeze threshold is set to be between 0.1 and 0.5, for example, 0.1, 0.15 or 0.3. Taking 0.15 as an example of the embodiment of the present disclosure, it is determined that the adjacent frames are non-freezing only when the adjacent frame change rate exceeds 0.02 and the adjacent frame content change rate exceeds 0.15. Otherwise, as long as at least one of the adjacent frame change rate or the adjacent frame content change rate is less than the corresponding first freeze threshold or second freeze threshold, it is determined that the adjacent video frames are freezing, and the freezing frame is discarded. Specifically, when the adjacent frame change rate of the video is less than 0.02 and / or the adjacent frame content change rate of the video is less than 0.15, it is determined that the adjacent video frames are freezing. When it is determined that the adjacent video frames are freezing, the freezing frame is discarded.

[0134] The determination module 607 is further configured to determine that the adjacent video frame is a frame freezing when a change rate of adjacent frames of the video is less than a first frame freezing threshold and / or a content change rate of adjacent frames of the video is less than a second frame freezing threshold.

[0135] The adjacent video frame is discarded when the adjacent video frame is determined to be a frame freezing.

[0136] The video frame rate detection device based on visual perception further comprises:

[0137] The visual perception frame rate calculation module is configured to calculate a video perception frame rate of the video, wherein the video perception frame rate is a number of non-freezing frames of the video divided by a playing time of a last non-freezing frame.

[0138] The video frame rate detection device based on visual perception further comprises:

[0139] The verification module is configured to record a video in a recording timing mode, compare a perception frame rate of the recorded video with an actual frame rate, and verify the perception frame rate of the recorded video.

[0140] Figure 6 The device can perform Figure 1 The method of the embodiment is not described in detail, and reference can be made to the related description of the Figure 1 The embodiment. The execution process and technical effects of the technical solution are described in the Figure 1 The embodiment, which will not be described here again.

[0141] Reference is made below to Figure 7 , which shows a structural schematic diagram of an electronic device 700 suitable for implementing another embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a vehicle terminal (such as a vehicle navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 7 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiment of the present disclosure.

[0142] As Figure 7As shown, the electronic device 700 can include a processing device (e.g., a central processor, a graphics processor, etc.) 701 that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 702 or loaded into a random access memory (RAM) 703 from a storage device 708. Various programs and data required by the electronic device 700 to operate are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other through a communication line 704. An input / output (I / O) interface 705 is also connected to the communication line 704.

[0143] Generally, the following devices can be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 can allow the electronic device 700 to communicate wirelessly or wired with other devices to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present.

[0144] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0145] It should be noted that the computer-readable medium in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium that can send, propagate or transmit the program for use by or in connection with an instruction execution system, apparatus or device. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including, but not limited to, a wire, cable, optical fiber, RF (radio frequency), or the like, or any suitable combination of the above.

[0146] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0147] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device and not be assembled into the electronic device.

[0148] The computer-readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the interaction method in the embodiments described above.

[0149] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0150] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0151] The units described in the embodiments of the present disclosure can be implemented by hardware, software, or a combination of hardware and software. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0152] The functions described in this specification can be implemented in part or in whole through one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0153] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0154] According to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any of the methods described in the first aspect above.

[0155] According to one or more embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform any of the methods described in the first aspect above.

[0156] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A method for detecting frame rate of a video based on visual perception, the method comprising: The method comprises: extracting the timestamp of each frame of the video, and determining the adjacent frames of the video according to the timestamp; converting the RGB three channels of each frame of the video into a gray single channel, and calculating the gray pixel point value of the gray single channel; calculating the adjacent frame change rate of the video according to the proportion of the number of different gray pixel point values; calculating the adjacent frame content change rate of the video according to the proportion of the sum of the change differences of the gray pixel point values; detecting the relationship between the adjacent frame change rate of the video and the first stall threshold; detecting the relationship between the adjacent frame content change rate of the video and the second stall threshold; and when the adjacent frame change rate is greater than or equal to the first stall threshold, and the adjacent frame content change rate is greater than or equal to the second stall threshold, determining that the adjacent video frame is not stalled. The timestamp of each frame of the video is the time point of each frame of the video after decoding, and the user calculates the playing time of the video frame, and determines the adjacent frames of the video according to the time point and the playing time of each frame.

2. The method of claim 1, wherein, The conversion formula of converting the RGB three channels of each frame of the video into a gray single channel and calculating the gray pixel point value of the gray single channel is:

3. The method of claim 1, wherein, wherein GRAY is the gray pixel point value of the gray single channel, R is the red channel component, G is the green channel component, and B is the blue channel component. ； The adjacent frame change rate is the proportion of the number of adjacent frame corresponding bit gray pixel point values not equal to the total number of pixel points, and the specific calculation formula is as follows:

4. The method of claim 1, wherein, The adjacent frame content change rate is the proportion of the sum of the change differences of the adjacent frame corresponding bit gray pixel point values and the number of adjacent frame corresponding bit gray pixel point values not equal, and the specific calculation formula is as follows: ； where H and W represent the height and width of the video frame, and respectively represent the gray pixel point values of the adjacent frame image coordinates i , j The adjacent frame change rate range is between 0-1.

5. The method of claim 1, wherein, The first stall threshold is set to 0.01 to 0.1, and the second stall threshold is set to 0.1 to 0.

5. ； where H and W represent the height and width of the video frame, and respectively represent the gray pixel point values of the adjacent frame image coordinates i , j , where the denominator multiplied by 256 is because the gray image pixel values have 256 in total, and the adjacent frame content change rate range is between 0-1.

6. The method of claim 1, wherein, The method further comprises:

7. The method of claim 1, wherein, when the adjacent frame change rate of the video is less than the first stall threshold, and / or the adjacent frame content change rate of the video is less than the second stall threshold, determining that the adjacent video frame is stalled; when it is determined that the adjacent video frame is stalled, discarding the video frame. The method further comprises:

8. The method of claim 1, wherein, calculating the video perception frame rate of the video; The video perception frame rate is the number of non-stalled frames divided by the playing time of the last non-stalled frame. The method further comprises:

9. The method of claim 8, wherein, recording the video in a recording timing manner, comparing the perception frame rate and the actual frame rate of the recorded video, and verifying the perception frame rate of the recorded video. The method comprises:

10. A visual perception based video frame rate detection apparatus, characterized in that, an extraction module configured to extract the timestamp of each frame of the video, and determine the adjacent frames of the video according to the timestamp; a conversion module configured to convert the RGB three channels of each frame of the video into a gray single channel, and calculate the gray pixel point value of the gray single channel; a first calculation module configured to calculate the adjacent frame change rate of the video according to the proportion of the number of different gray pixel point values; a second calculation module configured to calculate the adjacent frame content change rate of the video according to the proportion of the sum of the change differences of the gray pixel point values; a first detection module configured to detect the relationship between the adjacent frame change rate of the video and the first stall threshold; and a second detection module configured to detect the relationship between the adjacent frame content change rate of the video and the second stall threshold. a second detection module configured to detect a relationship between a content change rate of adjacent frames of the video and a second freezing threshold; and a determination module configured to determine that the adjacent video frames are not frozen when the adjacent frame change rate is greater than or equal to the first freezing threshold and the content change rate of the adjacent frames is greater than or equal to the second freezing threshold.

11. An electronic device, comprising: comprise: a memory configured to store computer readable instructions; and a processor configured to execute the computer readable instructions to enable the electronic device to implement the method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program, and the program is executed by the processor to implement the method according to any one of claims 1-9.

Images