Video fluency detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a video fluency detection method and device, electronic equipment and a storage medium. The method comprises the steps that M photos of a played video are acquired, the M photos correspond to N video frames of the played video, and one video frame corresponds to at least one photo; determining a picture similarity according to every two adjacent pictures in the M pictures to obtain M-1 picture similarities; determining the frame playing duration of each video frame in at least a part of video frames in the N video frames according to the M-1 picture similarities; and determining the fluency of playing the video according to the frame playing duration of each video frame in the at least part of video frames. According to the method, the fluency of the played video is calculated based on the frame playing duration of each video frame in the at least part of video frames of the played video, and the subjective feeling of a user on video playing lagging is represented by a specific numerical value, so that the evaluation on the video fluency is more objective and effective, and the user experience is improved. And the detection accuracy of video fluency detection is improved.

Description

Technical Field

[0001] The present application relates to the field of video detection technology, and in particular to a method, a detection device, an electronic device and a storage medium for detecting video fluency. Background Art

[0002] When users watch videos, they may sometimes feel that the picture or object movement has obvious lag. This is because the video itself has irregular frame sequences, the video frame rate does not match the screen refresh rate, or the GPU rendering is abnormal. Frame drops will cause uneven video frame playback time, resulting in changes in the speed of the picture or object movement, which will cause obvious lag visually. Stuttering will affect the user experience and even make users lose interest in video-related products.

[0003] Smoothness can be used as an evaluation indicator in many aspects. For example, smoothness can be used as an important indicator for evaluating video quality, smoothness can also be used as an important indicator for evaluating the network status of video playback, and smoothness can also be used as an important indicator for measuring the performance of electronic devices that play videos.

[0004] Currently, there is a lack of effective fluency detection methods, and electronic device manufacturers mainly test the fluency of electronic devices playing videos through manual perception. However, since everyone has different feelings about the fluency of electronic devices playing videos, the accuracy of video fluency detection is low. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a method, a detection device, an electronic device and a storage medium for detecting video fluency to overcome the above problems of the prior art.

[0006] In a first aspect, an embodiment of the present application provides a method for detecting video fluency, and the method for detecting video fluency includes: obtaining M photos of a played video, the M photos corresponding to N video frames of the played video, and one video frame corresponding to at least one photo; determining a picture similarity based on every two adjacent photos in the M photos to obtain M-1 picture similarities; determining a frame playback duration of each video frame in at least some of the N video frames based on the M-1 picture similarities; and determining the smoothness of the played video based on the frame playback duration of each video frame in at least some of the video frames.

[0007] The solution provided by the present application, in the process of detecting the video smoothness, determines the frame playback duration of each video frame in at least some of the N video frames of the video being played based on the picture similarity between every two adjacent photos in M ​​photos, and calculates the smoothness of the video playback based on the frame playback duration of each video frame, and expresses the user's subjective feeling of the video playback freeze with a specific numerical value, so that the evaluation of the video smoothness is more objective and effective, and the detection accuracy of the video smoothness is improved.

[0008] The smoothness of video playback is calculated based on the frame playback duration of the video frames. A unified method for quantifying smoothness is used to detect stuttering caused by various reasons, such as stuttering caused by mismatch between video frame rate and screen refresh rate, stuttering caused by GPU rendering abnormalities, and stuttering caused by frame drops, etc. This expands the application scenarios for video smoothness detection.

[0009] The picture similarity when the video frame switches is low. According to the low picture similarity, the switching duration of the video frame can be accurately calculated to obtain the frame playback duration of the video frame, thereby improving the calculation accuracy of the frame playback duration of the video frame.

[0010] In the above-mentioned step of determining the frame playback duration of each video frame in at least some of the N video frames based on M-1 picture similarities, the frame playback duration of each video frame in at least some of the video frames is determined based on the correspondence between the M-1 picture similarities and the M-1 time points in the playback duration of the playback video.

[0011] In the solution provided by this embodiment, the picture similarity when the video frames switch is low. According to the correspondence between the picture similarity and the playback time point of the played video, the switching duration of the video frames can be accurately calculated, and the frame playback duration of each video frame in at least some of the video frames can be obtained, thereby improving the calculation accuracy of the frame playback duration of each video frame in at least some of the video frames.

[0012] In the above-mentioned step of determining the frame playback duration of each video frame in at least part of the video frames according to the correspondence between M-1 picture similarities and M-1 time points in the playback duration of the playback video, according to the correspondence between the M-1 picture similarities and the M-1 time points in the playback duration of the playback video, the duration between two time points corresponding to each two adjacent minimum values ​​of picture similarities is determined as the frame playback duration of a video frame, so as to obtain the frame playback duration of each video frame in at least part of the video frames.

[0013] In the solution provided by this embodiment, the minimum value of the picture similarity corresponds to the video frame switching moment of the played video, and the frame playback duration of each video frame in at least some video frames is calculated based on the time corresponding to the minimum value of the picture similarity, thereby improving the calculation accuracy of the frame playback duration of each video frame in at least some video frames.

[0014] In the above-mentioned step of determining the smoothness of video playback based on the frame playback duration of each video frame in at least part of the video frames, the stuttering rate of video playback is determined based on the frame playback duration of each video frame in at least part of the video frames; and the smoothness of video playback is determined based on the stuttering rate.

[0015] In the above-mentioned step of determining the jamming rate of the played video based on the frame playback duration of each video frame in at least part of the video frames, the base frame playback duration is determined based on the frame playback duration of each video frame in at least part of the video frames; the jamming duration of the played video is determined based on the base frame playback duration and the frame playback duration of each video frame in at least part of the video frames; the jamming rate of the played video is determined based on the total playback duration of N video frames and the jamming duration.

[0016] In one example, the above-mentioned determination of the benchmark frame playback duration based on the frame playback duration of each video frame in at least part of the video frames includes: determining the frame playback duration with the largest number of repetitions among the frame playback durations of each video frame in at least part of the video frames as the benchmark frame playback duration.

[0017] Exemplarily, the freeze duration of the video playback is determined based on the playback duration of the benchmark frame and the frame playback duration of each video frame in at least part of the video frames, including: summing the absolute values ​​of the differences between the playback durations of each non-benchmark frame and the playback duration of the benchmark frame to obtain the freeze duration of the video playback.

[0018] In another example, a benchmark frame playback duration is determined based on the frame playback durations of each video frame in at least part of the video frames, including: calculating an average frame playback duration based on the frame playback durations of each video frame in at least part of the video frames; and determining the average frame playback duration as the benchmark frame playback duration.

[0019] In the above step of determining the smoothness of video playback according to the jam rate, the percentage of 1 minus the jam rate is calculated to obtain the smoothness of video playback.

[0020] In the above step of determining a picture similarity based on every two adjacent photos in M ​​photos, a first hash value of one photo and a second hash value of the other photo in any two adjacent photos are calculated; a Hamming distance between any two adjacent photos is calculated based on the first hash value and the second hash value; and a picture similarity of any two adjacent photos is calculated based on the Hamming distance.

[0021] The solution provided in this embodiment has the characteristics of high computational efficiency and high robustness to image transformation in the hash algorithm. The picture similarity of photos is calculated based on the hash algorithm, which improves the computational efficiency and computational robustness of calculating the picture similarity of photos.

[0022] In the above step of obtaining M photos of the playing video, a collection instruction is sent to the high-speed camera, so that the high-speed camera takes pictures of the playing screen of the playing video according to the collection instruction to obtain M photos; and the M photos returned by the high-speed camera are received.

[0023] The solution provided in this embodiment uses a high-speed camera to take M photos of the playing video. The M photos are closer to the continuous video frames of the playing video, which is beneficial to improving the calculation of the smoothness of the playing video, and further beneficial to improving the detection accuracy of the video smoothness.

[0024] In the second aspect, the embodiment of the present application provides a device for detecting the fluency of a video, and the device for detecting the fluency of a video includes an acquisition module, a first determination module, a second determination module, and a third determination module. The acquisition module is used to acquire M photos of the playing video, the M photos correspond to N video frames of the playing video, and one video frame corresponds to at least one photo; the first determination module is used to determine a picture similarity based on every two adjacent photos in the M photos to obtain M-1 picture similarities; the second determination module is used to determine the frame playback duration of each video frame in at least some of the N video frames based on the M-1 picture similarities; the third determination module is used to determine the fluency of the playing video based on the frame playback duration of each video frame in at least some of the video frames.

[0025] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and one or more processors call the computer instructions to enable the electronic device to execute the video fluency detection method provided in the first aspect above.

[0026] In a fourth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the one or more processors are used to call computer instructions to enable the electronic device to execute the video smoothness detection method provided in the first aspect above.

[0027] In some optional embodiments, the chip system also includes a memory, and the memory is connected to one or more processors through circuits or wires.

[0028] In some optional embodiments, the chip system also includes a communication interface.

[0029] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes instructions. When the instructions are executed on an electronic device, the electronic device executes the video smoothness detection method provided in the first aspect above.

[0030] In a sixth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the video fluency detection method provided in the first aspect above.

[0031] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 A structural diagram of a software system of an electronic device provided in an embodiment of the present application is shown.

[0034] Figure 2 A flow chart of a method for detecting video fluency provided in an embodiment of the present application is shown.

[0035] Figure 3 A schematic diagram of a scene in which a high-speed camera takes a photo in the video smoothness detection method provided in an embodiment of the present application is shown.

[0036] Figure 4 A schematic diagram of a scenario of a video fluency detection system provided in an embodiment of the present application is shown.

[0037] Figure 5 A schematic diagram of a scenario of a time-picture similarity curve in a method for detecting video fluency provided in an embodiment of the present application is shown.

[0038] Figure 6 A flow chart of a method for detecting video fluency provided in an embodiment of the present application in an application scenario is shown.

[0039] Figure 7 Another flow chart of a method for detecting video fluency provided in an embodiment of the present application is shown.

[0040] Figure 8A structural schematic diagram of a video fluency detection device provided in an embodiment of the present application is shown.

[0041] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0042] Fig.10 A functional block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0044] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed.

[0045] When users watch videos, they may sometimes feel that the picture or object movement has obvious lag. This is because the video itself has irregular frame sequences, the video frame rate does not match the screen refresh rate, or the GPU rendering is abnormal. Frame drops will cause uneven video frame playback time, resulting in changes in the speed of the picture or object movement, which will cause obvious lag visually. Stuttering will affect the user experience and even make users lose interest in video-related products.

[0046] Smoothness can be used as an evaluation indicator in many aspects. For example, smoothness can be used as an important indicator for evaluating video quality, smoothness can also be used as an important indicator for evaluating the network status of video playback, and smoothness can also be used as an important indicator for measuring the performance of electronic devices that play videos.

[0047] Currently, there is a lack of effective fluency detection methods, and electronic device manufacturers mainly test the fluency of electronic devices playing videos through manual perception. However, since everyone has different feelings about the fluency of electronic devices playing videos, the accuracy of video fluency detection is low.

[0048] In response to the above problems, the video fluency detection method, detection device, electronic device and storage medium provided in the embodiments of the present application obtain M photos of the playing video, the M photos correspond to N video frames of the playing video, and one video frame corresponds to at least one photo, and a picture similarity is determined according to every two adjacent photos in the M photos to obtain M-1 picture similarities, and the frame playback time of each video frame in at least some of the N video frames is determined according to the M-1 picture similarities, and the smoothness of the playing video is determined according to the frame playback time of each video frame in at least some of the video frames. In the detection process of detecting the video fluency, based on the picture similarity of every two adjacent photos in the M photos, the frame playback time of each video frame in at least some of the N video frames of the playing video is determined, and the smoothness of the playing video is calculated based on the frame playback time of each video frame, and the user's subjective feeling of the video jam is expressed by a specific numerical value, so that the evaluation of the video fluency is more objective and effective, and the detection accuracy of the video fluency is improved.

[0049] The smoothness of video playback is calculated based on the frame playback duration of the video frames. A unified method for quantifying smoothness is used to detect stuttering caused by various reasons, such as stuttering caused by mismatch between video frame rate and screen refresh rate, stuttering caused by GPU rendering abnormalities, and stuttering caused by frame drops, etc. This expands the application scenarios for video smoothness detection.

[0050] The picture similarity when the video frame switches is low. According to the low picture similarity, the switching duration of the video frame can be accurately calculated to obtain the frame playback duration of the video frame, thereby improving the calculation accuracy of the frame playback duration of the video frame.

[0051] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0052] The video fluency detection method provided in the embodiment of the present application can be applied to electronic devices. The electronic devices may include various terminal devices, which may also be called terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc.

[0053] The terminal device may be a mobile phone, a sweeping robot, a drone, a smart TV, a wearable device, a PDA, a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The type of terminal device is not limited here, and it can be set according to actual needs.

[0054] The software system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device.

[0055] See also Figure 1 , which shows a schematic diagram of the structure of the software system of an electronic device provided by an embodiment of the present application. The software system includes several layers, each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, namely, the application layer, the application framework layer, the system library, and the kernel layer from top to bottom.

[0056] The application layer may include a series of applications. For example, the application layer may include a fluency detection application, a camera application, a gallery application, a call application, a wireless local area network (Wireless Local Area Networks, WLAN) application, a video application, a media provider (MediaProvider) application, a user space file system (Filesystem in Userspace, FUSE) application, etc.

[0057] Among them, the fluency detection application can be used to obtain M photos of the playing video, and detect the fluency of the playing video according to the M photos. The fluency detection application can be set separately, and the fluency detection application can also be embedded in the video playback application (for example, Youku, iQiyi, Douyin, etc.), etc., which is not limited here.

[0058] The media provider is used to create multimedia files in FUSE or access multimedia files in FUSE. Each application in the application layer can create multimedia files in FUSE through the media provider MediaProvider or access multimedia files in FUSE through the media provider MediaProvider.

[0059] FUSE is used to store the multimedia files that the media provider creates. Of course, in other embodiments, FUSE also can be used to store other data.

[0060] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions. For example, the application framework layer may include a window manager, a content provider, a resource manager, and a view system.

[0061] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0062] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0063] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0064] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0065] System libraries may include Surface Manager, Media Libraries, Android Rruntime, etc.

[0066] The Android runtime consists of a core library and a virtual machine. The Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0067] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0068] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0069] The kernel layer may include modules such as camera driver, display driver, Wi-Fi driver, Bluetooth driver, and audio driver.

[0070] Understandably, Figure 1 The layers in the software structure shown and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer layers than shown in the figure, and each layer may include more or fewer components, which is not limited in the present application.

[0071] It should be noted that although the embodiments of the present application are described using the Android system as an example, the basic principles are also applicable to Symbian-based ( ), Android ( ), apple ( ),blackberries ( ), and electronic devices with operating systems such as Hongmeng (Harmony).

[0072] See also Figure 2 , which shows a flow chart of a method for detecting video fluency provided by an embodiment of the present application. In a specific embodiment, the method for detecting video fluency can be applied to electronic devices, and can also be applied to processors or chips in electronic devices. The following takes electronic devices as an example. Figure 2 The process shown is described in detail, and the method for detecting video fluency may include the following steps S110 to S140.

[0073] Step S110: The electronic device obtains M photos of the playing video.

[0074] In an embodiment of the present application, when a user needs to detect the smoothness of a video, a detection instruction can be sent to an electronic device, and the electronic device receives and responds to the detection instruction to obtain M photos of the playing video.

[0075] Among them, M photos can correspond to N video frames of the played video, one video frame corresponds to at least one photo, N is a positive integer greater than or equal to 1, and M is a positive integer greater than or equal to N.

[0076] Regarding the electronic device receiving and responding to detection instructions, in some embodiments, the electronic device may be provided with an input panel. When the user needs to detect the smoothness of the video, the detection instruction may be input into the input panel of the electronic device, for example, the detection instruction may be input by hand into the input panel of the electronic device, or the detection instruction may be input by pressing a button on the input panel of the electronic device. The electronic device receives the detection instruction through the input panel.

[0077] In other embodiments, the electronic device may be provided with a voice recognition module. When the user needs to detect the video fluency, the user may send voice information within the voice collection range of the voice recognition module. The voice recognition module collects the voice information sent by the user, and performs voice recognition on the collected voice information to obtain a voice recognition result. When it is determined that the voice recognition result contains keywords for indicating the detection of video fluency, for example, the keyword is "video fluency detection", or for another example, the keywords are "video fluency" and "detection", etc., it is determined that the detection instruction has been received.

[0078] As an example, the voice message sent by the user is: to detect the video fluency of the played video, and the voice recognition result of the voice recognition contains the keywords "video fluency" and "detection", then it is determined that the detection instruction is received.

[0079] In an embodiment of the present application, after the electronic device receives and responds to the detection instruction, M photos of the playing video can be obtained in a variety of ways.

[0080] In some embodiments, the electronic device can send an acquisition instruction to a high-speed camera. The high-speed camera receives and responds to the acquisition instruction, takes pictures of the playback screen of the playing video, obtains M pictures, and returns the M pictures to the electronic device. The electronic device receives the M pictures returned by the high-speed camera, and takes M pictures of the playing video by the high-speed camera. The M pictures are closer to the continuous video frames of the playing video, which is beneficial to improving the calculation of the smoothness of the playing video, and further beneficial to improving the detection accuracy of the video smoothness detection.

[0081] The high-speed camera can be installed at a position where the playback screen of the player that plays the video can be captured. The high-speed camera can be used to shoot the playback screen of the player playing the video, such as Figure 3 As shown, it shows a schematic diagram of a scene of a photo taken by a high-speed camera. The high-speed camera is connected to an electronic device through a network, and data is exchanged with the electronic device through the network.

[0082] A high-speed camera is a device that can capture moving images with an exposure of less than 1 / 1000 second. It is mainly used to capture the motion trajectory of high-speed moving objects, helping to capture images and motion processes that are invisible to the naked eye. At present, the shooting speed of a high-speed camera can reach 1000 to 10,000 photos per second. Since the shooting time of each photo taken by a high-speed camera is less than the playback time of a video frame in a normal video, a high-speed camera can capture one or more photos of the same video frame.

[0083] The player may be an electronic device or other electronic product with a video playback function, etc., which is not limited here and may be specifically configured according to actual needs.

[0084] The network can be any one of a ZigBee network, a Bluetooth (BT) network, a Wireless Fidelity (Wi-Fi) network, a Thread network, a Long Range Radio (LoRa) network, a Low-Power Wide-Area Network (LPWAN), an infrared network, a Narrow Band Internet of Things (NB-IoT), a Controller Area Network (CAN), a Digital Living Network Alliance (DLNA) network, a Wide Area Network (WAN), a Local Area Network (LAN), a Metropolitan Area Network (MAN) or a Wireless Personal Area Network (WPAN), etc. The type of network is not limited here and can be set according to actual needs.

[0085] As an example, Figure 4As shown, the video fluency detection system may include an electronic device 100 and a high-speed camera 200. The electronic device 100 is connected to the high-speed camera 200 via a network, and data is exchanged with the high-speed camera 200 via the network. The player is the electronic device 100, and the high-speed camera 200 may be installed at a position for collecting the playback screen of the electronic device 100. The high-speed camera 200 may be used to shoot the playback screen of the video played by the electronic device 100, obtain M photos, and send the M photos to the electronic device 100 via the network.

[0086] In other implementations, the electronic device is equipped with a camera, and the player is other electronic products with a video playback function. The electronic device can use the camera to shoot the playback screen of the video played by the player to obtain M photos.

[0087] The camera may be any one of a wide-angle camera, a macro camera, an ultra-wide-angle camera or a panoramic camera, etc. The type of camera is not limited here and can be set according to actual needs.

[0088] Step S120: the electronic device determines a picture similarity according to every two adjacent photos in the M photos to obtain M-1 picture similarities.

[0089] In an embodiment of the present application, after the electronic device obtains M photos of the playing video, it can determine a picture similarity based on every two adjacent photos in the M photos to obtain M-1 picture similarities.

[0090] It can be understood that among M photos, starting from the first photo, two adjacent photos are taken as a group of photos, which can form M-1 groups of photos. A screen similarity is determined based on two adjacent photos in each group of photos, and then M-1 screen similarities can be obtained. For example, M=8, starting from the first photo, two adjacent photos are taken as a group of photos, which can form 7 groups of photos. A screen similarity is determined based on two adjacent photos in each group of photos, and then 7 screen similarities can be obtained. The value of M is not limited here, and can be set according to actual needs.

[0091] In the embodiment of the present application, the electronic device can determine the similarities of M-1 pictures in a variety of ways based on every two adjacent photos in the M photos.

[0092] In some embodiments, after the electronic device obtains M photos of the playing video, it can calculate the first hash value of one photo and the second hash value of the other photo of any two adjacent photos, and calculate the Hamming distance of any two adjacent photos based on the first hash value and the second hash value, and calculate the picture similarity of any two adjacent photos based on the Hamming distance to obtain M-1 picture similarities. The hash algorithm has the characteristics of high computational efficiency and high robustness to image transformation. The picture similarity of photos is calculated based on the hash algorithm, which improves the computational efficiency and computational robustness of calculating the picture similarity of photos.

[0093] The electronic device may calculate a first hash value of one photo and a second hash value of another photo in any two adjacent photos based on an Average Hash algorithm or a Perceptual Hash algorithm.

[0094] The Hamming distance is inversely proportional to the image similarity. The inverse of the Hamming distance can be calculated to obtain the image similarity of any two adjacent photos. The smaller the Hamming distance, the higher the image similarity, and the larger the Hamming distance, the lower the image similarity.

[0095] The mean hash algorithm refers to reducing a photo to a fixed size (for example, 8×8 pixels), then converting the reduced photo into a grayscale image, calculating the average grayscale value of the grayscale image, comparing the grayscale value of each pixel of the grayscale image with the average grayscale value, marking pixels in the grayscale image that are larger than the average grayscale value as 1, and marking pixels in the grayscale image that are smaller than the average grayscale value as 0, obtaining a binary result, and combining the binary results into a hash value of a fixed length.

[0096] The perceptual hashing algorithm refers to an algorithm that converts a photo into a grayscale image, resizes the grayscale image to a fixed size (e.g., 32×32 pixels), applies a discrete cosine transform (DCT) to the resized grayscale image, retains low-frequency components, and converts the resized grayscale image into a binary hash value based on the relative size of the DCT coefficients.

[0097] In some embodiments, after the electronic device obtains M photos of the video being played, the first eigenvector of one photo and the second eigenvector of the other photo of any two adjacent photos can be extracted through a convolutional neural network, and the cosine similarity of any two adjacent photos is calculated based on the first eigenvector and the second eigenvector, and the screen similarity of any two adjacent photos is determined based on the cosine similarity to obtain M-1 screen similarities. The cosine similarity algorithm has a certain stability to noise and changes in the data. The screen similarity of the photos is calculated based on the cosine similarity algorithm, which improves the calculation accuracy of the screen similarity of the photos.

[0098] The calculation formula of cosine similarity is as follows: similarity = (A·B) / (││A││*││B││); where A represents the first eigenvector, B represents the second eigenvector, A·B represents the dot product of the first eigenvector and the second eigenvector, ││A││ represents the norm of the first eigenvector (i.e., the length of the first eigenvector), ││B││ represents the norm of the second eigenvector (i.e., the length of the second eigenvector), and similarity represents the cosine similarity between the first eigenvector and the second eigenvector.

[0099] The value range of similarity is between -1 and 1. The closer the similarity value is to 1, the more similar the first eigenvector is to the second eigenvector. The closer the similarity value is to -1, the less similar the first eigenvector is to the second eigenvector. The closer the similarity value is to 0, the less similarity or difference there is between the first eigenvector and the second eigenvector.

[0100] The feature vector is obtained by extracting features from the photo based on a convolutional neural network, and the cosine similarity between the first feature vector and the second feature vector can be determined as the picture similarity between any two corresponding adjacent photos.

[0101] In some embodiments, after the electronic device obtains M photos of the playing video, it can convert one of any two adjacent photos into a first grayscale image and the other photo into a second grayscale image, and perform histogram calculation on the first grayscale image based on a histogram algorithm to obtain a first grayscale histogram, perform histogram calculation on the second grayscale image based on a histogram algorithm to obtain a second grayscale histogram, and calculate the Euclidean distance between any two adjacent photos based on the first grayscale histogram and the second grayscale histogram, and determine the picture similarity between any two adjacent photos based on the Euclidean distance to obtain M-1 picture similarities.

[0102] Among them, the Euclidean distance is inversely proportional to the image similarity. The inverse of the Euclidean distance can be calculated to obtain the image similarity of any two adjacent photos. The smaller the Euclidean distance, the higher the image similarity, and the larger the Euclidean distance, the lower the image similarity.

[0103] In some embodiments, after the electronic device obtains M photos of the video being played, it can select any two adjacent photos in sequence, and input the selected any two adjacent photos in sequence into a pre-trained picture similarity extraction model. The picture similarity extraction model receives and responds to the input any two adjacent photos in sequence, and outputs the picture similarities of any two adjacent photos in sequence, to obtain M-1 picture similarities.

[0104] Among them, the picture similarity extraction model can be used to extract the picture similarity of any two photos. The picture similarity extraction model can be a convolutional neural network (CNN) model, a deep belief network (DBN) model, a stacked auto encoder network (SAE) model, a recurrent neural network (RNN) model, a deep neural network (DNN) model, a long short-term memory (LSTM) network model or a gated recurring unit (GRU) model, etc. The type of the picture similarity extraction model is not limited here, and it can be set according to actual needs.

[0105] Step S130: the electronic device determines the frame play duration of each video frame in at least part of the N video frames according to the M-1 picture similarities.

[0106] In an embodiment of the present application, the electronic device determines a picture similarity based on every two adjacent photos in M ​​photos to obtain M-1 picture similarities. Then, the frame playback duration of each video frame in at least some of the N video frames can be determined based on the M-1 picture similarities.

[0107] Among them, at least part of the video frames may be part of the continuous video frames in the N video frames, or may be all the video frames in the N video frames, etc. For example, N=5, the 5 video frames are respectively video frame 1, video frame 2, video frame 3, video frame 4 and video frame 5, at least part of the video frames may be video frame 1 and video frame 2, at least part of the video frames may also be video frame 1, video frame 2 and video frame 3, at least part of the video frames may also be video frame 3, video frame 4 and video frame 5, at least part of the video frames may also be video frame 1, video frame 2, video frame 3, video frame 4 and video frame 5, etc., which is not limited here.

[0108] In some embodiments, the electronic device determines a picture similarity based on every two adjacent photos in M ​​photos to obtain M-1 picture similarities. Then, according to the correspondence between the M-1 picture similarities and the M-1 time points in the playback duration of the playback video, the frame playback duration of each video frame in at least some of the video frames is determined. The picture similarity when the video frames are switched is low. According to the correspondence between the picture similarities and the playback time points of the playback video, the switching duration of the video frames can be accurately calculated to obtain the frame playback duration of each video frame in at least some of the video frames, thereby improving the calculation accuracy of calculating the frame playback duration of each video frame in at least some of the video frames.

[0109] Specifically, the electronic device determines a picture similarity based on every two adjacent photos in M ​​photos to obtain M-1 picture similarities. Then, according to the correspondence between the M-1 picture similarities and the M-1 time points in the playback duration of the playback video, the duration between two time points corresponding to the minimum values ​​of each two adjacent picture similarities is determined as the frame playback duration of a video frame to obtain the frame playback duration of each video frame in at least some of the video frames. The minimum value of the picture similarity corresponds to the video frame switching moment of the playback video. The frame playback duration of each video frame in at least some of the video frames is calculated based on the time corresponding to the minimum value of the picture similarity, thereby improving the calculation accuracy of calculating the frame playback duration of each video frame in at least some of the video frames.

[0110] In an application scenario, the correspondence between M-1 picture similarities and M-1 time points in the playback duration of the video can be represented by a time-picture similarity curve. If the frame rate (Frames Per Second, fps) of the video is 24, a picture similarity curve that changes with time can be drawn according to the M-1 picture similarities to obtain a time-picture similarity curve, such as Figure 5As shown, the time difference between adjacent troughs in the time-picture similarity curve can be calculated, and each trough corresponds to a minimum picture similarity value, that is, the duration between two time points corresponding to two adjacent minimum picture similarity values ​​is calculated, and each time difference is determined as the frame playback duration of a video frame, so as to obtain the frame playback duration of each video frame in at least part of the video frames.

[0111] For example, as shown in the figure, starting from left to right, the troughs can be selected, and the time difference corresponding to the first trough and the second trough can be determined as the frame playback duration of video frame 1 of the played video, the time difference corresponding to the second trough and the third trough can be determined as the frame playback duration of video frame 2 of the played video, the time difference corresponding to the third trough and the fourth trough can be determined as the frame playback duration of video frame 3 of the played video, the time difference corresponding to the fourth trough and the fifth trough can be determined as the frame playback duration of video frame 4 of the played video, the time difference corresponding to the fifth trough and the sixth trough can be determined as the frame playback duration of video frame 5 of the played video, and the time difference corresponding to the sixth trough and the seventh trough can be determined as the frame playback duration of video frame 6 of the played video. Video frames 1 to 6 are continuous video frames.

[0112] Among them, video frame 1 can be any video frame of the playing video. For example, video frame 1 can be the first video frame of the playing video, or it can be the second video frame of the playing video, etc., which is not limited here.

[0113] Step S140: the electronic device determines the smoothness of playing the video according to the frame playing duration of each video frame in at least part of the video frames.

[0114] In an embodiment of the present application, after the electronic device determines the frame playback duration of each video frame in at least some of the N video frames based on M-1 picture similarities, it can determine the smoothness of the video playback based on the frame playback duration of each video frame in at least some of the video frames, thereby achieving, in the detection process of detecting the video smoothness, based on the picture similarity of every two adjacent photos in the M photos, determining the frame playback duration of each video frame in at least some of the N video frames of the video playback, and calculating the smoothness of the video playback based on the frame playback duration of each video frame, and expressing the user's subjective feeling of the video playback freeze with a specific numerical value, making the evaluation of the video smoothness more objective and effective, and improving the detection accuracy of the video smoothness detection.

[0115] The smoothness of video playback is calculated based on the frame playback duration of the video frames. A unified method for quantifying smoothness is used to detect stuttering caused by various reasons, such as stuttering caused by mismatch between video frame rate and screen refresh rate, stuttering caused by GPU rendering abnormalities, and stuttering caused by frame drops, etc. This expands the application scenarios for video smoothness detection.

[0116] The picture similarity when the video frame switches is low. According to the low picture similarity, the switching duration of the video frame can be accurately calculated to obtain the frame playback duration of the video frame, thereby improving the calculation accuracy of the frame playback duration of the video frame.

[0117] In some implementations, after the electronic device determines the frame playback duration of each video frame in at least some of the N video frames based on M-1 picture similarities, it can determine the freeze rate of the video playback based on the frame playback duration of each video frame in at least some of the video frames, and determine the smoothness of the video playback based on the freeze rate.

[0118] In the above process of determining the freeze rate, the electronic device can determine the reference frame playback duration based on the frame playback duration of each video frame in at least part of the video frames, and determine the freeze duration of the played video based on the reference frame playback duration and the frame playback duration of each video frame in at least part of the video frames, and determine the freeze rate of the played video based on the total playback duration of N video frames and the freeze duration.

[0119] Among them, the baseline frame playback time can be used to characterize the standard playback time of each video frame of the played video, and the freeze rate can be calculated according to the total playback time of N video frames and the freeze time according to Formula 1. Formula 1 is: freeze rate = freeze time / total playback time of N video frames.

[0120] Regarding the process in which the above-mentioned electronic device can determine the benchmark frame playback duration based on the frame playback durations of each video frame in at least part of the video frames, as an implementation mode, the electronic device can count the number of repetitions of the frame playback durations of each video frame in at least part of the video frames, and determine the frame playback duration with the largest number of repetitions among the frame playback durations of each video frame in at least part of the video frames as the benchmark frame playback duration.

[0121] As another implementation, the electronic device may calculate the average frame playback duration of each video frame based on the frame playback duration of each video frame in at least part of the video frames, and determine the average frame playback duration as the reference frame playback duration.

[0122] Regarding the process of the above-mentioned electronic device determining the freeze duration of the video playback based on the baseline frame playback duration and the frame playback duration of each video frame in at least part of the video frames, as an implementation mode, the electronic device can sum the absolute values ​​of the differences between the playback durations of each non-baseline frame and the playback duration of the baseline frame to obtain the freeze duration of the video playback.

[0123] After obtaining the freeze duration of the video playback, the electronic device can calculate the quotient of the freeze duration divided by the total playback duration of the N video frames according to Formula 1 based on the freeze duration and the total playback duration of the N video frames to obtain the freeze rate of the video playback.

[0124] Regarding the process of the electronic device determining the smoothness of video playback based on the jam rate, in some implementations, the electronic device can calculate 1 minus the percentage of the jam rate to obtain the smoothness of video playback.

[0125] In an application scenario, when the video frame rate of the played video is 24fps and the screen refresh rate of the played video is 30Hz, each periodic frame of the played video includes video frame 1, video frame 2, video frame 3 and video frame 4. According to the picture similarity, the frame playback time of video frame 1 can be determined to be 1 / 30s, the frame playback time of video frame 2 can be 1 / 30s, the frame playback time of video frame 3 can be 1 / 30s, and the frame playback time of video frame 4 can be determined to be 2 / 30s.

[0126] The frame playback duration of 1 / 30s is repeated three times, that is, the frame playback duration of 1 / 30s is repeated the most times, and the frame playback duration of 1 / 30s is determined as the reference frame playback duration.

[0127] The absolute value of the difference between the non-reference frame playback duration 2 / 30s and the reference frame playback duration 1 / 30s is calculated, and the freeze duration = |2 / 30s-1 / 30s|=1 / 30s is obtained.

[0128] The total playback time of each periodic frame = (1 / 30s + 1 / 30s + 1 / 30s + 2 / 30s) = 1 / 6s, the freeze rate of the video playback = freeze time / total playback time of each periodic frame = (1 / 30s) / (1 / 6s) = 0.2, the smoothness of the video playback = 1-0.2*100% = 80%.

[0129] The periodic frame is used to represent the number of frames that match the video frame and the screen refresh rate, and the periodic frame is associated with the player that plays the video.

[0130] In an application scenario, the correspondence between the video frame rate, screen refresh rate, number of video frame sequence refreshes, frame playback time of the video frame sequence, reference frame playback time, stuttering rate and smoothness of the video can be shown in Table 1.

[0131] Table 1

[0132]

[0133] In one application scenario, such as Figure 6 As shown, the video fluency detection method may include steps S210 to S270. It should be noted that, Figure 6 It is a combination Figure 4 The video fluency detection system shown in the figure schematically illustrates a video fluency detection method in an application scenario.

[0134] Step S210: The electronic device uses a high-speed camera to take pictures of the playback screen of the video to obtain M pictures.

[0135] The M photos correspond to N video frames of the played video, and one video frame corresponds to at least one photo.

[0136] Step S220: The electronic device calculates the picture similarity between every two adjacent photos in the M photos through an image similarity algorithm to obtain M-1 picture similarities.

[0137] Among them, the image similarity algorithm can include any one of the mean hash algorithm, perceptual hash algorithm, cosine similarity algorithm, histogram algorithm and picture similarity model detection algorithm, etc. The type of image similarity algorithm is not limited here and can be set according to actual needs.

[0138] Step S230: the electronic device draws a time-picture similarity curve according to the M-1 picture similarities.

[0139] Step S240: the electronic device determines the frame play duration of each video frame in at least part of the N video frames based on the time-picture similarity curve.

[0140] Step S250: the electronic device determines a reference frame playback duration according to the frame playback durations of each video frame in at least part of the video frames.

[0141] Specifically, the frame playback duration with the largest number of repetitions among the frame playback durations of the respective video frames in at least part of the video frames is determined as the reference frame playback duration.

[0142] Step S260: The electronic device determines the freeze duration of the video playback based on the playback duration of the reference frame.

[0143] Specifically, the absolute values ​​of the differences between the playback durations of each non-reference frame and the playback duration of the reference frame are summed to obtain the freeze duration of the playback video.

[0144] Step S270: The electronic device calculates the smoothness of the video playback based on the freeze duration.

[0145] Specifically, according to the total playback duration and the freeze duration of N video frames, the freeze rate of the playback video is calculated, and the smoothness of the playback video is calculated according to the freeze rate.

[0146] In some embodiments, after the electronic device determines the smoothness of video playback based on the frame playback duration of each video frame in at least part of the video frames, the smoothness can be determined as a first drawing vector, and the playback time corresponding to the smoothness can be determined as a second drawing vector, and the first drawing vector and the second drawing vector can be used to draw and display a fluency change graph, so that the user can more intuitively and simply view the changes in video fluency through the fluency change graph, thereby improving the user's detection experience of video smoothness detection.

[0147] In some embodiments, after the electronic device determines the smoothness of video playback based on the frame playback duration of each video frame in at least part of the video frames, it can generate a detection result including the smoothness, and send the detection result to a server for storage, so that the user can trace the video smoothness detection process according to the detection result stored in the server, thereby improving the user's detection experience of video smoothness detection.

[0148] The server is connected to the electronic device through the network and exchanges data with the electronic device through the network. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), big data or artificial intelligence platforms, etc. The type of server is not limited here and can be set according to actual needs.

[0149] This embodiment provides a solution, by acquiring M photos of a playing video, the M photos correspond to N video frames of the playing video, one video frame corresponds to at least one photo, and a picture similarity is determined according to every two adjacent photos in the M photos to obtain M-1 picture similarities, and the frame playback duration of each video frame in at least some of the N video frames is determined according to the M-1 picture similarities, and the smoothness of the playing video is determined according to the frame playback duration of each video frame in at least some of the video frames, so that in the detection process of detecting the video smoothness, the frame playback duration of each video frame in at least some of the N video frames of the playing video is determined based on the picture similarity of every two adjacent photos in the M photos, and the smoothness of the playing video is calculated based on the frame playback duration of each video frame, and the user's subjective feeling of the video freeze is expressed by a specific numerical value, so that the evaluation of the video smoothness is more objective and effective, and the detection accuracy of the video smoothness is improved.

[0150] The smoothness of video playback is calculated based on the frame playback duration of the video frames. A unified method for quantifying smoothness is used to detect stuttering caused by various reasons, such as stuttering caused by mismatch between video frame rate and screen refresh rate, stuttering caused by GPU rendering abnormalities, and stuttering caused by frame drops, etc. This expands the application scenarios for video smoothness detection.

[0151] The picture similarity when the video frame switches is low. According to the low picture similarity, the switching duration of the video frame can be accurately calculated to obtain the frame playback duration of the video frame, thereby improving the calculation accuracy of the frame playback duration of the video frame.

[0152] See also Figure 7 , which shows a flow chart of a method for detecting video fluency provided by another embodiment of the present application. In a specific embodiment, the method for detecting video fluency can be applied to electronic devices, and can also be applied to processors or chips in electronic devices. The following takes electronic devices as an example. Figure 7 The process shown is described in detail, and the method for detecting video fluency may include the following steps S310 to S350.

[0153] Step S310: The electronic device obtains M photos of the playing video.

[0154] Step S320: the electronic device determines a picture similarity according to every two adjacent photos in the M photos to obtain M-1 picture similarities.

[0155] Step S330: the electronic device determines the frame play duration of each video frame in at least part of the N video frames according to the M-1 picture similarities.

[0156] Step S340: the electronic device determines the smoothness of the video playback according to the frame playback duration of each video frame in at least part of the video frames.

[0157] In this embodiment, step S310, step S320, step S330 and step S340 may refer to the contents of the corresponding steps in the aforementioned embodiments, which will not be described in detail here.

[0158] Step S350: When the fluency is less than or equal to the fluency threshold, the electronic device generates a playback mode switching prompt message.

[0159] In this embodiment, the electronic device determines the smoothness of video playback based on the frame playback duration of each video frame in at least part of the video frames, and when the smoothness is less than or equal to the smoothness threshold, generates playback mode switching prompt information, so that the user can switch the playback mode of the video to the standard definition playback mode according to the playback mode switching prompt information, which is beneficial to improving the smoothness of video playback and enhancing the user's detection experience of video smoothness.

[0160] Among them, the playback mode switching prompt information may include at least any one of text prompt information, sound prompt information and light prompt information, etc. The type of the playback mode switching prompt information is not limited here and can be set according to actual needs.

[0161] The playback mode can be any one of standard definition playback mode, high definition playback mode, ultra high definition playback mode or Blu-ray playback mode, etc. The type of playback mode is not limited here and can be set according to actual needs.

[0162] The fluency threshold may be 80%, 95%, 70%, etc. The fluency threshold may be a fluency preset by a user, or may be a fluency automatically generated by an electronic device according to a detection process of multiple video fluency detections, etc. The setting method of the fluency threshold is not limited here, and may be set according to actual needs.

[0163] In some embodiments, after the electronic device determines the smoothness of video playback based on the frame playback duration of each video frame in at least part of the video frames, and when the smoothness is less than or equal to the smoothness threshold, it obtains the stack information of the player playing the video, and determines the cause of the freeze in the video playback based on the stack information, so that the user can process the video playback according to the cause of the freeze to improve the smoothness of the video playback.

[0164] Among them, generally the interface display and rendering threads are the main threads. The blockage of the main thread will cause the video playback to freeze. Therefore, by checking the stack information of the main thread, it is possible to determine which operation caused the video playback freeze based on the information.

[0165] The present embodiment provides a solution, by acquiring M photos of a video to be played, and determining a picture similarity according to every two adjacent photos in the M photos to obtain M-1 picture similarities, and determining the frame playback duration of each video frame in at least some of the N video frames according to the M-1 picture similarities, and determining the smoothness of the video playback according to the frame playback duration of each video frame in at least some of the video frames, when the smoothness is less than or equal to the smoothness threshold, generating a playback mode switching prompt information, so that in the detection process of detecting the video smoothness, based on the picture similarity of every two adjacent photos in the M photos, the frame playback duration of each video frame in at least some of the N video frames of the video to be played is determined, and the smoothness of the video playback is calculated based on the frame playback duration of each video frame, and the user's subjective feeling of the video playback freeze is expressed by a specific numerical value, so that the evaluation of the video fluency is more objective and effective, and the detection accuracy of the video fluency detection is improved.

[0166] Furthermore, when the fluency is less than or equal to the fluency threshold, a playback mode switching prompt message is generated so that the user can switch the playback mode of the video to the standard definition playback mode according to the playback mode switching prompt message, which is beneficial to improving the fluency of the video playback and enhancing the user's detection experience of the video fluency.

[0167] See also Figure 8 , which shows a video fluency detection device 500 provided by an embodiment of the present application. The video fluency detection device 500 can be applied to electronic devices, and can also be applied to processors or chips in electronic devices. The following takes electronic devices as an example. Figure 8 The video fluency detection device 500 shown in the figure is described in detail. The video fluency detection device 500 may include an acquisition module 510, a first determination module 520, a second determination module 530 and a third determination module 540.

[0168] The acquisition module 510 can be used to acquire M photos of the playing video, where the M photos can correspond to N video frames of the playing video, and one video frame can correspond to at least one photo; the first determination module 520 can be used to determine a picture similarity based on every two adjacent photos in the M photos to obtain M-1 picture similarities; the second determination module 530 can be used to determine the frame playback duration of each video frame in at least some of the N video frames based on the M-1 picture similarities; the third determination module 540 can be used to determine the smoothness of the playing video based on the frame playback duration of each video frame in at least some of the video frames.

[0169] In some implementations, the second determination module 530 may include a first determination submodule.

[0170] The first determination submodule may be used to determine the frame playback duration of each video frame in at least part of the video frames according to the correspondence between the M-1 picture similarities and the M-1 time points in the playback duration of the playback video.

[0171] In some implementations, the first determining submodule may include a first determining unit.

[0172] The first determination unit can be used to determine the duration between two time points corresponding to the minimum values ​​of each two adjacent picture similarities as the frame playback duration of a video frame based on the correspondence between M-1 picture similarities and M-1 time points in the playback duration of the playback video, so as to obtain the frame playback duration of each video frame in at least part of the video frames.

[0173] In some implementations, the third determination module 540 may include a second determination submodule and a third determination submodule.

[0174] The second determination submodule can be used to determine the jam rate of the played video based on the frame playback duration of each video frame in at least part of the video frames; the third determination submodule can be used to determine the smoothness of the played video based on the jam rate.

[0175] In some implementations, the second determining submodule may include a second determining unit, a third determining unit, and a fourth determining unit.

[0176] The second determination unit can be used to determine the base frame playback duration based on the frame playback duration of each video frame in at least part of the video frames; the third determination unit can be used to determine the freeze duration of the playback video based on the base frame playback duration and the frame playback duration of each video frame in at least part of the video frames; the fourth determination unit can be used to determine the freeze rate of the playback video based on the total playback duration of N video frames and the freeze duration.

[0177] In some embodiments, the second determining unit may include a first determining subunit.

[0178] The first determining subunit may be configured to determine the frame playback duration having the largest number of repetitions among the frame playback durations of the respective video frames in at least part of the video frames as the reference frame playback duration.

[0179] In some implementations, the third determining unit may include a summing subunit.

[0180] The summing subunit may be used to sum the absolute values ​​of the differences between the playback durations of each non-reference frame and the playback duration of the reference frame to obtain the freeze duration of the playback video.

[0181] In some implementations, the second determining unit may further include a calculating subunit and a second determining subunit.

[0182] The calculation subunit may be used to calculate an average frame playback duration according to the frame playback durations of each video frame in at least part of the video frames; the second determination subunit may be used to determine the average frame playback duration as the reference frame playback duration.

[0183] In some implementations, the third determining submodule may be a computing unit.

[0184] The calculation unit can be used to calculate the percentage of 1 minus the freeze rate to obtain the smoothness of the video playback.

[0185] In some implementations, the first determination module 520 may include a first calculation submodule, a second calculation submodule, and a third calculation submodule.

[0186] The first calculation submodule can be used to calculate the first hash value of one photo and the second hash value of the other photo of any two adjacent photos; the second calculation submodule can be used to calculate the Hamming distance of any two adjacent photos according to the first hash value and the second hash value; the third calculation submodule can be used to calculate the picture similarity of any two adjacent photos according to the Hamming distance.

[0187] In some implementations, the acquisition module 510 may include a sending submodule and a receiving submodule.

[0188] The sending submodule can be used to send an acquisition instruction to the high-speed camera, so that the high-speed camera shoots the playback screen of the playing video according to the acquisition instruction to obtain M photos; the receiving submodule can be used to receive the M photos returned by the high-speed camera.

[0189] The solution provided by this embodiment obtains M photos of the video being played, the M photos correspond to N video frames of the video being played, one video frame corresponds to at least one photo, and a picture similarity is determined according to every two adjacent photos in the M photos to obtain M-1 picture similarities, and the frame playback duration of each video frame in at least some of the N video frames is determined according to the M-1 picture similarities, and the smoothness of the video being played is determined according to the frame playback duration of each video frame in at least some of the video frames. In the detection process of detecting the video smoothness, based on the picture similarity of every two adjacent photos in the M photos, the frame playback duration of each video frame in at least some of the N video frames of the video being played is determined, and the smoothness of the video being played is calculated based on the frame playback duration of each video frame, and the user's subjective feeling of the video freeze is expressed by a specific numerical value, so that the evaluation of the video smoothness is more objective and effective, and the detection accuracy of the video smoothness is improved.

[0190] The smoothness of video playback is calculated based on the frame playback duration of the video frames. A unified method for quantifying smoothness is used to detect stuttering caused by various reasons, such as stuttering caused by mismatch between video frame rate and screen refresh rate, stuttering caused by GPU rendering abnormalities, and stuttering caused by frame drops, etc. This expands the application scenarios for video smoothness detection.

[0191] The picture similarity when the video frame switches is low. According to the low picture similarity, the switching duration of the video frame can be accurately calculated to obtain the frame playback duration of the video frame, thereby improving the calculation accuracy of the frame playback duration of the video frame.

[0192] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Any processing method described in the method embodiment can be implemented by the corresponding processing module in the device embodiment, and will not be repeated one by one in the device embodiment.

[0193] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0194] See also Fig. 9 , which shows a hardware structure diagram of an electronic device 100 provided by an embodiment of the present application. Fig. 9 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a Subscriber Identification Module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0195] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0196] For example, Fig. 9 The processor 110 shown may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0197] The AP may be used to control and manage a fluency detection application. For example, the AP may control the fluency detection application to perform a fluency detection on a played video.

[0198] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0199] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0200] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General Purpose Input / Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and / or a Universal Serial Bbus (USB) interface, etc.

[0201] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, and the GPU connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0202] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light emitting diode (OLED), an active-matrix organic light emitting diode, an active-matrix organic light emitting diode (AMOLED), a flexible light emitting diode (FLED), Mini-LED, Micro-LED, Micro-OLED, quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0203] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0204] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0205] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0206] The external memory interface 120 can be used to connect an external memory card, such as a Secure Digital (SD) card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as captured images and videos are saved in the external memory card.

[0207] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as an audio acquisition function, an image shooting function, etc.), etc. The data storage area may store data (such as audio data, image data) created during the use of the electronic device 100, etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (Universal Flash Storage, UFS), etc.

[0208] See also Fig.10, which shows a functional block diagram of an electronic device 300 according to an embodiment of the present application. Fig.10 As shown, the electronic device 300 includes: one or more processors 310 ( Fig.10 Only one processor is shown in the figure) and a memory 320, the memory 320 is coupled to one or more processors 310, the memory 320 is used to store computer program code 330, the computer program code 330 includes computer instructions, and one or more processors 310 call the computer instructions to enable the electronic device 300 to implement the steps in any of the above methods.

[0209] Those skilled in the art will understand that Fig.10 This is only an example of the electronic device 300 and does not constitute a limitation on the electronic device 300. In practice, the electronic device 300 may include more or fewer components than shown in the figure, or combine certain components, or different components, for example, it may also include input and output devices, network access devices, etc. The electronic device 300 may also be the same device as the electronic device 100 described in the above embodiment.

[0210] The processor 310 may be a central processing unit (CPU), other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0211] In some embodiments, the memory 320 may be an internal storage unit of the electronic device 300, such as a hard disk or memory of the electronic device 300. In other embodiments, the memory 320 may also be an external storage device of the electronic device 300, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 300. Optionally, the memory 320 may also include both an internal storage unit of the electronic device 300 and an external storage device. The memory 320 is used to store an operating system, an application program, a boot loader, data, and other programs, such as program codes of a computer program. The memory 320 may also be used to temporarily store data that has been output or is to be output.

[0212] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0213] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0214] An embodiment of the present application also provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device implements the steps in any one of the above methods.

[0215] In some embodiments, the chip system also includes a memory, and the memory is connected to one or more processors through circuits or wires.

[0216] In some embodiments, the chip system also includes a communication interface.

[0217] An embodiment of the present application also provides a computer-readable storage medium, which includes instructions. When the instructions are executed on an electronic device, the electronic device implements the methods described in the above-mentioned various method embodiments.

[0218] The embodiments of the present application also provide a computer program product. When the computer program product is run on an electronic device, the electronic device executes the above-mentioned related steps to implement the methods described in the above-mentioned various method embodiments.

[0219] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0220] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0221] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0222] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0223] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0224] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0225] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0226] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0227] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some implementations", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0228] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting video fluency, characterized in that: The method comprises: Obtaining M photos of the playing video, where the M photos correspond to N video frames of the playing video, and one video frame corresponds to at least one photo; Determine a picture similarity according to every two adjacent photos in the M photos to obtain M-1 picture similarities; Determine the frame playback duration of each video frame in at least part of the N video frames according to the M-1 picture similarities; The smoothness of the video playback is determined according to the frame playback duration of each video frame in the at least part of the video frames.

2. The method according to claim 1, characterized in that: The determining, according to the M-1 picture similarities, the frame playback duration of each video frame in at least part of the N video frames comprises: The frame playback duration of each video frame in the at least part of the video frames is determined according to the correspondence between the M-1 picture similarities and the M-1 time points in the playback duration of the playback video.

3. The method according to claim 2, characterized in that The determining, according to the correspondence between the M-1 picture similarities and the M-1 time points in the playback duration of the playback video, the frame playback duration of each video frame in the at least part of the video frames comprises: According to the correspondence between the M-1 picture similarities and the M-1 time points in the playback duration of the playback video, the duration between the two time points corresponding to each two adjacent picture similarity minima is determined as the frame playback duration of a video frame to obtain the frame playback duration of each video frame in the at least part of the video frames.

4. The method according to any one of claims 1 to 3, characterized in that The determining the smoothness of the video playback according to the frame playback duration of each video frame in the at least part of the video frames includes: Determining a freeze rate of the played video according to a frame play duration of each video frame in the at least part of the video frames; The smoothness of the video playback is determined according to the jam rate.

5. The method according to claim 4, characterized in that The determining the jam rate of the played video according to the frame play duration of each video frame in the at least part of the video frames includes: Determining a reference frame playback duration according to the frame playback duration of each video frame in the at least part of the video frames; Determining the freeze duration of the played video according to the playback duration of the reference frame and the frame playback duration of each video frame in the at least part of the video frames; The jam rate of the played video is determined according to the total playback duration of the N video frames and the jam duration.

6. The method according to claim 5, characterized in that The determining the reference frame playback duration according to the frame playback duration of each video frame in the at least part of the video frames includes: The frame playback duration with the largest number of repetitions among the frame playback durations of the respective video frames in the at least part of the video frames is determined as the reference frame playback duration.

7. The method according to claim 5 or 6, characterized in that: The determining the freeze duration of the played video according to the playback duration of the reference frame and the frame playback duration of each video frame in the at least part of the video frames includes: The absolute values ​​of the differences between the playback durations of each non-reference frame and the playback duration of the reference frame are summed to obtain the freeze duration of the playback video.

8. The method according to claim 5, characterized in that The determining the reference frame playback duration according to the frame playback duration of each video frame in the at least part of the video frames includes: Calculating an average frame playback duration according to the frame playback duration of each video frame in the at least part of the video frames; The average frame playback duration is determined as the reference frame playback duration.

9. The method according to any one of claims 4 to 8, characterized in that The determining the smoothness of the video playback according to the jam rate includes: The percentage of the jam rate is calculated by subtracting 1 to obtain the smoothness of the video playback.

10. The method according to any one of claims 1 to 9, characterized in that The determining a picture similarity based on every two adjacent photos in the M photos includes: Calculate a first hash value of one photo and a second hash value of the other photo in any two adjacent photos; Calculate the Hamming distance between any two adjacent photos according to the first Hash value and the second Hash value; The picture similarity between any two adjacent photos is calculated according to the Hamming distance.

11. The method according to any one of claims 1 to 10, characterized in that The step of obtaining M photos of the video to be played includes: Sending a collection instruction to the high-speed camera, so that the high-speed camera takes pictures of the playback screen of the video according to the collection instruction, and obtains M pictures; Receive the M photos returned by the high-speed camera.

12. A device for detecting video fluency, characterized in that: The device comprises: An acquisition module, used to acquire M photos of the playing video, wherein the M photos correspond to N video frames of the playing video, and one video frame corresponds to at least one photo; A first determination module is used to determine a picture similarity according to every two adjacent photos in the M photos to obtain M-1 picture similarities; A second determination module is used to determine the frame playback duration of each video frame in at least part of the N video frames according to the M-1 picture similarities; The third determination module is used to determine the smoothness of the video playback according to the frame playback duration of each video frame in the at least part of the video frames.

13. An electronic device, characterized in that: The electronic device includes: one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method as claimed in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, and when the instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 11.

Images