Method and device for detecting video lag based on frame difference momentum, and storage medium

Abstract

The invention belongs to the technical field of video quality detection, and particularly discloses a method and equipment for detecting video lag based on frame difference momentum and a storage medium. According to the method provided by the invention, whether the video frames behind the video stream are lagged or not is judged according to the dynamic factors calculated according to the video frames before the video stream, and the method provided by the invention not only can save computing resources, but also realizes higher detection speed on the premise of ensuring the detection accuracy due to a simple detection process and a small amount of calculation; besides, the dynamic factor is generated along with the video stream and does not need to be manually set, and mapping exists between the dynamic factor and the frame rate of the video stream, so that the mapping ensures one-to-one correspondence between the dynamic factor and the video frame rate, and the video frame rate is more accurate. Therefore, the method provided by the invention not only can improve the accuracy of video lagging detection on the premise of ensuring generalization, but also realizes self-adaptive video lagging detection without manual intervention.

Description

Technical Field

[0001] The present invention belongs to the technical field of video quality detection, and in particular relates to a method, device and storage medium for detecting video freeze based on frame difference momentum. Background Art

[0002] In modern video technology, video smoothness is an important factor affecting user experience. For leisure and entertainment videos, stuttering will affect the viewing experience, while for non-entertainment video transmissions, stuttering may cause interruption of important information transmission. In addition, in some online conference scenarios, video stuttering will cause picture delays, inconsistent sound and picture, and other phenomena, which will greatly reduce the effectiveness of online meetings.

[0003] At present, the conventional technical means is to train the neural network model through deep learning methods, and then classify and regress the video to detect the video freeze phenomenon. However, on the one hand, due to the huge parameters of the neural network, not only are there requirements for the hardware equipment for calculation, but the calculation time is also difficult to achieve real-time, and it may even occupy some computing resources, further aggravating the freeze phenomenon of the original video; on the other hand, when the trained neural network detects the video, because it needs to further classify and regress the detection results, it is necessary to manually set the threshold, and it is impossible to adaptively adjust the detection threshold according to the actual detection video. This will make the video detection method lack generalization. If there are large differences between the training and test videos, the detection effect will also be unsatisfactory. Summary of the invention

[0004] In order to address the deficiencies in the prior art, the present invention provides a method, device and storage medium for detecting video freeze based on frame difference momentum. The method, device and medium provided by the present invention can not only improve the detection accuracy while ensuring generalization, but also realize adaptive video freeze detection without human intervention; in addition, compared with deep neural networks or other machine learning methods, the method proposed in the present invention can not only save computing resources, but also achieve faster detection speed while ensuring detection accuracy.

[0005] According to a first aspect of the present invention, a method for detecting video freeze based on frame differential momentum is provided, and the specific steps are as follows:

[0006] S1, obtaining a time-series image data set of the video stream from the previous video frames of the video stream, specifically, saving each frame of the image in the previous video frames of the video stream to obtain the time-series image data set.

[0007] S2, calculating the average motion energy of the previous video frames of the video stream according to the time-series image data set, specifically comprising the steps of:

[0008] S201, calculating a motion energy list of previous video frames of the video stream from the extracted time-series image data set.

[0009] S202: Calculate the average motion energy of the previous video frames of the video stream from the motion energy list of the previous video frames.

[0010] S3, calculating the dynamic factor of the previous video frame of the video stream according to the calculated average motion energy, specifically comprising the steps of:

[0011] S301, calculating the average motion energy to obtain a dynamic factor. Specifically, the calculation formula of the dynamic factor is:

[0012]

[0013] Where Dfact is the dynamic factor of the time series image dataset, IN() is the natural logarithm operation, and Avera motion_en is the average kinetic energy, A and B are constant terms.

[0014] S302, performing a restriction operation on the obtained dynamic factor to obtain the dynamic factor after the restriction operation. Specifically, the calculation formula of the restriction operation is:

[0015] Dfact=Max(Dfact,0)

[0016] Where Dfact is the dynamic factor of the time series image dataset, and Max() is the maximum value operation.

[0017] S303: Using the calculated dynamic factor of the time-series image data set as the dynamic factor of the previous video frame of the video stream.

[0018] The present invention proposes to compare the dynamic factor calculated from the front video frame of the video stream with the motion energy calculated from the rear video frame of the video stream, and then judge whether there is a freeze in the rear video frame of the video stream. The dynamic factor is generated with the video stream, and no manual setting is required. Moreover, there is a mapping between the dynamic factor and the frame rate of the video stream, and this mapping ensures a one-to-one correspondence between the dynamic factor and the video frame rate, so that the method proposed by the present invention can not only improve the accuracy of video freeze detection under the premise of ensuring generalization, but also realize adaptive video freeze detection without manual intervention.

[0019] S4, judging whether there is a freeze in the subsequent video frames of the video stream according to the dynamic factor of the previous video frames of the video stream.

[0020] Specifically, the latter video frames of the video stream refer to the remaining frame rates except the former x frames.

[0021] S401, extract two frames of images from the following video frames of the video stream.

[0022] S402, calculating the motion energy of the two frames of images from the two frames of images.

[0023] S403, establishing a video freeze list, judging whether the two frames of images are freezed based on the calculated motion energy of the two frames of images, and if freezes are present, adding the current judgment result to the freeze list; otherwise, ignoring the calculation result.

[0024] The specific calculation formula for determining whether these two frames of images are stuck is:

[0025]

[0026] In the formula, Current motion_en is the motion energy of the two frames in the following video frame of the video stream, Dfact is the dynamic factor of the time series image dataset, and M_DROP is the freeze threshold coefficient.

[0027] S404, otherwise, repeat the above steps until all subsequent video frames in the video stream are traversed.

[0028] In view of the shortcomings of conventional technical means that not only have requirements for computing hardware equipment but also the calculation time is difficult to achieve real-time, the present invention proposes to compare the dynamic factor calculated according to the front video frame of the video stream with the motion energy calculated according to the rear video frame of the video stream, and then judge whether there is a freeze in the rear video frame of the video stream. Compared with deep neural networks or other machine learning methods, the method proposed by the present invention not only makes the entire detection process simpler, but also greatly reduces the amount of calculation. This makes the method proposed by the present invention not only save computing resources, but also achieves a faster detection speed while ensuring detection accuracy.

[0029] According to the second aspect of the present invention, the present invention also provides a computer device, which includes a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement any of the above-mentioned methods for detecting video freeze based on frame differential momentum.

[0030] According to the third aspect of the present invention, the present invention also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is loaded and executed by a processor to implement any of the above-mentioned methods for detecting video freeze based on frame differential momentum. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose and features of the present invention more obvious and easy to understand, the present technical solution is described in detail below through embodiments and in conjunction with the accompanying drawings.

[0033] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0034] Embodiment 1:

[0035] like Figure 1 As shown, the embodiment of the present invention provides a method for detecting video freeze based on frame difference momentum, and the specific implementation process is as follows:

[0036] S1, obtaining a time-series image data set of the video stream from the previous video frames of the video stream, specifically, saving each frame of the image in the previous video frames of the video stream to obtain the time-series image data set.

[0037] Specifically, the range of the previous video frame can be determined according to actual needs. Optionally, the first 100 frames of the video stream are used as the previous video frame.

[0038] In order to ensure the subsequent frame difference calculation, in this step, it is also necessary to perform size detection on each frame image to prevent images of different sizes from appearing.

[0039] Preferably, when performing frame rate extraction, it is necessary to ensure that the intervals between every two frame rate images are the same.

[0040] Preferably, before obtaining a time-series image data set of a video stream from the video stream, videos with freezes or screen distortion may be eliminated manually or by a designed algorithm logic.

[0041] Preferably, after obtaining the time-series image data set of the video stream from the video stream, the images in the time-series image data set may be preprocessed to reduce the amount of subsequent calculations, such as adjusting the image size.

[0042] S2, calculating the average motion energy of the previous video frames of the video stream according to the time-series image data set, specifically comprising the steps of:

[0043] S201, calculating the motion energy list of the previous video frames of the video stream from the extracted time-series image data set, the specific process is as follows:

[0044] A1, calculates the inter-frame difference between two frames of images in the time series image dataset.

[0045] Specifically, the calculation formula for the frame difference is:

[0046] ΔIm=|Im1-Im2|

[0047] Where Im1 and Im2 are two frames of images, and ΔIm is the inter-frame difference between the two frames of images.

[0048] In the specific implementation process, the frame difference is to directly calculate the pixel value difference between the two frames of images. When (x, y) is set as the coordinates of the pixels in the two frames of images, the pixel value difference between the two frames of images is calculated by traversing and calculating the absolute value of the difference between each pixel value (x, y) in the two frames of images.

[0049] A2, binarizing the calculated inter-frame difference to obtain a binarized image of the inter-frame difference.

[0050] A3, the number of moving pixels between two frames of images is calculated from the binary image.

[0051] The calculation formula for the number of moving pixels is:

[0052] Motion_piels=ΔIm / 255

[0053] Wherein, Motion_pixels is the number of motion pixels between two frames of images, and ΔIm is the inter-frame difference between two frames of images.

[0054] A4, the motion energy between two frames of images is calculated by the number of moving pixels.

[0055] The calculation formula of kinetic energy is:

[0056] motion_en=Motion_pixels 2

[0057] Where motion_en is the motion energy between two frames, and Motion_pixels is the number of motion pixels between two frames.

[0058] A5, repeat A1-A until the motion energy between all two frames of images in the time series image data set is obtained, and a motion energy list is established based on the obtained motion energy between all two frames of images.

[0059] S202, calculating the average motion energy of the previous video frames of the video stream from the motion energy list of the previous video frames, the specific process is as follows:

[0060] B1, sorting the kinetic energy in the kinetic energy list according to the size of the kinetic energy.

[0061] B2, perform interval operation on the sorted motion energy list to obtain a motion energy sequence.

[0062] The calculation formula for specific interval operation is:

[0063]

[0064] In the formula, List motion_en is the motion energy sequence, the lower and upper limits of the Min and Max motion energy sequences, motion_en i is the i-th motion energy in the motion energy sequence.

[0065] Optionally, the Min value and the Max value may be set based on experience.

[0066] In an optional implementation, 5% of the motion energy values ​​in the motion energy sequence may be set as the Min value, and 95% of the motion energy values ​​in the motion energy sequence may be set as the Max value. The specific settings of the Min value and the Max value are not limited in the present invention.

[0067] B3, calculate the mean of the motion energy sequence to obtain the average motion energy.

[0068] The calculation formula for average kinetic energy is:

[0069]

[0070] In the formula, Avera motion_en is the average kinetic energy, List motion_en is the motion energy sequence, Num_motion_energy is the number of motion energies in the motion energy sequence, motion_en i is the i-th motion energy in the motion energy sequence.

[0071] S3, calculating the dynamic factor of the previous video frame of the video stream according to the calculated average motion energy, specifically comprising the steps of:

[0072] S301, calculating the average motion energy to obtain a dynamic factor. Specifically, the calculation formula of the dynamic factor is:

[0073]

[0074] Where Dfact is the dynamic factor of the time series image dataset, IN() is the natural logarithm operation, and Avera motion_en is the average kinetic energy, A and B are constant terms.

[0075] During the specific implementation process, A and B can be set according to the actual situation of the video stream.

[0076] As an example but not a limitation, in this embodiment, the value range of A and B is (-1, 1), and the present invention does not limit their specific values.

[0077] S302, performing a restriction operation on the obtained dynamic factor to obtain the dynamic factor after the restriction operation. Specifically, the calculation formula of the restriction operation is:

[0078] Dfact=Max(Dfact,0)

[0079] Where Dfact is the dynamic factor of the time series image dataset, and Max() is the maximum value operation.

[0080] S303: Using the calculated dynamic factor of the time-series image data set as the dynamic factor of the previous video frame of the video stream.

[0081] The present invention proposes to compare the dynamic factor calculated from the front video frame of the video stream with the motion energy calculated from the rear video frame of the video stream, and then judge whether there is a freeze in the rear video frame of the video stream. The dynamic factor is generated with the video stream, and no manual setting is required. Moreover, there is a mapping between the dynamic factor and the frame rate of the video stream, and this mapping ensures a one-to-one correspondence between the dynamic factor and the video frame rate, so that the method proposed by the present invention can not only improve the accuracy of video freeze detection under the premise of ensuring generalization, but also realize adaptive video freeze detection without manual intervention.

[0082] S4, judging whether there is a freeze in the subsequent video frames of the video stream according to the dynamic factor of the previous video frames of the video stream, specifically comprising the steps of:

[0083] S401, extract two frames of images from the following video frames of the video stream.

[0084] Specifically, the latter video frames of the video stream refer to the remaining frame rates except the former x frames.

[0085] In this embodiment, the subsequent video frames of the video stream refer to the remaining frame rates except the first 100 frames.

[0086] S402, calculating the motion energy of the two frames of images from the two frames of images, the specific process is as follows:

[0087] C1, calculate the inter-frame difference between the two frames of images.

[0088] C2, binarization processing is performed on the calculated inter-frame difference to obtain a binarized image of the inter-frame difference.

[0089] C3, the number of moving pixels between the two frames is calculated from the binary image.

[0090] C4, the motion energy between two frames of images is calculated by the number of moving pixels.

[0091] S403, establishing a video freeze list, judging whether the two frames of images are freezed based on the calculated motion energy of the two frames of images, and if freezes are present, adding the current judgment result to the freeze list; otherwise, ignoring the calculation result.

[0092] The specific calculation formula for determining whether these two frames of images are stuck is:

[0093]

[0094] In the formula, Current motion_en is the motion energy of the two frames in the following video frame of the video stream, Dfact is the dynamic factor of the time series image dataset, and M_DROP is the freeze threshold coefficient.

[0095] The jamming threshold is obtained by Dfact*M_DROP.

[0096] In a specific implementation process, the M_DROP freeze threshold coefficient can be set according to actual conditions, and the present invention does not limit this.

[0097] During the specific implementation process, the freeze threshold can be set according to the actual situation of the video stream.

[0098] Whether the two frames of images are stuck is determined by comparing the dynamic factor of the previous video frame of the video stream and the motion energy of the two frames of images.

[0099] S404, otherwise, repeat the above steps until all subsequent video frames in the video stream are traversed.

[0100] In view of the shortcomings of conventional technical means that not only have requirements for computing hardware equipment but also the calculation time is difficult to achieve real-time, the present invention proposes to compare the dynamic factor calculated according to the front video frame of the video stream with the motion energy calculated according to the rear video frame of the video stream, and then judge whether there is a freeze in the rear video frame of the video stream. Compared with deep neural networks or other machine learning methods, the method proposed by the present invention not only makes the entire detection process simpler, but also greatly reduces the amount of calculation. This makes the method proposed by the present invention not only save computing resources, but also achieves a faster detection speed while ensuring detection accuracy.

[0101] Example 2

[0102] The embodiment of the present invention further provides a computer device, which can implement the steps in any of the embodiments of the method for detecting video freeze based on frame difference momentum provided in the embodiment of the present invention. Therefore, the embodiment of the present invention can achieve the beneficial effects of the method for detecting video freeze based on frame difference momentum provided in the embodiment of the present invention, which is detailed in the previous embodiment and will not be repeated here.

[0103] Example 3

[0104] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention also provides a storage medium, which stores computer instructions, and the computer instructions can be loaded by a processor to execute the steps of any embodiment of the method for detecting video freeze based on frame differential momentum provided in an embodiment of the present invention.

[0105] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. Since the instructions stored in the storage medium can execute the steps in any of the embodiments of the method for detecting video freeze based on frame difference momentum provided in the embodiments of the present invention, the embodiments of the present invention can achieve the beneficial effects that can be achieved by any of the methods for detecting video freeze based on frame difference momentum provided in the embodiments of the present invention. For details, please refer to the previous embodiments, which will not be repeated here.

[0106] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0108] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for detecting video freeze based on frame difference momentum, characterized in that: Includes steps: S1, obtain the time-series image data set of the video stream from the previous video frame of the video stream; S2, calculates the average motion energy of the previous video frames of the video stream based on the temporal image dataset; S3, calculating the dynamic factor of the previous video frame of the video stream according to the calculated average motion energy; S4, judging whether there is a freeze in the subsequent video frames of the video stream according to the dynamic factor of the previous video frames of the video stream.

2. The method for detecting video freeze based on frame difference momentum according to claim 1, characterized in that: The step of obtaining the time-series image data set of the video stream from the previous video frames of the video stream specifically involves saving each frame of the image in the previous video frames of the video stream, thereby obtaining the time-series image data set.

3. The method for detecting video freeze based on frame difference momentum according to claim 1, characterized in that: S2 includes the steps: S201, calculating a motion energy list of previous video frames of the video stream from the extracted time-series image data set; S202: Calculate the average motion energy of the previous video frames of the video stream from the motion energy list of the previous video frames.

4. The method for detecting video freeze based on frame difference momentum according to claim 3, characterized in that: S201 includes the steps of: A1, calculate the inter-frame difference between two adjacent frames in the time series image data set. Specifically, The calculation formula for the frame difference is: ΔIm=|Im1-Im2| Where Im1 and Im2 are two frames of images, and ΔIm is the inter-frame difference between the two frames of images; A2, performing binarization processing on the calculated inter-frame difference to obtain a binarized image of the inter-frame difference; A3, calculate the number of moving pixels between two frames of images from the binary image, specifically, The calculation formula for the number of moving pixels is: Motion_pixels=ΔIm / 255 Where Motion_pixels is the number of motion pixels between two frames, and ΔIm is the inter-frame difference between two frames; A4, the motion energy between two frames of images is calculated by the number of moving pixels. Specifically, The calculation formula of kinetic energy is: motion_en=Motion_pixels 2 Where motion_en is the motion energy between two frames, and Motion_pixels is the number of motion pixels between two frames. A5, repeat A1-A4 until the motion energy between all two frames of images in the time series image data set is obtained, and a motion energy list is established based on the obtained motion energy between all two frames of images.

5. The method for detecting video freeze based on frame difference momentum according to claim 3, characterized in that: S202 includes the steps of: B1, sorting the kinetic energy in the kinetic energy list according to the size of the kinetic energy; B2, perform interval operation on the sorted motion energy list to obtain the motion energy sequence, The calculation formula for specific interval operation is: In the formula, List motion_en is the motion energy sequence, the lower and upper limits of the Min and Max motion energy sequences, motion_en i is the i-th motion energy in the motion energy sequence; B3, calculate the mean of the motion energy sequence to obtain the average motion energy. Specifically, The calculation formula for average kinetic energy is: In the formula, Avera motion_en is the average motion energy, Num_motion_energy is the number of motion energies in the motion energy sequence, motion_en i is the i-th motion energy in the motion energy sequence.

6. The method for detecting video freeze based on frame difference momentum according to claim 1, characterized in that: S3 includes the following steps: S301, calculate the average motion energy to obtain the dynamic factor, Specifically, the calculation formula of the dynamic factor is: Where Dfact is the dynamic factor of the time series image dataset, IN() is the natural logarithm operation, and Avera motion_en is the average kinetic energy, A and B are constant terms; S302, performing a restriction operation on the obtained dynamic factor to obtain the dynamic factor after the restriction operation, Specifically, the calculation formula for the restriction operation is: Dfact=Max(Dfact,0) Where Dfact is the dynamic factor of the time series image data set, and Max() is the maximum value operation; S303: Using the calculated dynamic factor of the time-series image data set as the dynamic factor of the previous video frame of the video stream.

7. The method for detecting video freeze based on frame difference momentum according to claim 1, characterized in that: S4 includes the steps: S401, extracting two frames of images from the following video frames of the video stream; S402, calculating the motion energy of the two frames of images from the two frames of images; S403, establishing a video freeze list, judging whether the two frames of images are freezed based on the calculated motion energy of the two frames of images, and if freezes are present, adding the current judgment result to the freeze list; otherwise, ignoring the current calculation result; S404, otherwise, repeat the above steps until all subsequent video frames in the video stream are traversed.

8. The method for detecting video freeze based on frame difference momentum according to claim 7, characterized in that: S402 includes the steps of: C1, calculate the inter-frame difference between the two frames of images; C2, binarizing the calculated inter-frame difference to obtain a binarized image of the inter-frame difference; C3, the number of moving pixels between the two frames is calculated from the binary image; C4, the motion energy between two frames of images is calculated by the number of moving pixels.

9. The method for detecting video freeze based on frame difference momentum according to claim 7, characterized in that: The calculation formula for determining whether the two frames of images are stuck in S403 is: In the formula, Current motion_en is the motion energy of the two frames in the following video frame of the video stream, Dfact is the dynamic factor of the time series image dataset, and M_DROP is the freeze threshold coefficient.

10. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the method for detecting video freeze based on frame differential momentum as described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is loaded and executed by a processor to implement the method for detecting video freeze based on frame differential momentum as described in any one of claims 1-9.

Images