Video lag detection method and device based on structural similarity index

Abstract

The invention discloses a video lag detection method and device based on a structural similarity index. The method belongs to the technical field of video lagging detection, and comprises the following steps: determining a lagging threshold value of a to-be-detected video according to a preset number of initial frame sequences of the to-be-detected video; performing frame-by-frame image extraction on the real-time video stream of the to-be-detected video to obtain a frame sequence of the current video stream of the to-be-detected video; respectively calculating similarity indexes between adjacent frames in the frame sequence, and obtaining a similarity score sequence of the frame sequence; calculating a similarity score difference value of adjacent frames in the similarity score sequence, and determining a similarity score difference value sequence; and comparing each similarity score difference value in the similarity score difference value sequence with a lagging threshold value, and determining a lagging condition of the current video stream according to a comparison result.

Description

Technical Field

[0001] The present invention relates to the technical field of video freeze detection, and more specifically, to a video freeze detection method and device based on a structural similarity index. Background Art

[0002] In current video playback, freeze detection mainly relies on methods such as network bandwidth and frame loss detection, but these methods cannot fully utilize the change information of video content. Existing technologies often tend to ignore the impact of subtle freezes on the user's viewing experience, thus affecting the user's viewing experience. Therefore, how to accurately detect freezes in video playback has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a video freeze detection method and device based on a structural similarity index.

[0004] According to one aspect of the present invention, a video freeze detection method based on a structural similarity index is provided, comprising:

[0005] Determining a freeze threshold of the video to be detected according to a preset number of initial frame sequences of the video to be detected;

[0006] Extract the real-time video stream of the video to be detected frame by frame to obtain the frame sequence of the current video stream of the video to be detected;

[0007] Calculate the similarity indexes between adjacent frames in the frame sequence respectively, and obtain a similarity score sequence of the frame sequence;

[0008] Calculating similarity score differences between adjacent frames in a similarity score sequence to determine a similarity score difference sequence;

[0009] Each similarity score difference in the similarity score difference sequence is compared with the jamming threshold, and the jamming condition of the current video stream is determined according to the comparison result.

[0010] Optionally, the preset number of initial frame sequences is a sequence of 100 frames of images starting from the initial frame of the video to be detected, and

[0011] Determining a freeze threshold of the video to be detected according to a preset number of initial frame sequences of the video to be detected, including:

[0012] Calculate the similarity indexes between adjacent frames in the initial frame sequence respectively to obtain an initial similarity score sequence;

[0013] Sorting the initial similarity score sequence, and removing abnormal data from the sorted similarity score sequence to obtain a valid initial similarity score sequence;

[0014] The variance of the effective initial similarity score sequence is calculated as the jamming threshold.

[0015] Optionally, respectively calculating similarity indexes between adjacent frames in the frame sequence to obtain a similarity score sequence of the frame sequence includes:

[0016] Compressing each frame image in the frame sequence to obtain a compressed frame sequence;

[0017] The similarity indexes between adjacent frames in the compressed frame sequence are calculated respectively to obtain a similarity score sequence.

[0018] Optionally, respectively calculating similarity indexes between adjacent frames in the frame sequence to obtain a similarity score sequence of the frame sequence includes:

[0019] Calculating brightness comparison indices between adjacent frames in the frame sequence respectively, and obtaining a brightness comparison result sequence of the frame sequence;

[0020] Calculating contrast comparison indices between adjacent frames in the frame sequence respectively, and obtaining a contrast comparison result sequence of the frame sequence;

[0021] Calculate the structural similarity indexes between adjacent frames in the frame sequence respectively, and obtain a structural comparison result sequence of the frame sequence;

[0022] A similarity score sequence of the frame sequence is obtained according to the brightness comparison result sequence, the contrast comparison result sequence and the structure comparison result sequence.

[0023] Optionally, respectively calculating brightness comparison indices between adjacent frames in the frame sequence to obtain a brightness comparison result sequence of the frame sequence includes:

[0024] Calculate the average brightness of each frame image in the frame sequence respectively to obtain an average brightness sequence of the frame sequence;

[0025] The brightness comparison index between adjacent frames is calculated according to the average brightness sequence to obtain a brightness comparison result sequence, where the calculation expression of the brightness comparison index is:

[0026]

[0027] In the formula, μ X and μ Y are the average brightness of two adjacent frames, C 1 Is a constant.

[0028] Optionally, respectively calculating contrast comparison indices between adjacent frames in the frame sequence to obtain a contrast comparison result sequence of the frame sequence includes:

[0029] Calculate the standard deviation of each frame image in the frame sequence respectively to obtain the contrast sequence of the frame sequence;

[0030] The contrast comparison index between adjacent frames is calculated according to the contrast sequence to obtain a contrast comparison result sequence, where the calculation expression of the contrast comparison index is:

[0031]

[0032] In the formula, σ X and σ Y are the standard deviations of two adjacent frames, C 2 Is a constant.

[0033] Optionally, respectively calculating structural similarity indices between adjacent frames in the frame sequence to obtain a structural comparison result sequence of the frame sequence includes:

[0034] Calculate the covariance of each frame image in the frame sequence respectively to obtain the structure sequence of the frame sequence;

[0035] The structural similarity index between adjacent frames is calculated according to the structural sequence to obtain a structural comparison result sequence, where the calculation expression of the structural similarity index is:

[0036]

[0037] in,

[0038]

[0039] Where, X i is the i-th pixel in the frame image X, Y i is the i-th pixel in the frame image Y, N is the number of pixels in the frame image, μ X and μ Y are the average brightness of two adjacent frames, σ X and σ Y are the standard deviations of two adjacent frames, C 3 Is a constant.

[0040] Optionally, each similarity score difference in the similarity score difference sequence is compared with a freeze threshold, and the freeze condition of the current video stream is determined according to the comparison result, including:

[0041] When a similarity score difference in the similarity score difference sequence is greater than a freeze threshold, it is determined that the current video stream has freeze; otherwise, it is determined that the video stream has freeze.

[0042] Optionally, the method further includes: when the number of non-stuttering frames of the video stream to be detected reaches a preset number, updating the stuttering threshold according to the preset number of non-stuttering frames.

[0043] According to another aspect of the present invention, a video freeze detection device based on a structural similarity index is provided, comprising:

[0044] A determination module, used to determine a freeze threshold of the video to be detected based on a preset number of initial frame sequences of the video to be detected;

[0045] An extraction module is used to extract frame-by-frame images of the real-time video stream of the video to be detected, and obtain a frame sequence of the current video stream of the video to be detected;

[0046] A first calculation module is used to respectively calculate similarity indexes between adjacent frames in a frame sequence to obtain a similarity score sequence of the frame sequence;

[0047] A second calculation module is used to calculate the similarity score difference between adjacent frames in the similarity score sequence to determine a similarity score difference sequence;

[0048] The comparison module is used to compare each similarity score difference in the similarity score difference sequence with the jamming threshold, and determine the jamming situation of the current video stream according to the comparison result.

[0049] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method described in any one of the above aspects of the present invention.

[0050] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of the above aspects of the present invention.

[0051] Therefore, the present invention provides a video freeze detection method based on structural similarity index, which sets a freeze threshold through an initial frame sequence, analyzes changes in the similarity score sequence between video frames, and detects the freeze of the video stream by comparing the similarity score difference between adjacent frames in the similarity score sequence with the freeze threshold, thereby avoiding the problem of one-sided evaluation errors of a single freeze threshold, thereby achieving accurate and real-time freeze detection, and effectively improving the user viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:

[0053] Figure 1 is a flow chart of a video freeze detection method based on a structural similarity index provided by an exemplary embodiment of the present invention;

[0054] Figure 2is a structural schematic diagram of a video freeze detection device based on a structural similarity index provided by an exemplary embodiment of the present invention;

[0055] Figure 3 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0056] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described here.

[0057] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0058] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0059] It should also be understood that, in the embodiments of the present invention, “plurality” may refer to two or more than two, and “at least one” may refer to one, two or more than two.

[0060] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0061] In addition, the term "and / or" in the present invention is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.

[0062] It should also be understood that the description of the various embodiments of the present invention focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0063] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0064] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0065] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0066] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0067] Embodiments of the present invention can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0068] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0069] Exemplary Methods

[0070] Figure 1 FIG. 1 is a flow chart of a method for detecting video freeze based on a structural similarity index provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the video freeze detection method 100 based on the structural similarity index includes the following steps:

[0071] Step 101, determining a freeze threshold of the video to be detected according to a preset number of initial frame sequences of the video to be detected;

[0072] Step 102, extracting images frame by frame from the real-time video stream of the video to be detected, and obtaining a frame sequence of the current video stream of the video to be detected;

[0073] Step 103, respectively calculating similarity indexes between adjacent frames in the frame sequence to obtain a similarity score sequence of the frame sequence;

[0074] Step 104, calculating the similarity score differences between adjacent frames in the similarity score sequence to determine a similarity score difference sequence;

[0075] Step 105 , compare each similarity score difference in the similarity score difference sequence with the freeze threshold, and determine the freeze condition of the current video stream according to the comparison result.

[0076] Specifically, the present invention proposes a video freeze detection method based on SSIM calculation. By analyzing the SSIM score changes of adjacent frames of the video and combining statistical indicators such as variance and standard deviation, the freeze problem in video playback can be accurately determined. The specific process is as follows:

[0077] In step 101, for the beginning of the video, the video images of the initial 100 frames of the current video are collected to obtain a time-series image data set; the inter-frame similarity of adjacent images in the time-series data set is calculated to obtain an inter-frame similarity sequence table; the inter-frame similarity sequence table is sorted by value, and in order to avoid false detection caused by lens switching, the largest 5% and the smallest 5% of the data are removed; the variance of the remaining data in the sequence is taken and stored as a threshold.

[0078] It should be pointed out that in actual application, different numbers of frames of initial video images can be collected as needed, and different data elimination ratios can be set as needed.

[0079] In step 102, image extraction is performed frame by frame on the real-time video stream of the video to be detected to obtain a frame sequence of the current video stream of the video to be detected.

[0080] In step 103, a structural similarity index SSIM is calculated for adjacent frames in the frame sequence to obtain a similarity score sequence. The scores in the similarity score sequence reflect the similarity of image quality and motion trend between adjacent frames.

[0081] In step 104, based on the similarity score ssim value sequence, adjacent frames take their current frame ssim values ​​and subtract them from the ssim values ​​obtained in the previous frame to obtain a similarity score difference value sequence.

[0082] In step 105, the similarity score difference in the similarity score difference sequence is compared with the jamming threshold. When the similarity score difference is greater than the jamming threshold, jamming occurs. Otherwise, it can be determined that the monitored video stream is normal. By setting the jamming threshold through the initial frame sequence, different degrees of jamming can be accurately detected.

[0083] Furthermore, every time 100 non-stuttering frames are stored, the operation in step 1 is repeated to obtain a new variance and a weighted average is performed with the current threshold as a new stuttering threshold for detection.

[0084] Therefore, the system can further optimize the video stream according to the jam detection results, such as providing jam prompts, adjusting video playback parameters, etc.

[0085] Furthermore, in step 103, the structural similarity index SSIM is calculated for adjacent frames of the frame sequence to obtain a similarity score sequence. In order to speed up the SSIM detection process, the frame images in the frame sequence can be compressed before the similarity index SSIM calculation, or the similarity index SSIM calculation can be performed directly on the frame sequence.

[0086] Furthermore, in step 103, the structural similarity index SSIM mainly measures the similarity between two frame images from three aspects, namely, luminance, contrast and structure.

[0087] Brightness comparison:

[0088] First, the average brightness of the frame image is calculated. For the image frames X and Y, the average brightness is μ X and μ Y The brightness comparison formula is:

[0089]

[0090] In the formula, C 1 is a constant to avoid the denominator being zero.

[0091] Contrast comparison:

[0092] Calculate the standard deviation of the image to represent the contrast. The standard deviations of frame images X and Y are σ X and σ Y .

[0093] The contrast comparison formula is:

[0094]

[0095] Among them, C 2 is also a constant.

[0096] Structural comparison:

[0097] Calculate the covariance of the image to represent the structural similarity. The covariance of the frame images X and Y is σ XY :

[0098]

[0099] The structural comparison formula is:

[0100]

[0101] In the formula, x i is the i-th pixel in the frame image, C 3 Is a constant.

[0102] Finally, the calculation formula of the structural similarity index SSIM is:

[0103] SSIM(X,Y)=[l(X,Y)] α ·[c(X,Y)] β ·[s(X,Y)] γ

[0104] Among them, α, β, and γ are weight coefficients, and usually α=β=γ=1.

[0105] The SSIM proposed in the present invention does not consider only a single factor like the traditional image quality evaluation method, but considers three aspects: brightness, contrast and structure at the same time. This enables it to measure the similarity between images more comprehensively, which is more in line with the perception of image quality by the human visual system. And by updating and setting the jamming threshold in the playback scene, the jamming of the video stream can be judged in real time, so as to accurately detect the jamming of the video stream according to the playback scene of the video.

[0106] The algorithm proposed by the present invention is embedded in online video, streaming live broadcast and other scenarios to achieve real-time monitoring of video freezes. The method is applicable to various playback devices and application scenarios, and has good versatility and real-time performance.

[0107] Therefore, the present invention provides a video freeze detection method based on structural similarity index, which sets a freeze threshold through an initial frame sequence, analyzes changes in the similarity score sequence between video frames, and detects the freeze of the video stream by comparing the similarity score difference between adjacent frames in the similarity score sequence with the freeze threshold, thereby avoiding the problem of one-sided evaluation errors of a single freeze threshold, thereby achieving accurate and real-time freeze detection, and effectively improving the user viewing experience.

[0108] Exemplary Devices

[0109] Figure 2 FIG. 1 is a schematic diagram of a video freeze detection device based on a structural similarity index provided by an exemplary embodiment of the present invention. Figure 2 As shown, the device 200 includes:

[0110] A determination module 210, configured to determine a freeze threshold of the video to be detected based on a preset number of initial frame sequences of the video to be detected;

[0111] An extraction module 220 is used to extract frame-by-frame images of the real-time video stream of the video to be detected, and obtain a frame sequence of the current video stream of the video to be detected;

[0112] A first calculation module 230 is used to respectively calculate similarity indexes between adjacent frames in a frame sequence to obtain a similarity score sequence of the frame sequence;

[0113] A second calculation module 240 is used to calculate the similarity score differences between adjacent frames in the similarity score sequence to determine a similarity score difference sequence;

[0114] The comparison module 250 is used to compare each similarity score difference in the similarity score difference sequence with the freeze threshold, and determine the freeze condition of the current video stream according to the comparison result.

[0115] Optionally, the preset number of initial frame sequences is a sequence of 100 frames of images starting from the initial frame of the video to be detected, and

[0116] The determination module 210 includes:

[0117] A first calculation module is used to respectively calculate similarity indexes between adjacent frames in the initial frame sequence to obtain an initial similarity score sequence;

[0118] A sorting submodule is used to sort the initial similarity score sequence and remove abnormal data from the sorted similarity score sequence to obtain a valid initial similarity score sequence;

[0119] The second calculation submodule is used to calculate the variance of the effective initial similarity score sequence as the jamming threshold.

[0120] Optionally, the first calculation module 230 includes:

[0121] A compression submodule, used for compressing each frame image in the frame sequence to obtain a compressed frame sequence;

[0122] The third calculation submodule is used to respectively calculate the similarity indexes between adjacent frames in the compressed frame sequence to obtain a similarity score sequence.

[0123] Optionally, the first calculation module 230 includes:

[0124] A fourth calculation submodule, used to respectively calculate brightness comparison indicators between adjacent frames in the frame sequence, and obtain a brightness comparison result sequence of the frame sequence;

[0125] A fifth calculation submodule, used to respectively calculate contrast comparison indicators between adjacent frames in the frame sequence, and obtain a contrast comparison result sequence of the frame sequence;

[0126] A sixth calculation submodule, used to respectively calculate the structural similarity index between adjacent frames in the frame sequence, and obtain a structural comparison result sequence of the frame sequence;

[0127] The acquisition submodule is used to acquire a similarity score sequence of the frame sequence according to the brightness comparison result sequence, the contrast comparison result sequence and the structure comparison result sequence.

[0128] Optionally, the third computing submodule includes:

[0129] The first calculation unit is used to calculate the average brightness of each frame image in the frame sequence, and obtain an average brightness sequence of the frame sequence;

[0130] The first acquisition unit is used to calculate the brightness comparison index between adjacent frames according to the average brightness sequence, and obtain a brightness comparison result sequence, wherein the calculation expression of the brightness comparison index is:

[0131]

[0132] In the formula, μ X and μ Y are the average brightness of two adjacent frames, C 1 Is a constant.

[0133] Optionally, the fourth computing submodule includes:

[0134] A second calculation unit is used to calculate the standard deviation of each frame image in the frame sequence respectively to obtain a contrast sequence of the frame sequence;

[0135] The second acquisition unit is used to calculate the contrast comparison index between adjacent frames according to the contrast sequence, and obtain a contrast comparison result sequence, wherein the calculation expression of the contrast comparison index is:

[0136]

[0137] In the formula, σ X and σ Y are the standard deviations of two adjacent frames, C 2 Is a constant.

[0138] Optionally, the fifth computing submodule includes:

[0139] A third calculation unit is used to calculate the covariance of each frame image in the frame sequence respectively to obtain a structure sequence of the frame sequence;

[0140] The third acquisition unit is used to calculate the structural similarity index between adjacent frames according to the structural sequence, and obtain the structural comparison result sequence, wherein the calculation expression of the structural similarity index is:

[0141]

[0142] in,

[0143]

[0144] Where, X i is the i-th pixel in the frame image X, Y i is the i-th pixel in the frame image Y, N is the number of pixels in the frame image, μ X and μ Y are the average brightness of two adjacent frames, σ X and σ Y are the standard deviations of two adjacent frames, C 3 Is a constant.

[0145] Optionally, the comparison module 250 includes:

[0146] The determination submodule is used to determine that the current video stream has a stuck condition when a similarity score difference in the similarity score difference sequence is greater than a stuck threshold, otherwise, determine that the video stream has a stuck condition.

[0147] Optionally, the device 200 further includes: when the number of non-stuttering frames of the video stream to be detected reaches a preset number, updating the stuttering threshold according to the preset number of non-stuttering frames.

[0148] Exemplary Electronic Devices

[0149] Figure 3 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. Figure 3 As shown, the electronic device 30 includes one or more processors 31 and a memory 32 .

[0150] The processor 31 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0151] The memory 32 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 31 may run the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above and / or other desired functions. In one example, the electronic device may also include: an input device 33 and an output device 34, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0152] In addition, the input device 33 may also include, for example, a keyboard, a mouse, and the like.

[0153] The output device 34 can output various information to the outside. The output device 34 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto.

[0154] Of course, to simplify, Figure 3 Only some of the components related to the present invention in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application conditions.

[0155] Exemplary computer program products and computer-readable storage media

[0156] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above-mentioned "Exemplary Method" section of this specification.

[0157] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the embodiments of the present invention, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0158] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0159] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, system or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0160] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details disclosed above are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0161] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0162] The block diagrams of the devices, systems, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, systems, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0163] The method and system of the present invention may be implemented in many ways. For example, the method and system of the present invention may be implemented by software, hardware, firmware or any combination of software, hardware, firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present invention are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present invention may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers a recording medium storing a program for executing the method according to the present invention.

[0164] It should also be noted that in the system, device and method of the present invention, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. The above description of the disclosed aspects is provided to enable any technician in the field to make or use the present invention. Various modifications to these aspects are very obvious to those skilled in the art, and the general principles defined here can be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown here, but in accordance with the widest range consistent with the principles and novel features disclosed here.

[0165] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A video freeze detection method based on structural similarity index, characterized in that: include: Determining a freeze threshold of the video to be detected according to a preset number of initial frame sequences of the video to be detected; Extracting images frame by frame from the real-time video stream of the video to be detected, and obtaining a frame sequence of the current video stream of the video to be detected; Respectively calculating similarity indexes between adjacent frames in the frame sequence to obtain a similarity score sequence of the frame sequence; Calculating similarity score differences between adjacent frames in the similarity score sequence to determine a similarity score difference sequence; Each similarity score difference in the similarity score difference sequence is compared with the freeze threshold, and the freeze condition of the current video stream is determined according to the comparison result.

2. The method according to claim 1, characterized in that The preset number of initial frame sequences is a sequence of 100 frames of images starting from the initial frame of the video to be detected, and Determining a freeze threshold of the video to be detected according to a preset number of initial frame sequences of the video to be detected includes: Respectively calculating similarity indexes between adjacent frames in the initial frame sequence to obtain an initial similarity score sequence; Sorting the initial similarity score sequence, and removing abnormal data from the sorted similarity score sequence to obtain a valid initial similarity score sequence; The variance of the effective initial similarity score sequence is calculated as the jamming threshold.

3. The method according to claim 1, characterized in that Respectively calculating similarity indexes between adjacent frames in the frame sequence to obtain a similarity score sequence of the frame sequence includes: Compressing each frame image in the frame sequence to obtain a compressed frame sequence; The similarity indexes between adjacent frames in the compressed frame sequence are respectively calculated to obtain the similarity score sequence.

4. The method according to claim 1, characterized in that: Respectively calculating similarity indexes between adjacent frames in the frame sequence to obtain a similarity score sequence of the frame sequence includes: Respectively calculating brightness comparison indices between adjacent frames in the frame sequence to obtain a brightness comparison result sequence of the frame sequence; respectively calculating contrast comparison indices between adjacent frames in the frame sequence to obtain a contrast comparison result sequence of the frame sequence; respectively calculating structural similarity indices between adjacent frames in the frame sequence to obtain a structural comparison result sequence of the frame sequence; The similarity score sequence of the frame sequence is acquired according to the brightness comparison result sequence, the contrast comparison result sequence, and the structure comparison result sequence.

5. The method according to claim 4, characterized in that Respectively calculating brightness comparison indices between adjacent frames in the frame sequence to obtain a brightness comparison result sequence of the frame sequence, including: Calculating the average brightness of each frame image in the frame sequence respectively to obtain an average brightness sequence of the frame sequence; The brightness comparison index between adjacent frames is calculated according to the average brightness sequence to obtain the brightness comparison result sequence, wherein the calculation expression of the brightness comparison index is: In the formula, μ X and μ Y are the average brightness of two adjacent frames, and C1 is a constant.

6. The method according to claim 4, characterized in that Calculating contrast comparison indices between adjacent frames in the frame sequence respectively to obtain a contrast comparison result sequence of the frame sequence includes: Calculating the standard deviation of each frame image in the frame sequence respectively to obtain a contrast sequence of the frame sequence; The contrast comparison index between adjacent frames is calculated according to the contrast sequence to obtain the contrast comparison result sequence, wherein the calculation expression of the contrast comparison index is: In the formula, σ X and σ Y are the standard deviations of two adjacent frames, and C2 is a constant.

7. The method according to claim 4, characterized in that Calculating structural similarity indexes between adjacent frames in the frame sequence respectively to obtain a structural comparison result sequence of the frame sequence includes: Calculating the covariance of each frame image in the frame sequence respectively to obtain a structural sequence of the frame sequence; The structural similarity index between adjacent frames is calculated according to the structural sequence to obtain the structural comparison result sequence, wherein the calculation expression of the structural similarity index is: in, Where, X i is the i-th pixel in the frame image X, Y i is the i-th pixel in the frame image Y, N is the number of pixels in the frame image, μ X and μ Y are the average brightness of two adjacent frames, σ X and σ Y are the standard deviations of two adjacent frames, and C3 is a constant.

8. The method according to claim 1, characterized in that: Comparing each similarity score difference in the similarity score difference sequence with the freeze threshold, and determining the freeze condition of the current video stream according to the comparison result, including: When there is a similarity score difference in the similarity score difference sequence that is greater than the freeze threshold, it is determined that the current video stream has freeze; otherwise, it is determined that the video stream has freeze.

9. The method according to claim 1, characterized in that: Also includes: When the number of non-stuttering frames in the video stream to be detected reaches the preset number, the stuttering threshold is updated according to the preset number of non-stuttering frames.

10. A video freeze detection device based on structural similarity index, characterized in that: include: A determination module, used to determine a freeze threshold of the video to be detected based on a preset number of initial frame sequences of the video to be detected; An extraction module is used to extract frame-by-frame images from the real-time video stream of the video to be detected, and obtain a frame sequence of the current video stream of the video to be detected; A first calculation module, used to respectively calculate similarity indexes between adjacent frames in the frame sequence to obtain a similarity score sequence of the frame sequence; A second calculation module, used for calculating the similarity score difference between adjacent frames in the similarity score sequence to determine a similarity score difference sequence; The comparison module is used to compare each similarity score difference in the similarity score difference sequence with the freeze threshold, and determine the freeze condition of the current video stream according to the comparison result.

Images