A method and apparatus for detecting a quiet frame

Abstract

The application discloses a still frame detection method and device, comprising: obtaining an Nth image group G N ; decoding a key frame in the obtained image group G N ; determining the similarity between the key frame in the image group G N and the key frame of an (N-1)th image group G N‑1 ; and determining whether there is a still frame based on the similarity and whether there is a still frame starting moment, wherein the scheme is realized based on a skip frame detection strategy of the key frame, that is, the still frame detection is preferentially performed on the key frame in the video signal, and if no still frame is found, the detection of other frame types in the current image group is directly skipped; the implementation not only reduces the overhead of the decoder in processing the non-key frame, but also greatly reduces the frame data required for the still frame detection, so that the detection speed is greatly improved; and the efficient detection manner significantly reduces the occupation of the server resources, and reduces the hardware cost and energy consumption.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and more particularly to a still frame detection method and device. BACKGROUND

[0002] With the rapid development of video technology, the importance of video quality is increasingly prominent. However, in these real-time signal source scenarios, due to monitoring equipment or camera failure, line interruption or human operation error, etc., the video often appears still frame phenomenon, also known as picture frozen quadrant, which is manifested as the picture remaining static state for continuous multiple frames or even a long time. If the still frame is caused by video playing failure (such as picture long time static caused by lag), it will seriously affect the video quality, similar to the influence of bad frame.

[0003] The traditional still frame detection method often faces problems of large consumption of computing resources, slow response speed and high false detection rate when processing high frame rate, high resolution live streaming. Especially in the live scenes of video monitoring and television signals, with the continuous improvement of video clarity, resolution and live video source concurrency, higher requirements are put forward for the real-time, accuracy and efficiency of still frame detection. SUMMARY

[0004] Therefore, the present application provides the following technical solutions:

[0005] The first aspect of the present application provides a still frame detection method, comprising:

[0006] obtaining an Nth image group G N , wherein N is a positive integer;

[0007] decoding to obtain a key frame in the image group G N ;

[0008] determining the similarity between the key frame in the image group G N and the key frame of the (N-1)th image group G N-1 ;

[0009] determining whether there is a still frame based on the similarity and whether there is a still frame start time, the still frame start time being data set when the similarity between the key frame of the image group G M and the key frame of the image group G M-1 meets a set condition, and M is less than N.

[0010] In one possible implementation, the determination of whether there is a still frame based on the similarity and whether there is a still frame start time comprises:

[0011] if the similarity is greater than or equal to a noise threshold, further determining whether there is a still frame start time.

[0012] if the still frame start time exists, continuing the still frame detection process of the subsequent image group, and if the still frame start time does not exist, determining the still frame start time based on the image frame data of the buffered historical image group;

[0013] if the similarity is less than the noise threshold, further determining whether the still frame start time exists;

[0014] if the still frame start time exists, determining the still frame end time based on the image frame data of the image group G N-1 , and if the still frame start time does not exist, continuing the still frame detection process of the subsequent image group;

[0015] determining whether the still frame exists based on the still frame start time and the still frame end time.

[0016] In one possible implementation, the determining the similarity between the key frame in the image group G N and the key frame of the (N-1)th image group G N-1 includes:

[0017] determining the average absolute difference of all components in the image encoding format between the key frame in the image group G N and the key frame of the image group G N-1 .

[0018] In one possible implementation, during the still frame detection process, the obtained image groups are sequentially buffered as the video signal is received, and the maximum number of buffered image groups is three. When the buffered image groups are three, the buffered image groups include the image group G N currently processed, the image group G N-1 , and the image group G N-2 .

[0019] The determining the still frame start time based on the image frame data of the buffered historical image group includes:

[0020] if there are only two image groups currently buffered, determining the time corresponding to the key frame of the image group G N-1 as the still frame start time;

[0021] if there are three image groups currently buffered, extracting all frame data of the image group G N-2 for full decoding, and comparing all image frames obtained by decoding with the key frame of the image group G N , and determining the time corresponding to the image frame with a similarity greater than or equal to a noise threshold with the key frame of the image group G N as the still frame start time;

[0022] if there are three image groups currently buffered, and the image group G N-2all the image frames obtained by decoding are compared with the key frames of the image group G N , and the time corresponding to the image frame with a similarity greater than or equal to the noise threshold is determined as the still frame start time. N-1

[0023] In a possible implementation, the still frame end time is determined based on the image group G N-1 , and the determination includes:

[0024] all the frame data of the image group G N-1 are extracted for full decoding, and all the image frames obtained by decoding are compared with the key frames of the image group G N-1 , and the time corresponding to the image frame with a similarity greater than or equal to the noise threshold is determined as the still frame end time. N-1

[0025] In a possible implementation, the determination of whether there is a still frame based on the still frame start time and the still frame end time includes:

[0026] determining a time difference between the still frame start time and the still frame end time;

[0027] if the time difference is greater than a time threshold, it is determined that there is a still frame;

[0028] if the time difference is less than or equal to the time threshold, it is determined that there is no still frame, and the still frame start time is set to null.

[0029] In a possible implementation, the method further includes:

[0030] the noise threshold and the time threshold are preconfigured.

[0031] In a possible implementation, the method further includes:

[0032] if there is a still frame, the still frame data containing the still frame time is saved or uploaded to the cloud.

[0033] In a possible implementation, the still frame detection method is multiplexed with a first multimedia processing framework.

[0034] The second aspect of the present application provides a still frame detection device, which includes:

[0035] an image group obtaining module, configured to obtain an Nth image group G N , where N is a positive integer;

[0036] a decoding processing module, configured to decode key frames in the image group G N ;

[0037] ​​The similarity determination module is configured to determine a similarity between the key frame in the image group G N and a key frame in the N-1th image group G N-1 .

[0038] The still frame determination module is configured to determine whether a still frame exists based on the similarity and whether a still frame start time exists at present, the still frame start time being data set in a case where the similarity between the key frame in the image group G M and the key frame in the image group G M-1 meets a set condition, and the M is less than the N.

[0039] According to the technical solution described above, the embodiment of the present application discloses a still frame detection method and device, which comprises: obtaining an Nth image group G N ; decoding a key frame in the image group G N ; determining a similarity between the key frame in the image group G N and a key frame in an N-1th image group G N-1 ; and determining whether a still frame exists based on the similarity and whether a still frame start time exists at present, the still frame start time being data set in a case where the similarity between the key frame in the image group G M and the key frame in the image group G M-1 meets a set condition. This solution is based on a key frame skip frame detection strategy, that is, the still frame detection is performed on the key frame in the video signal first, and if no still frame is found, the detection of other frame types in the current image group is directly skipped. This implementation not only reduces the overhead of the decoder in processing non-key frames, but also greatly reduces the frame data amount required for still frame detection, so that the detection speed is greatly improved. Moreover, this efficient detection method significantly reduces the occupation of server resources, and reduces the hardware cost and energy consumption. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without any creative effort.

[0041] Figure 1 A flowchart of a still frame detection method disclosed by the embodiment of the present application;

[0042] Figure 2 A first flowchart of determining whether a still frame exists disclosed by the embodiment of the present application;

[0043] Figure 3 A second flowchart of determining whether a still frame exists disclosed by the embodiment of the present application;

[0044] Figure 4 Flow chart of tool implementation of the still frame detection scheme disclosed in the embodiments of the present application;

[0045] Figure 5 Overall implementation flowchart of the scheme disclosed in the embodiments of the present application.

[0046] Figure 6 Structure diagram of a still frame detection device disclosed in the embodiments of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0048] Figure 1 Flow chart of a still frame detection method disclosed in the embodiments of the present application. Referring to FIG. 1, Figure 1 the still frame detection method can include:

[0049] Step 101: obtaining an Nth image group G N wherein N is a positive integer.

[0050] During transmission, the video stream is continuously transmitted to the device requiring video output in the form of image groups, and the image groups are transmitted in time sequence. The image group, which can be abbreviated as GOP, is a group of continuous image frames, usually starting with an I frame, followed by P frames and B frames. The I frame is a key frame containing complete image information, while the P frame (predictively encoded through the previous I frame or P frame, only containing the difference part from the reference frame) and the B frame (bidirectionally predictively encoded through the previous and subsequent I frame or P frame, usually used to improve compression efficiency) are encoded through the difference from the previous or previous several I frames.

[0051] Step 102: decoding the key frame in the obtained image group G N .

[0052] In the traditional scheme, when performing still frame detection, all frame images in the image group are usually decoded, and then compared and determined frame by frame. Such implementation not only consumes a lot of resources, but also has very low efficiency. In the embodiments of the present application, the key frame in the image group G NAfterwards, only the key frames in the image group, namely the I frames, are decoded, and the comparison of the subsequent frame images is also the comparison of the key frames of different image groups. Only when certain conditions are met, will the image frames other than the key frames be decoded.

[0053] That is, the key frame is a frame type that can be completely decoded without relying on other frames. The key frame of each image group can be decoded first, and other types of frames are only decoded when needed. Thus, compared with the traditional scheme, the amount of data to be decoded is greatly reduced, thereby saving relevant resources and significantly improving the still frame detection efficiency.

[0054] Step 103: determining the similarity between the key frame in the image group G N and the key frame in the N-1th image group G N-1 .

[0055] The image group G N is the latest obtained image group. After obtaining the latest image group each time, the key frame of the image group is decoded, and then the similarity between the key frame of the current processing image group and the key frame of the previous image group is determined.

[0056] Step 104: determining whether there is a still frame based on the similarity and whether there is a still frame start time. The still frame start time is the data set when the similarity between the key frame of the image group G M and the key frame of the image group G M-1 meets the set condition, and M is less than N.

[0057] The conditions that need to be met for the determination of the still frame include two aspects, one is whether the front and rear images are the same (determined based on the similarity), and the other is the duration of the image remaining unchanged. Therefore, to determine whether there is a still frame state, it is necessary to determine whether there is a still frame based on the similarity and whether there is a still frame start time. The specific implementation will be described in detail in the embodiments below, and will not be described in detail here.

[0058] Among them, the still frame start time is a parameter that is set once after receiving a new image group each time, which indicates the starting time of the possible still frame state, and the specific determination method will be described in detail in the embodiments below.

[0059] The still frame detection method described in the embodiment is realized based on the skip frame detection strategy of key frames, that is, the still frame detection is preferentially performed on the key frames in the video signal, and if no still frame is found, the detection of other frame types in the current GOP is directly skipped. This implementation not only reduces the overhead of the decoder in processing non-key frames, but also greatly reduces the frame data required for still frame detection, so that the detection speed is greatly improved. Moreover, this efficient detection method significantly reduces the occupation of server resources, and reduces the hardware cost and energy consumption.

[0060] Figure 2 The first flowchart for determining whether a still frame exists is disclosed in the embodiments of the present application. Referring to FIG. 2, the determination of whether a still frame exists based on the similarity and the start time of the still frame can include the following steps. Figure 2

[0061] Step 201: If the similarity is greater than or equal to the noise threshold value and the start time of the still frame exists, the still frame detection process of the subsequent GOP is continued.

[0062] The similarity greater than or equal to the noise threshold value indicates that the key frame of the current GOP is the same as or similar to the key frame of the previous GOP, and the start time of the still frame has been determined. Therefore, the detection is continued until the end time of the still frame is detected.

[0063] Step 202: If the similarity is greater than or equal to the noise threshold value and the start time of the still frame does not exist, the start time of the still frame is determined based on the buffered image frame data of the historical GOP, and step 205 is entered.

[0064] In a specific implementation, during the still frame detection process, the image groups obtained in sequence are buffered as the video signal is received. The maximum number of buffered image groups is three. When the buffered image groups are three, the buffered image groups include the image group G N , the image group G N-1 , and the image group G N-2 .

[0065] The determination of the start time of the still frame based on the buffered image frame data of the historical GOP can include the following steps.

[0066] If only two image groups are buffered, the time corresponding to the key frame of the image group G N-1 is determined as the start time of the still frame. If only two image groups are buffered and the key frames of the two image groups are the same or similar, it is indicated that the state may be a still frame state from the beginning. Therefore, the time of the first frame of the first buffered image group (the image group G N-2 ), that is, the time of the key frame of the first image group, is recorded as the start time of the still frame.

[0067] If three image groups are buffered, the time corresponding to the key frame of the image group G​N-2 all frame data of the image group G N are compared with the key frame of the image group G N , and the time corresponding to the image frame with similarity greater than or equal to the noise threshold to the key frame of the image group G N-1 is determined as the still frame start time. If three image groups are buffered, and the key frame of the second buffered image group (image group G N ) and the key frame of the third buffered image group (image group G N-2 ) are the same or similar, it is indicated that the still frame start time is at least from the time of the key frame of the second image group, and can also be at the time of the non-key frame of the first image group, thus all frame data of the first image group, i.e., all frame data of the image group G N , are decoded, and the time corresponding to the image frame the same or highly similar to the key frame of the third image group, i.e., the key frame of the image group G N-2 , is determined as the still frame start time.

[0068] If three image groups are currently buffered, and the similarity of all decoded image frames of the image group G N to the key frame of the image group G N-1 is less than the noise threshold, the time corresponding to the key frame of the image group G N-2 is determined as the still frame start time. If three image groups are buffered, and the similarity of all image frames in the first image group G N to the key frame of the third image group G N-1 is low, the time corresponding to the key frame of the second image group G N-1 is directly determined as the still frame start time.

[0069] Step 203: If the similarity is less than the noise threshold, and the still frame start time exists, the still frame end time is determined based on the image group G N-1 , and step 205 is entered.

[0070] Specifically, the still frame end time determined based on the image group G N-1 may include: all frame data of the image group G N-1 are decoded, and all decoded image frames are compared with the key frame of the image group G N-1 , and the time corresponding to the image frame with similarity greater than or equal to the noise threshold to the key frame of the image group G N is determined as the still frame end time.

[0071] If the key frame of the third image group G N is the same as the key frame of the second image group G N-1the key frame of the second image group G N-1 does not satisfy the requirement, that is, the similarity is low, since the first frame of each image group is a key frame, it is indicated that the still frame ending moment is in the time period corresponding to the second image group G N-1 , and the image frame whose similarity to the key frame of the second image group G

[0072] Step 204: If the similarity is less than the noise threshold and there is no still frame starting moment, continue to perform still frame detection processing on the subsequent image group.

[0073] Step 205: Determine whether there is a still frame based on the still frame starting moment and the still frame ending moment.

[0074] Figure 3 A second flowchart for determining whether there is a still frame is disclosed in the embodiments of the present application, as shown in Figure 3 , the determination of whether there is a still frame based on the still frame starting moment and the still frame ending moment can include:

[0075] Step 301: Determine the time difference value from the still frame starting moment to the still frame ending moment.

[0076] Step 302: If the time difference value is greater than the time threshold, determine that there is a still frame.

[0077] Step 303: If the time difference value is less than or equal to the time threshold, determine that there is no still frame, and set the still frame starting moment to be empty.

[0078] For example, if the time threshold is 3 seconds, and the time difference value is 2.5 seconds, it is determined that there is no still frame; if the time difference value is 3.5 seconds, it is determined that there is a still frame.

[0079] It can be understood that if the time difference value from the still frame starting moment to the still frame ending moment is less than or equal to the time threshold, there may be a reasonable range of picture stillness in the video content, such as timed display of a certain thing to create a tense atmosphere, but this does not belong to the still frame case, and at this time the still frame starting moment needs to be set to be empty, and the process of determining whether there is a still frame case is determined again for the subsequent image group.

[0080] In the implementation of the still frame detection scheme described in the embodiments, a cache mechanism is introduced to save the data of the last three image groups processed. When the key frames of the adjacent two groups are detected as suspected still frame pictures, the cached image groups can be fully decoded according to the specific situation to determine the starting and ending moment frames of the still frame picture. This backtracking decoding mechanism effectively avoids errors caused by frame skipping detection, and ensures the accuracy of still frame detection.

[0081] In an implementation, the noise threshold and the time threshold can be pre-configured, so that the noise threshold and the time threshold can be directly called for corresponding judgment processing when performing the still frame detection.

[0082] If the still frame is detected, the still frame data containing the still frame time can be saved or uploaded to the cloud, so as to timely lock the video abnormal state and perform corresponding processing.

[0083] In the foregoing embodiment, the similarity between the key frame in the image group G N and the key frame of the N-1th image group G N-1 may include: determining the average absolute difference of all components of the image encoding format of the key frame in the image group G N and the key frame of the image group G N-1 .

[0084] The image encoding format of the key frame is not fixed, which can be RGB format or YUV format. Each encoding format contains multiple components. The average absolute difference of all components of the image encoding format of the key frame in the image group G N and the key frame of the image group G N-1 , that is, the similarity between the key frame in the image group G N and the key frame of the image group G N-1 . If the average absolute difference of all components is larger, the similarity between the two key frames is smaller. If the average absolute difference of all components is smaller, the similarity between the two key frames is larger.

[0085] The comparison result of the average absolute difference of all components and the noise threshold can be used to determine whether one of the conditions of the still frame state is met, that is, whether there is the same or similar image content in different image groups.

[0086] The still frame detection scheme described in the present application not only significantly improves the efficiency of still frame detection, but also ensures the accuracy of the detection result, providing an efficient and reliable still frame detection solution for high-concurrency and high-resolution live video signal sources. For enterprises that need to process large-scale video signals (such as video monitoring service providers, television stations, and relay centers), this scheme can significantly reduce operating costs, improve the overall throughput and response speed of the system, and can be easily integrated into existing video monitoring and live systems, suitable for different application scenarios.

[0087] In practical applications, the still frame detection method can reuse a first multimedia processing framework, such as an FFmpeg framework. Based on a freezedetect filter in the FFmpeg framework, video still frame detection can be implemented. The freezedetect filter compares the average absolute difference of all components of a key frame in two adjacent image groups after decoding with a preset noise threshold. When the average absolute difference of all components is less than the noise threshold (corresponding to a similarity greater than or equal to the noise threshold) and exceeds a preset still frame duration, the filter records a still frame message.

[0088] The application scheme can fully utilize the powerful audio and video processing capability of the FFmpeg framework to achieve efficient engineering toolization. The AVFormat library of the FFmpeg framework supports common public live transmission protocols and conversion encapsulation protocols on the market, including the RTSP protocol widely used in the video monitoring field, the UDP multicast protocol in the digital television field, the HTTP-HLS and RTMP protocols with high delay, and the low-delay transmission protocols such as WebRTC and SRT that have been open sourced in recent years. The application directly reuses the AVFormat library to realize transmission protocol reception and encapsulation format analysis of a video signal source. In addition, the AVCodec library of the FFmpeg framework integrates the functions of most decoders on the market. The application implements on-demand decoding logic in a specific decoder, and only decodes key frames, further optimizing resource utilization. At the same time, the application realizes multi-GOP group buffering logic at the FFtools tool layer, and adds backtracking decoding control logic and still frame detection logic to ensure the accuracy of the detection results.

[0089] The still frame detection scheme of the application is realized based on the engineering toolization of the FFmpeg framework, and can quickly process various video format sources in the form of calling FFmpeg command lines, greatly enhancing the simplicity and ease of use of the still frame detection algorithm. By reusing the powerful audio and video processing capability of the FFmpeg framework, especially its decoding library, transmission protocol library and conversion encapsulation format library, the application avoids the need to repeatedly develop related functional components, significantly saving development and maintenance costs. For enterprises, this not only reduces the human input of the technical team, but also shortens the online cycle of functions, so as to realize commercial application faster. In addition, as an open source framework, FFmpeg has extensive support and community resources, which further reduces technical risks and maintenance costs.

[0090] Figure 4 A flowchart of toolization implementation of the still frame detection scheme disclosed in the embodiments of the application is shown in Figure 5 A schematic diagram of the overall implementation process of the scheme disclosed in the embodiments of the application is shown in Figure 4 and Figure 5 As shown, the still frame detection scheme includes the following contents:

[0091] S101, threshold parameter configuration, according to the specific application scene, determine the threshold parameter required for static frame detection. Mainly includes two key parameters: one parameter is noise threshold (noise), used to judge the similarity between two frames of key frames. When the average absolute difference (mafd) of two frames is less than or equal to the threshold (corresponding to the similarity greater than or equal to the threshold), it is considered that the two frames are similar, and there may be static frame; Another parameter is the static frame duration threshold (duration), used to judge whether the duration of the static frame reaches the preset minimum value. Only when the static frame duration exceeds the threshold, it is determined as an effective static frame scene;

[0092] S102, video signal source receiving and analysis, multiplexing AVFormat library of FFmpeg framework, realizing the support for multiple common live transmission protocols (such as RTSP, UDP, HTTP-HLS, RTMP, WebRTC, SRT, etc.). Through the AVFormat library, the system can receive live video signal source and analyze its encapsulation format to extract video data for subsequent processing. This step ensures that the system can be compatible with multiple video transmission protocols and adapt to different live scene requirements;

[0093] S103, GOP group buffer management, in the FFmpeg framework tool layer, the system performs buffer management on the adjacent 3 GOP (Group of Pictures) groups parsed out. Specifically, it includes: the current processing GOP group (G n ), the GOP group being decoded and detected; the previous GOP group (G n-1 ), used for inter-frame comparison with the current GOP group to determine whether there is a static frame and the end time of the possible static frame scene is in this GOP group; the previous two GOP groups (G n-2 ), used for backtracking detection when needed to accurately determine the start time of the static frame. By buffering multiple GOP groups, the system can backtrack historical frame data when detecting suspected static frames to ensure the accuracy of the detection results.

[0094] S104, key frame decoding and buffering, multiplexing AVCodec library of FFmpeg framework, calling the corresponding decoder to decode and buffer the IDR frames (I n ) in the current GOP group (G n ). For other types of frames in the G n group (such as P frames and B frames), the system only buffers them temporarily without decoding to reduce the consumption of computing resources. When the system first buffers a GOP group, record the first frame time (t0) of the video signal and initialize the static frame start time (T0) as empty. This step provides an initial time reference for subsequent static frame detection;

[0095] S105, key frame similarity calculation, when the system caches at least two GOP groups, the IDR frame (I n ) of the current GOP group (G n ) and the IDR frame (I n-1 ) of the previous GOP group (G n-1 ) are detected for similarity. The specific steps are as follows: first, calculate the average absolute difference (mafd) of all components of the two key frames. Second, when mafd ≤ noise, it is determined to be a suspected still frame scene, at this time: if T0 ≠ empty, it means that there is a suspected still frame scene in the previous period, the system only needs to continue to detect the IDR frame of the subsequent GOP group, that is, continue to execute S105, and finally determine the end time of the still frame; if T0 = empty, the start time of the still frame needs to be determined, if there are only two GOP groups, the start time T0 of the still frame is the time t0 of the first frame (corresponding to the key frame / I frame), otherwise, the system needs to extract all frame data of G n-2 group for full decoding, and compare with I n frame by frame to accurately determine the start time of the still frame. Third, if mafd > noise, it means that the similarity of the two key frames is not high, and the system needs to further process according to whether T0 is empty (see S106);

[0096] S106, non-still frame scene processing, that is, when the mafd of the two adjacent key frames is greater than noise, it means that the two frames are not similar, and the system needs to handle it according to the situation: when T0 ≠ empty, it means that there is a suspected still frame scene in front, the system needs to determine the end time (T1) of the still frame in the previous GOP group (G n-1 ). The specific operation is: full decoding of all frames of G n-1 group, comparison with I n-1 frame by frame to determine the end time of the still frame, if T1-T0 ≥ duration, it is determined to be a valid still frame scene, otherwise, it is only a suspected still picture with insufficient duration, and the T0 value that is not empty needs to be re-assigned to empty, and the next still frame scene detection is entered; when T0 = empty, it means that the adjacent three GOP groups cached do not belong to the still frame scene, and the system continues to detect the key frame still frame of the subsequent live video signal.

[0097] S107, continuous detection and result saving, the system continuously detects the still frame of the live signal source according to the above steps until the live signal ends. During the detection process, the system saves the detected still frame scene in the form of a file to the local or reports it to the cloud database for storage, which is convenient for subsequent viewing and processing. By saving the detection results in real time, the system can provide detailed still frame information for operation and maintenance personnel, which is convenient for quick positioning and problem solving.

[0098] During the whole still frame detection process, the detection result of the key frame needs to be dynamically frame detection controlled according to the situation. For non-key frames, the system only decodes and detects when necessary. For example, when the key frame detects a still frame, the decoding of the non-key frame can be dynamically started to further confirm the duration of the still frame. This dynamic frame detection control strategy not only guarantees the detection accuracy, but also further reduces the consumption of computing resources. This scheme not only can significantly reduce the consumption of computing resources, but also can maintain high detection speed and accuracy in high concurrency and high resolution live scenes.

[0099] In actual application scenarios, the key frame based skip frame detection mechanism involved in the present application can be integrated into the transcoding FFmpeg tool. A ffmpeg task process is enabled for each television live video signal source for separate detection, which can greatly reduce the computational complexity while ensuring the detection accuracy, thereby realizing efficient real-time processing of live stream still frame detection. On the basis of the original hardware monitoring resources, quality monitoring and detection processing of multi-channel high concurrency and high resolution live video signals can be realized, greatly improving the throughput of the system processing.

[0100] An example of the ffmpeg detection command line used for each television live signal source is as follows:. / ffmpeg -i udp: / / 233.233.233.233:1234 -work_threads_num 4 -keyframe_decode_enable 1 -video_freeze_noise_threshold 0.001 -video_freeze_duration_seconds 5.0 -detect_file_output. / info.txt -f null / dev / null. The task command line can achieve static frame detection on the multicast live video source udp: / / 233.233.233.233:1234, work_threads_num=4 indicates that 4 threads are started to decode the source signal in parallel to speed up, keyframe_decode_enable=1 indicates that key frame decoding is started, the specific decoder control is transmitted to only decode the key frame, video_freeze_noise_threshold=0.001 indicates that when the average absolute difference of all components of two frames is less than 0.001, the frames are judged as similar frames, video_freeze_duration_seconds=5.0 indicates that when the continuous similar frame duration exceeds 5.0 seconds, the static frame scene is judged, detect_file_output=. / info.txt indicates that the static frame scene detection result saving file, -f null dev / null indicates that the decoded data frame does not need subsequent encoding processing and file saving. Compared with the way of detecting all types of frames of the live video signal source, the speed of the application is improved by several tens of times.

[0101] The static frame detection scheme proposed in the application preferentially decodes and detects the key frame (I frame) in the video stream, and skips the decoding process of a large number of non-key frames (P frame and B frame). The strategy has three technical advantages, first, the computing resource consumption is significantly reduced: by reducing the decoding of non-key frames, the system can greatly reduce the computing resource consumption, especially in high-resolution and high-frame-rate video streams, the resource optimization effect is more obvious; second, the detection speed is improved, the frame skipping detection strategy reduces the number of frames to be processed, significantly improves the detection speed, and can meet the real-time requirement; third, the detection accuracy is guaranteed: by buffering multiple GOP groups and combining the backtracking detection mechanism, the system can accurately determine the start and end time of the static frame when detecting the suspected static frame, and avoid misjudgment.

[0102] In addition, the scheme is implemented as an engineering tool based on the FFmpeg framework, which fully utilizes the powerful audio and video processing capabilities of the FFmpeg framework. This approach has three advantages. First, it reuses the powerful functions of FFmpeg: by reusing the AVFormat library and the AVCodec library of FFmpeg, the system can support multiple common live transmission protocols (such as RTSP, UDP, HTTP-HLS, RTMP, WebRTC, SRT, etc.), and implement efficient video decoding and encapsulation format parsing. Second, it reduces development and maintenance costs: reusing the FFmpeg framework avoids the need to develop video processing modules repeatedly, significantly reducing development and maintenance costs. Third, it improves system compatibility: the wide support of the FFmpeg framework enables the present technology to be compatible with multiple video formats and transmission protocols, and suitable for different application scenarios.

[0103] For each of the foregoing method embodiments, for the sake of simple description, each is described as a combination of a series of actions, but those skilled in the art should appreciate that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should appreciate that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0104] The method is described in detail in the embodiments disclosed in the present application. The method of the present application can be implemented in various forms of devices, and therefore the present application also discloses a device. The following specific embodiments are given to explain in detail.

[0105] Figure 6 A structural schematic diagram of a still frame detection device disclosed in an embodiment of the present application is shown in FIG. 6. Figure 6 As shown in FIG. 6, the still frame detection device 60 can include:

[0106] An image group obtaining module 601 is configured to obtain an Nth image group G N , where N is a positive integer.

[0107] A decoding processing module 602 is configured to decode key frames in the obtained image group G N .

[0108] A similarity determining module 603 is configured to determine a similarity between the key frames in the image group G N and the key frames in an (N-1)th image group G N-1 .

[0109] A still frame judging module 604 is configured to determine whether there is a still frame based on the similarity and whether there is a still frame start time, the still frame start time being a time at which the key frames in the image group G M and the image group G M-1The data set is set in a case where the similarity of the key frames satisfies a set condition, and M is less than N.

[0110] The still frame detection device described in the embodiment is realized based on a skip frame detection strategy of key frames, that is, the still frame detection is preferentially performed on the key frames in the video signal, and if no still frame is found, the detection of other frame types in the current GOP is directly skipped. This implementation not only reduces the overhead of the decoder in processing non-key frames, but also greatly reduces the frame data amount required for still frame detection, so that the detection speed is greatly improved. Moreover, this efficient detection manner significantly reduces the occupation of server resources, and reduces the hardware cost and energy consumption.

[0111] The specific implementation of the still frame detection device and each module contained therein, and other possible implementations can be referred to the content introduction of the corresponding part in the method embodiment, which will not be repeated here.

[0112] The still frame detection device in any of the above embodiments includes a processor and a memory. The image group obtaining module, the decoding processing module, the decoding processing module, the still frame judging module, and the like in the above embodiments are all stored in the memory as program modules, and the corresponding functions are realized by the processor executing the above program modules stored in the memory.

[0113] The processor includes a core, and the core retrieves the corresponding program modules from the memory. The core can be set as one or more, and the processing of the revisit data is realized by adjusting the core parameters.

[0114] The memory can include a non-permanent memory in a computer readable medium, a random access memory (RAM), and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory includes at least one memory chip.

[0115] In an exemplary embodiment, a computer readable storage medium directly loadable into the internal memory of a computer is also provided, which contains software codes. After the computer program is loaded and executed by the computer, the steps shown in any of the above still frame detection methods can be realized.

[0116] In an exemplary embodiment, a computer program product directly loadable into the internal memory of a computer is also provided, which contains software codes. After the computer program is loaded and executed by the computer, the steps shown in any of the above still frame detection methods can be realized.

[0117] The various embodiments described in this specification are intended to be exemplary only. The subject matter described in this specification can be implemented in software, hardware, or a combination thereof. The various embodiments described in this specification can be implemented as part of a method, device, manufacture, composition of matter, process, or machine. The various embodiments described in this specification can be implemented in one or more of the following example forms:

[0118] It should also be noted that the term "comprising" is used herein to mean that the elements following the term in the list of claim are included, but not to the exclusion of any other elements not specifically listed. Thus, the term "comprising" is used herein to mean that the claims include the elements in the list following the term, but not to the exclusion of other elements of additional elements not specifically listed.

[0119] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, hard disk can be used as a storage medium.

[0120] The above description of disclosed embodiments is intended to be illustrative only. Numerous modifications employing the principles of the present application will occur to those skilled in the art. The disclosure shows the generic principles of the application which can be applied in a variety of embodiments and each of the example embodiments described herein is not necessarily to be construed as limiting the scope of the disclosure. Accordingly, the disclosure is not intended to be limited to the examples contained herein which are for illustrative purposes only. Rather, the scope of the disclosure is to be limited solely by the claims set forth below.

Claims

1. A method of detecting a quiet frame, the method comprising: The method comprises: obtaining an Nth image group G N where N is a positive integer; Decoding obtains the key frame in the image group G N ​ determining a similarity of key frames in the image group G N to key frames of the N-1th image group G N-1 ​ determining whether there is a static frame based on the similarity and whether there is a static frame start time at present, comprising: if the similarity is greater than or equal to a noise threshold value and there is a static frame start time, continuing the static frame detection processing of the subsequent image group; if the similarity is greater than or equal to the noise threshold value and there is no static frame start time, determining the static frame start time based on the buffered image frame data of the historical image group; If the similarity is less than a noise threshold and a still frame start time exists, determining a still frame end time based on the group of images G N-1 determining a still frame end time; if the similarity is less than the noise threshold value and there is no static frame start time, continuing the static frame detection processing of the subsequent image group; determining whether there is a static frame based on the static frame start time and the static frame end time; The start time of the still frame is in image group G. M Keyframes and Image Groups G M-1 The data is set when the similarity of the keyframes meets the set conditions, and M is less than N.

2. The quiet frame detection method of claim 1, wherein, The determining the similarity between the key frame in the image group G N and the key frame of the N-1th image group G N-1 includes: determining key frames in the image group G N and the average absolute difference of all components in the image encoding format of the key frames of the image group G N-1 .

3. The quiet frame detection method of claim 1, wherein, In the process of still frame detection, the image groups obtained in sequence are cached with the reception of the video signal, the maximum number of cached image groups is three, when the cached image groups are three, the cached image groups include the current processing image group G N , the image group G N-1 and the image group G N-2 ; the determination of the static frame start time based on the buffered image frame data of the historical image group comprises: If there are only two image groups cached at present, the image group G N-1 whose key frame corresponds to the moment determined as the still frame start moment is selected as the image group to be cached. If there are three image groups in the current cache, extract all frame data of the image group G N-2 , perform full decoding, and compare all image frames obtained by decoding with the key frame of the image group G N , and determine the time corresponding to the image frame with a similarity greater than or equal to the noise threshold to the key frame of the image group G N as the still frame start time. If there are three image groups in the current cache, and the image group G N-2 Decode all the image frames obtained and the key frame of the image group G N If the similarity between the key frame of the image group G N-1 The time corresponding to the key frame of the image group G is determined as the still frame start time.

4. The quiet frame detection method of claim 1, wherein, The image-based group G N-1 determining the end of the static frame, comprising: extracting the image group G N-1 decodes all frame data of the image group G N-1 compares all decoded image frames with key frames of the image group G N-1 determines a time point corresponding to an image frame with a similarity greater than or equal to a noise threshold to a key frame of the image group G 5. The quiet frame detection method of claim 1, wherein, the determination of whether there is a static frame based on the static frame start time and the static frame end time comprises: determining a time difference value from the static frame start time to the static frame end time; if the time difference value is greater than a time threshold value, determining that there is a static frame; if the time difference value is less than or equal to the time threshold value, determining that there is no static frame and setting the static frame start time as null.

6. The quiet frame detection method of claim 5, wherein, Further comprising: pre-configuring the noise threshold value and the time threshold value.

7. The quiet frame detection method of claim 1, wherein, Further comprising: if there is a static frame, saving or uploading the static frame data containing the static frame time to the cloud.

8. The quiet frame detection method of any of claims 1-7, wherein, The static frame detection method is implemented by reusing a first multimedia processing framework.

9. A quiet frame detection apparatus, characterized by comprising: The method comprises: an image group obtaining module, configured to obtain an Nth image group G N wherein N is a positive integer; a decoding processing module, configured to decode the key frame in the obtained image group G N ​ The similarity determination module is configured to determine the similarity between the key frame in the image group G N and the key frame of the N-1th image group G N-1 . a static frame judgment module, configured to determine whether there is a static frame based on the similarity and whether there is a static frame start time at present, comprising: if the similarity is greater than or equal to a noise threshold value and there is a static frame start time, continuing the static frame detection processing of the subsequent image group; if the similarity is greater than or equal to the noise threshold value and there is no static frame start time, determining the static frame start time based on the buffered image frame data of the historical image group; If the similarity is less than a noise threshold and a still frame start time exists, determining a still frame end time based on the group of images G N-1 determining a still frame end time; if the similarity is less than the noise threshold value and there is no static frame start time, continuing the static frame detection processing of the subsequent image group; determining whether there is a static frame based on the static frame start time and the static frame end time; The still frame start time is set as a time point at which the image group G M The similarity between the key frame of the image group G M-1 The M is smaller than the N.

Images