Live broadcast picture jitter suppression method and device and storage medium

Abstract

The invention relates to a live broadcast picture jitter suppression method and device and a storage medium. The method comprises the following steps: S1, judging whether a picture jitters or not based on a plurality of continuous sampling frames; s2, when the picture shakes, selecting one of the historical frames as a continuous display frame; and S3, returning the picture to the continuous display frame. Compared with the prior art, the problem of picture jitter caused by poor contact of a network or a display socket of a user side in the live broadcast process can be effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of streaming media, in particular to a live picture jitter suppression method and device and storage medium. BACKGROUND

[0002] With the rapid development of network technology and streaming media applications, live streaming has become an important form of information dissemination and interactive entertainment. In the live streaming process, the video stream received and displayed by the user end needs to maintain a high degree of smoothness and stability to obtain a good viewing experience. However, in actual application scenarios, due to fluctuations in network transmission environment, bandwidth limitations, or poor contact of user end device display interfaces (such as HDMI, DisplayPort, etc.), and other factors, abnormal jitter, lag or repeated frames often occur in the live picture. Such picture jitter not only seriously affects the viewing quality, but also may cause discomfort to the user, and even lead to the user interrupting the viewing.

[0003] Currently, there are some common solutions to the problems in video stream transmission. For example, through network protocols such as forward error correction (FEC), automatic repeat request (ARQ) to ensure reliable transmission of data packets, or setting a buffer at the receiving end to smooth the unevenness of data flow caused by network delay and jitter. However, these methods mainly focus on solving the data packet loss or delay at the network level, and have limited effect on suppressing picture jitter caused by poor contact of display interfaces and other local hardware problems. In addition, although the existing buffering technology can alleviate the lag to some extent, it may introduce additional delay, and when the picture jumps abnormally, simple buffering cannot intelligently identify and repair such picture defects, and sometimes even causes repeated display of jitter frames, exacerbating the visual discontinuity.

[0004] Therefore, the existing technology lacks a solution that can effectively and in real time detect and suppress live picture jitter caused by various reasons (including network problems and local hardware problems), and there is an urgent need for a method that can quickly respond when detecting picture abnormalities and maintain picture stability by intelligently selecting historical frames. SUMMARY

[0005] The purpose of the present application is to provide a live picture jitter suppression method, device and storage medium to solve the defects in the prior art.

[0006] The purpose of the present application can be achieved by the following technical solutions: A live picture jitter suppression method, comprising: Step S1: judging whether the picture has jitter based on a plurality of consecutive sample frames; Step S2: when the picture has jitter, selecting one of the historical frames as a continuous display frame; Step S3: Return the screen to the continuously displayed frame.

[0007] Step S1 includes: Step S1-1: When the live streaming window is in the foreground, continuously capture the current scene as a sampling frame according to the first pre-configured sampling interval; Step S1-2: Initialize the jump flag to 0; Step S1-3: Obtain two consecutive sampling frames, compare the difference in the background part of the two sampling frames, if the difference is less than the pre-configured threshold, initialize the status flag to 0, otherwise initialize the status flag to 1; Step S1-4: Use the two sampled frames as the first buffer frame and the second buffer frame respectively, wherein the first buffer frame is located before the second buffer frame; Step S1-5: Obtain the latest sampling frame and determine whether the initialization status flag is 1. If it is, proceed to step S1-6; otherwise, proceed to step S1-7. Step S1-6: Determine the background differences between the latest sampled frame and the first buffered frame, as well as the background differences between the latest sampled frame and the second buffered frame. If the background differences between the latest sampled frame and both the first and second buffered frames exceed a pre-configured threshold, then the image is determined to be jittery; otherwise, proceed to step S1-7. Step S1-7: If the difference between the background portion of the latest sampled frame and the second buffered frame is less than the pre-configured threshold, then the second buffered frame is used as the new first buffered frame, the latest sampled frame is used as the new second buffered frame, and the process returns to step S1-5. If the difference between the background portion of the latest sampled frame and the first buffered frame is less than the pre-configured threshold, but the difference between the background portion of the latest sampled frame and the second buffered frame exceeds the pre-configured threshold, determine whether the jump flag is 0. If it is, then the latest sampled frame is used as the new second buffered frame, the jump flag is set to 1, and the process returns to steps S1-5. Otherwise, it is determined that the image is jittering.

[0008] The method for obtaining the difference in the background portion of two sampled frames is as follows: The background portions of the two samples are extracted based on the pre-configured mask image; Calculate the difference in the background portion.

[0009] The method for obtaining the difference in the background portion of two sampled frames is as follows: The two sampled frames are input into the target detection model to obtain the position of each target. The differences in the positions of each target are calculated separately, and the differences in the position and size information of the multiple targets with the smallest differences are taken as the differences in the background part of the sampling frame.

[0010] The method of using the difference in position information of multiple targets with the smallest difference as the difference in the background part of the sampling frame is specifically as follows: For all targets in the first sampling frame, calculate their positional differences with the targets in the second sampling frame, and take the target with the smallest positional difference in the second sampling frame as the matching target of the target in the first sampling frame; Sort all targets in the first sampled frame in ascending order of the difference in distance to their matching targets; The difference between the position and size information of the first half of the target and its matching target is summed to represent the difference in the background portion of the sampled frame: in: The difference in the background portion between the two sampled frames, This is the background portion of the first sampled frame. The second sampled frame contains the background portion, where w1 is the weight of the position term, w2 is the weight of the area term, and N is the integer obtained by rounding down half of the detected target. This represents the position of the i-th target in the first sampling frame. Let be the position of the matching target of the i-th target in the first sampling frame in the second sampling frame. Let be the area of ​​the i-th target in the first sampling frame. Let be the area of ​​the matching target of the i-th target in the first sampling frame in the second sampling frame.

[0011] The target's location is the location of its center.

[0012] In step S2, the sampling frame preceding the last sampling frame is selected as the continuous display frame.

[0013] In step S3, the screen is returned to the continuously displayed frame and remains there for 1-2 seconds.

[0014] A live streaming image jitter suppression device includes a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the method described above.

[0015] A storage medium having a program stored thereon, which, when executed, implements the method described above.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. A complete jitter suppression process has been constructed, systematically solving the entire process from problem identification to repair implementation, rather than just local optimization. It specifically targets image jitter caused by network issues or poor contact of the display interface, rather than simple image blurring or regular stuttering, solving specific problems that have not been effectively addressed in existing technologies. It adopts a strategy of reverting to historical continuously displayed frames, rather than complex frame interpolation or image restoration algorithms, resulting in low computational overhead and fast response speed. It can quickly stabilize the image without significantly increasing latency, making it ideal for live streaming scenarios with high real-time requirements.

[0017] 2. By introducing a first buffer frame, a second buffer frame, an initialization status flag, and a jump flag, this algorithm can distinguish between normal changes and abnormal jitter in scenes such as slow camera movement or small object movement. It requires abnormal differences between multiple consecutive frames to be considered jitter, effectively reducing the false positive rate. The jump flag mechanism allows the system to tolerate a possible instantaneous anomaly or single-frame difference, avoiding unnecessary corrections triggered by a single slight fluctuation, thus enhancing the algorithm's robustness.

[0018] 3. The background is extracted using a pre-configured mask image, avoiding complex image analysis or model calculations. The processing speed is extremely fast, and the system's computing resource requirements are low, which is conducive to real-time operation on user-end devices with limited performance.

[0019] 4. Utilizing an object detection model, this method can intelligently identify static or slowly moving background targets in an image, calculating differences by comparing the positional changes of these stable targets. This method better reflects overall image movement or jitter, and compared to simple pixel differences, it is less sensitive to interference factors such as changes in lighting and the movement of foreground objects, resulting in more accurate detection results.

[0020] 5. By matching targets and selecting the half of the targets with the smallest differences as stable targets that are most likely to be the background, and taking into account their position and area changes, the difference value can more accurately represent the true degree of background offset, further eliminating the interference of mismatched targets and foreground movement, and improving the accuracy of jitter detection.

[0021] 6. By clearly defining the position of the target center as the comparison benchmark, the calculation of positional differences becomes consistent and stable, avoiding positioning deviations caused by irregular target shapes and simplifying the calculation process.

[0022] 7. Select the frame preceding the last sampled frame, which is usually a frame that is temporally close and has been verified as relatively stable. This frame avoids using the latest frame that may have already experienced jitter and minimizes the time delay caused by image rewind, ensuring the stability and timeliness of the image after the switch. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the main steps of the method of the present invention. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solution of the present invention, providing detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0025] A method for suppressing live stream jitter, such as Figure 1 As shown, it includes: Step S1: Determine whether image jitter has occurred based on multiple consecutive sampled frames, including: Step S1-1: When the live streaming window is in the foreground, continuously capture the current scene as a sampling frame according to the first pre-configured sampling interval; Whether the live streaming window is in the foreground can be determined by the status of the live streaming process. The first pre-configured sampling interval can generally be 1 second or 0.1 seconds.

[0026] Step S1-2: Initialize the jump flag to 0. This flag is used to record whether a possible abnormal frame has been tolerated during the detection process. It is an important mechanism to improve the robustness of the algorithm and prevent misjudgment. Step S1-3: Obtain two consecutive sampling frames, compare the difference in the background part of the two sampling frames, if the difference is less than the pre-configured threshold, initialize the status flag to 0, otherwise initialize the status flag to 1; Step S1-4: Use the two sampled frames as the first buffer frame and the second buffer frame respectively, wherein the first buffer frame is located before the second buffer frame; Step S1-5: Obtain the latest sampling frame and determine whether the initialization status flag is 1. If it is, proceed to step S1-6; otherwise, proceed to step S1-7. Step S1-6: Determine the background differences between the latest sampled frame and the first buffered frame, as well as the background differences between the latest sampled frame and the second buffered frame. If the background differences between the latest sampled frame and both the first and second buffered frames exceed a pre-configured threshold, then the image is determined to be jittery; otherwise, proceed to step S1-7. Step S1-7: If the difference between the background portion of the latest sampled frame and the second buffered frame is less than the pre-configured threshold, then the second buffered frame is used as the new first buffered frame, the latest sampled frame is used as the new second buffered frame, and the process returns to step S1-5. If the difference between the background portion of the latest sampled frame and the first buffered frame is less than the pre-configured threshold, but the difference between the background portion of the latest sampled frame and the second buffered frame exceeds the pre-configured threshold, determine whether the jump flag is 0. If it is, then the latest sampled frame is used as the new second buffered frame, the jump flag is set to 1, and the process returns to steps S1-5. Otherwise, it is determined that the image is jittering.

[0027] In some embodiments, the background portion is defined by a mask image, and the difference between the background portions of two sampled frames is obtained as follows: The background portions of the two samples are extracted based on the pre-configured mask image; Calculate the difference in the background portion.

[0028] This method is suitable for live streaming scenarios with relatively fixed backgrounds (such as news broadcasts and lecture broadcasts). The system pre-stores a "mask image," a binary image that marks areas of static background (such as studio walls and the podium) and potentially dynamic foreground areas. When calculating the difference, the pure background pixel regions are first extracted from the two sampled frames using this mask image. Then, the average difference or mean square error of the pixel values ​​in these two background regions is calculated as the difference value. This method is fast and has low resource consumption.

[0029] In this embodiment, the difference in the background portion of the two sampling frames is obtained as follows: The two sampled frames are input into the target detection model to obtain the position of each target. The differences in the positions of each target are calculated separately, and the differences in the position and size information of the multiple targets with the smallest differences are taken as the differences in the background part of the sampling frame.

[0030] The difference in the positional information of multiple targets with the smallest difference is taken as the difference in the background part of the sampling frame, specifically: For all targets in the first sampling frame, calculate their positional differences with the targets in the second sampling frame, and take the target with the smallest positional difference in the second sampling frame as the matching target of the target in the first sampling frame; Sort all targets in the first sampled frame in ascending order of the difference in distance to their matching targets; The difference between the position and size information of the first half of the target and its matching target is summed to represent the difference in the background portion of the sampled frame: in: The difference in the background portion between the two sampled frames, This is the background portion of the first sampled frame. The second sampled frame contains the background portion, where w1 is the weight of the position term, w2 is the weight of the area term, and N is the integer obtained by rounding down half of the detected target. This represents the position of the i-th target in the first sampling frame. Let be the position of the matching target of the i-th target in the first sampling frame in the second sampling frame. Let be the area of ​​the i-th target in the first sampling frame. Let be the area of ​​the matching target of the i-th target in the first sampling frame in the second sampling frame, and let be the position of the target's center.

[0031] Step S2: When the screen jitters, select one frame from the historical frames as the continuous display frame. In step S2, the previous sampling frame of the last sampling frame is selected as the continuous display frame because this frame is the most recent in time and has not yet been judged as abnormal. It is likely to be a stable screen, which can minimize the sense of time regression caused by screen rewind.

[0032] Step S3: Return the screen to the continuous display frame. In step S3, return the screen to the continuous display frame and continue for 1-2 seconds.

[0033] The system performs a stabilization operation: immediately stops playing the currently jittery video stream and switches the displayed content to the continuously displayed frame selected in step S2, keeping this frame static for a period of time, such as 1.5 seconds. This brief static period serves two purposes: first, to provide a time window for the system or network to recover stability; and second, to give the user a smooth visual transition, avoiding continuous jumps. After 1.5 seconds, the system clears the buffer and flag state, and restarts a new round of monitoring and sampling from step S1, resuming normal playback.

[0034] After the time is up, it will return to step S1.

[0035] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for suppressing live stream jitter, characterized in that, include: Step S1: Determine whether the image is jittering based on multiple consecutive sampled frames; Step S2: When screen jitter occurs, select one frame from the history frames as the continuous display frame; Step S3: Return the screen to the continuously displayed frame.

2. The method for suppressing live stream jitter according to claim 1, characterized in that, Step S1 includes: Step S1-1: When the live streaming window is in the foreground, continuously capture the current scene as a sampling frame according to the first pre-configured sampling interval; Step S1-2: Initialize the jump flag to 0; Step S1-3: Obtain two consecutive sampling frames, compare the difference in the background part of the two sampling frames, if the difference is less than the pre-configured threshold, initialize the status flag to 0, otherwise initialize the status flag to 1; Step S1-4: Use the two sampled frames as the first buffer frame and the second buffer frame respectively, wherein the first buffer frame is located before the second buffer frame; Step S1-5: Obtain the latest sampling frame and determine whether the initialization status flag is 1. If it is, proceed to step S1-6; otherwise, proceed to step S1-7. Step S1-6: Determine the background differences between the latest sampled frame and the first buffered frame, as well as the background differences between the latest sampled frame and the second buffered frame. If the background differences between the latest sampled frame and both the first and second buffered frames exceed a pre-configured threshold, then the image is determined to be jittery; otherwise, proceed to step S1-7. Step S1-7: If the difference between the background portion of the latest sampled frame and the second buffered frame is less than the pre-configured threshold, then the second buffered frame is used as the new first buffered frame, the latest sampled frame is used as the new second buffered frame, and the process returns to step S1-5. If the difference between the background portion of the latest sampled frame and the first buffered frame is less than the pre-configured threshold, but the difference between the background portion of the latest sampled frame and the second buffered frame exceeds the pre-configured threshold, determine whether the jump flag is 0. If it is, then the latest sampled frame is used as the new second buffered frame, the jump flag is set to 1, and the process returns to steps S1-5. Otherwise, it is determined that the image is jittering.

3. The method for suppressing live stream jitter according to claim 2, characterized in that, The method for obtaining the difference in the background portion of two sampled frames is as follows: The background portions of the two samples are extracted based on the pre-configured mask image; Calculate the difference in the background portion.

4. The method for suppressing live stream jitter according to claim 2, characterized in that, The method for obtaining the difference in the background portion of two sampled frames is as follows: The two sampled frames are input into the target detection model to obtain the position of each target. The differences in the positions of each target are calculated separately, and the differences in the position and size information of the multiple targets with the smallest differences are taken as the differences in the background part of the sampling frame.

5. The live streaming image jitter suppression method according to claim 4, characterized in that, The method of using the difference in position information of multiple targets with the smallest difference as the difference in the background part of the sampling frame is specifically as follows: For all targets in the first sampling frame, calculate their positional differences with the targets in the second sampling frame, and take the target with the smallest positional difference in the second sampling frame as the matching target of the target in the first sampling frame; Sort all targets in the first sampled frame in ascending order of the difference in distance to their matching targets; The difference between the position and size information of the first half of the target and its matching target is summed to represent the difference in the background portion of the sampled frame: in: The difference in the background portion between the two sampled frames, This is the background portion of the first sampled frame. The second sampled frame contains the background portion, where w1 is the weight of the position term, w2 is the weight of the area term, and N is the integer obtained by rounding down half of the detected target. This represents the position of the i-th target in the first sampling frame. Let be the position of the matching target of the i-th target in the first sampling frame in the second sampling frame. Let be the area of ​​the i-th target in the first sampling frame. Let be the area of ​​the matching target of the i-th target in the first sampling frame in the second sampling frame.

6. The method for suppressing live stream jitter according to claim 5, characterized in that, The target's location is the location of its center.

7. The live streaming image jitter suppression method according to claim 4, characterized in that, In step S2, the sampling frame preceding the last sampling frame is selected as the continuous display frame.

8. The method for suppressing live stream jitter according to claim 4, characterized in that, In step S3, the screen is returned to the continuously displayed frame and remains there for 1-2 seconds.

9. A live streaming image jitter suppression device, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-8.

10. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-8.

Images