Screen cracking protection method, device and storage medium based on active frame loss
By identifying and discarding erroneous data packets caused by electrostatic interference in the live broadcast device, the problem of screen splitting of the live broadcast device is solved, achieving a lower cost and unaffected picture protection effect.
Patent Information
- Application Number
- CN202211703596.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-29
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-12-29
AI Technical Summary
Live broadcast equipment is prone to instability of SDI or HDMI input source signals under static interference, resulting in errors or loss of video and audio packets, which will lead to screen splitting problems. The existing hardware protection methods affect other functions and cannot eliminate all screen splitting signals and High cost issue.
Through software methods, the backlog rate of video and audio packets on the push streaming end is obtained, the error data and frames generated by the cracking of the screen is identified and actively discarded, and the data is collected is sent to the acquisition end and the data is re-acquisitioned, and the FFmpeg software push streaming technology is pushed to Streaming server.
It effectively solves the problem of screen splitting in live broadcast scenarios, avoids hardware modification and the use of electrostatic protection networks, reduces costs, and does not affect other functions of live broadcast equipment.
Smart Images

Figure CN116112739B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of live video picture transmission, and in particular to a screen cracking protection method, device and storage medium based on active frame loss. Background Art
[0002] At present, the main video transmission technology used is FFmpeg software streaming technology, which can be divided into the following steps: ① The SDI / HDMI input source collects the original video and audio data; ② The encoder encodes the video and audio data into encoding packets of the specified encoding format (such as video data encoded into H264 encoding packets, and audio data encoded into AAC encoding packets); ③ The FFmpeg streaming end initializes two queues for storing video encoding packets and audio encoding packets; ④ The FFmpeg streaming end encapsulates the video encoding packets and audio encoding packets according to the set streaming protocol (such as RTMP) according to the timestamp and pushes the stream to the specified streaming media server; ⑤ The user uses ffplay or vlc player to pull the stream from the streaming media server to watch the video at the pulling end.
[0003] The main input sources of live broadcast equipment are SDI and HDMI. When the live broadcast equipment is interfered by static electricity, it is easy to cause the SDI or HDMI input source signal to be unstable, triggering a cracked screen signal. The cracked screen signal will cause the video or audio packets collected by the live broadcast equipment to be erroneous or lost. These erroneous or lost data packets will cause the FFmpeg software to push data disordered and unable to push data out as a complete frame. When the FFmpeg push end interleaves the data corresponding to two frames into one frame and pushes it out, the problem of cracked screen will occur on the streaming playback end.
[0004] At present, the method to solve screen cracking mainly relies on hardware protection, such as hardware anti-static protection on the input source side, or automatic reset after the SDI or HDMI input source hardware recognizes the screen cracking signal. However, the method of solving screen cracking based on hardware protection has the following three disadvantages: First, adding anti-static metal shielding nets may affect other functions of the live broadcast equipment. For example, for live broadcast equipment with wireless dial-up modules for Internet access, the presence of metal shielding nets will seriously affect the wireless network signal of the live broadcast equipment; second, hardware protection cannot eliminate all screen cracking signals; third, hardware modification is costly. Summary of the invention
[0005] The purpose of the present invention is to solve the problem of screen cracking in a live broadcast scenario through a software method.
[0006] The technical solution adopted by the present invention is: the present invention provides a screen cracking protection method based on active frame loss, which is characterized by comprising:
[0007] Obtain the backlog rate of the video packet storage queue and the audio packet storage queue of the streaming end;
[0008] The streaming end determines whether a screen cracking signal occurs according to the backlog rate;
[0009] After the streaming end identifies the screen tearing signal, it actively discards the erroneous data and frames caused by the screen tearing;
[0010] After the streaming end actively completes frame loss, it sends a notification message to the acquisition end, and the acquisition end re-acquires the video and audio data;
[0011] The streaming end streams the video and audio data to the streaming media server.
[0012] Furthermore, the backlog rate calculation method of the video packets at the streaming end in the time period from t to t1 is:
[0013] V video =(|S video {t1}-S video {t}|) / (|t1-t|)
[0014] Among them, V video Indicates the backlog rate of video data packets. The software loops to obtain the video packet storage queue Q at the streaming end at time t. video The number of packets is S video {t}, the video packet storage queue Q at the streaming end at time t1 video The number of packets is S video {t1};
[0015] The calculation method of the backlog rate of the audio packets at the streaming end in the time period from t to t1 is:
[0016] V audio =(|S audio {t1}-S audio {t}|) / (|t1-t|)
[0017] Among them, V audio Indicates the backlog rate of audio data packets. The software loops to obtain the audio packet storage queue Q at the streaming end at time t. audio The number of packets is S audio {t}, the video packet storage queue Q at the streaming end at time t1 audio The number of packets is S audio {t1}.
[0018] Furthermore, the step of determining whether screen tearing occurs based on the backlog rate includes: setting a backlog rate M of video or audio data packets corresponding to a screen tearing signal, and when the backlog rate M is or is greater than M, it is considered that screen tearing occurs.
[0019] Furthermore, the erroneous data and erroneous frames generated by screen tearing include or backlogged video or audio data packets.
[0020] Furthermore, the active frame loss is achieved by releasing a memory that has accumulated erroneous data and erroneous frames.
[0021] Furthermore, the streaming end adopts FFmpeg software streaming technology to perform streaming.
[0022] Furthermore, the acquisition end re-acquires the video and audio data, and the input source of the acquisition end includes an SDI input source and an HDMI input source.
[0023] Furthermore, the streaming end encapsulates the video encoding package and the audio encoding package into a specified format according to the set streaming protocol according to the timestamps and streams them to the specified streaming media server.
[0024] On the other hand, the present invention provides a computer device, characterized in that it includes a memory and a processor, the memory is used to store at least one program, and the processor is used to load the at least one program to execute any one of the active frame loss-based screen tearing protection methods.
[0025] On the other hand, the present invention provides a computer-readable storage medium, which stores a program executable by a processor, characterized in that the program executable by the processor is used to execute any one of the screen tearing protection methods based on active frame loss when executed by the processor.
[0026] The beneficial effects of the present invention are: solving the problem of screen cracking in live broadcast scenarios through software methods, the effect is better than the method based on hardware protection; and because there is no need to modify the hardware, the cost is low; there is no need to add a metal shielding net for electrostatic protection, and it will not affect other functions of the live broadcast equipment. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 This is a demonstration of the actual effect of screen cracking;
[0028] Figure 2 This is the process of streaming using FFmpeg software in the embodiment of the present invention;
[0029] Figure 3 This is the encapsulation process when the FFmpeg software is used to push the stream according to the RTMP protocol in the embodiment of the present invention;
[0030] Figure 4 The figure is a flow chart of a software cracked screen protection algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION
[0031] In order to describe the technical solution and advantages of the relational data table organization structure more completely and clearly, the data structure will be analyzed and explained in conjunction with examples and drawings. Obviously, the example is only used to explain the present invention and is not used to limit the present invention.
[0032] With the rapid development of Internet technology and the live broadcast industry, various applications have higher and higher requirements for video quality. However, there are various video quality problems in actual live broadcast scenarios, such as freezes, audio and video asynchrony, mosaics, etc. In more serious cases, there may even be screen cracking, screen flashing, and black screens. In the embodiments of the present invention, the problem of screen cracking in live broadcast scenarios is discussed through software methods. Screen cracking is manifested as a complete video frame being split into two frames and displayed on the screen, such as Figure 1 shown.
[0033] In current live broadcast scenarios, FFmpeg software streaming technology is mostly used. Figure 2 The process of streaming using FFmpeg software in the embodiment of the present invention is mainly divided into the following steps:
[0034] S101.SDI / HDMI input source captures raw video and audio data;
[0035] S102. The encoder encodes the video and audio data into a coding packet of a specified coding format (e.g., the video data is encoded into an H264 coding packet, and the audio data is encoded into an AAC coding packet)
[0036] S103. FFmpeg streaming end initializes two queues to store video encoding packets and audio encoding packets
[0037] S104. The FFmpeg streaming end encapsulates the video encoding package and the audio encoding package into a specified format (such as FLV) according to the set streaming protocol (such as RTMP) according to the timestamp and streams them to the specified streaming media server;
[0038] S105. The user uses ffplay or vlc player to pull the stream from the streaming media server to watch the video at the streaming end.
[0039] Figure 3 The embodiment of the present invention uses the encapsulation process of FFmpeg software when pushing the stream according to the RTMP protocol to encapsulate the video H264 encoding package and the audio AAC encoding package into an FLV video stream corresponding to the RTMP protocol.
[0040] The main input sources of live broadcast equipment are SDI and HDMI. When the live broadcast equipment is interfered by static electricity, it is easy to cause the SDI or HDMI input source signal to be unstable, triggering a cracked screen signal. The cracked screen signal will cause the video or audio packets collected by the live broadcast equipment to be erroneous or lost. These erroneous or lost data packets will cause the FFmpeg software to push data disordered and unable to push data out as a complete frame. When the FFmpeg push end interleaves the data corresponding to two frames into one frame and pushes it out, the problem of cracked screen will occur on the streaming playback end.
[0041] At present, the methods to solve screen cracking mainly rely on hardware protection, such as hardware anti-static protection on the input source side, or automatic reset after the SDI or HDMI input source hardware recognizes the screen cracking signal. However, the method of solving screen cracking based on hardware protection has the following three disadvantages: First, adding anti-static metal shielding nets may affect other functions of the live broadcast equipment. For example, for live broadcast equipment with wireless dial-up modules for Internet access, the presence of metal shielding nets will seriously affect the wireless network signal of the live broadcast equipment; second, hardware protection cannot eliminate all screen cracking signals; third, hardware modification is costly.
[0042] In view of the above technical shortcomings, an embodiment of the present invention provides a screen tearing protection method based on active frame loss. This method is a software method and does not require hardware changes, and can overcome the shortcomings of hardware protection methods.
[0043] Figure 4 Flow chart of the software cracked screen protection algorithm according to an embodiment of the present invention, as shown in FIG. Figure 4 The normal FFmepg streaming process is: Figure 4 Experiments have found that when the input source is unstable, the video or audio packets collected by the SDI or HDMI input source will be erroneous or lost. After these erroneous video or audio packets arrive at the FFmepg streaming end data storage queue, the FFmpeg streaming end cannot encapsulate and stream these erroneous data packets, causing the data packets to be accumulated in the FFmepg streaming end data storage queue Q. video or Q audio In addition, the more unstable the input source signal is, the faster the FFmpeg streaming end data packets will accumulate. Therefore, the software can store the video packets in the queue Q according to the FFmpeg streaming end. video Or audio packet storage queue Q audio The packet backlog rate is used to identify the screen tearing signal. The software is implemented as follows:
[0044] S201. Calculate Q for the time period from t to t1 video and Q audio Packet backlog rate V videoand V audio ;
[0045] S202. Based on the backlog rate V of the data packets video and V audio Determine whether a cracked screen signal occurs;
[0046] S203. Lost FFmpeg streaming end Q on software video or Q audio Backlog of video or audio packets;
[0047] S204. The software sends a message to the acquisition end to let the SDI or HDMI input source re-acquire the video and audio data.
[0048] In step S201, Figure 4 In ⑦, the software loops to obtain the FFmpeg streaming end video packet storage queue Q at time t video The number of packets is S video {t}, the FFmpeg streaming end video packet storage queue Q at time t1 video The number of packets is S video {t1}, then the backlog rate of video packets at the FFmpeg streaming end during the time period from t to t1 is:
[0049] V video =(|S video {t1}-S video {t}|) / (|t1-t|);
[0050] Similarly, the software obtains the FFmpeg streaming end audio packet storage queue Q at time t audio The number of packets is S audio {t}, at time t1, the FFmpeg streaming end audio packet storage queue Q audio The number of packets is S audio {t1}, then the backlog rate of audio packets on the FFmpeg streaming end during the time period from t to t1 is:
[0051] V audio =(|S audio {t1}-S audio {t}|) / (|t1-t|)
[0052] In step S202, experiments have shown that under normal circumstances, S video {t} and S audio {t} is basically equal to 0; when the SDI or HDMI input source is slightly unstable, S video {t} and S audio {t} will change slightly and quickly clear to zero, V video or V audioVery small; when the SDI or HDMI input source is seriously interfered with and a cracked signal occurs, S video {t} and S audio {t} will increase rapidly, V video or V audio Therefore, if Figure 4 In ⑧, we can use V video or V audio The M value can be adjusted flexibly according to the actual application scenario to improve the sensitivity of the software in identifying the cracked screen signal.
[0053] In step S203, if Figure 4 ⑨ in the figure, that is, discarding the erroneous data and erroneous frames caused by the screen tearing signal. In the embodiment of the present invention, the backlog of video and audio packets is discarded by directly releasing the memory.
[0054] In step S204, if Figure 4 In step ⑩, after the FFmpeg streaming end pushes the normal video and audio data to the streaming media server, the live broadcast picture on the streaming end can be restored from the cracked screen to the normal picture.
[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A screen tearing protection method based on active frame loss, characterized in that: include: Obtain the backlog rate of the video packet storage queue and the audio packet storage queue of the streaming end; The streaming end determines whether a screen cracking signal occurs according to the backlog rate; After the streaming end identifies the screen tearing signal, it actively discards the erroneous data and frames caused by the screen tearing; After the push end actively completes frame loss, it sends a notification message to the acquisition end, and the acquisition end re-acquires the multimedia data stream; The streaming end streams the video and audio data to a streaming media server; The calculation method of the backlog rate of the video packets at the streaming end in the time period from t to t1 is: V video =(|S video {t1}-S video {t}|) / (|t1-t|) Among them, V video Indicates the backlog rate of video data packets. The software loops to obtain the video packet storage queue Q at the streaming end at time t. video The number of packets is S video {t}, the video packet storage queue Q at the streaming end at time t1 video The number of packets is S video {t1}; The calculation method of the backlog rate of the audio packets at the streaming end in the time period from t to t1 is: 5 audio =(|S audio {t1}-S audio {t}|) / (|t1-t|) Among them, V audio Indicates the backlog rate of audio data packets. The software loops to obtain the audio packet storage queue Q at the streaming end at time t. audio The number of packets is S audio {t}, the video packet storage queue Q at the streaming end at time t1 audio The number of packets is S audio {t1}.
2. The screen tearing protection method based on active frame loss according to claim 1, characterized in that: The step of determining whether a screen-breaking signal occurs according to the backlog rate includes: setting a backlog rate M of video or audio data packets corresponding to the screen-breaking signal, when the V video or the V audio When it is greater than M, it is considered that a screen cracking signal occurs.
3. The screen tearing protection method based on active frame loss according to claim 1, characterized in that: The erroneous data and erroneous frames generated by screen tearing include or backlog video or audio data packets.
4. The screen tearing protection method based on active frame loss according to claim 1, characterized in that: The active frame loss is achieved by releasing the memory that has accumulated error data and error frames.
5. The screen tearing protection method based on active frame loss according to claim 1, characterized in that: The streaming end uses FFmpeg software streaming technology to perform streaming.
6. The screen tearing protection method based on active frame loss according to claim 1, characterized in that: The acquisition end re-acquires video and audio data, and the input source of the acquisition end includes an SDI input source and an HDMI input source.
7. The screen tearing protection method based on active frame loss according to claim 1, characterized in that: The streaming end encapsulates the video encoding package and the audio encoding package into a specified format according to the set streaming protocol and streams them to the specified streaming media server.
8. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory is used to store at least one program, and the processor is used to load the at least one program to execute the screen tearing protection method based on active frame loss as described in any one of claims 1 to 7.
9. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to execute the screen tearing protection method based on active frame loss as described in any one of claims 1 to 7 when executed by the processor.
Citation Information
Patent Citations
Method and device for reducing live broadcasting time delay and standstill as well as live broadcasting system
CN106713947A
Mobile terminal audio and video synchronization method and device, equipment and storage medium
CN113225598A
