A video ultra-low latency edge device solution and system based on WebRTC

Abstract

The present invention relates to the technical field of video acquisition, encoding and transmission, and provides a solution for ultra-low latency edge devices for video based on WebRTC, including the following steps: S1: Setting a signaling service for forwarding signaling information between a sender and a receiver, wherein the signaling information includes session control information, encoding and decoding information and other control information for interaction between the sender and the receiver; S2: The sender acquires video information, performs hardware encoding on the video information, and transmits the video information to the receiver after encoding; S3: The receiver uses hardware accelerated decoding for the received video information, and hardware accelerated rendering and display. The above technical solution relates to a solution for realizing ultra-low latency in edge devices. Specifically, a solution for ultra-low latency of less than 100 milliseconds is implemented on a low-configuration edge device based on the WebRTC architecture.

Description

Technical Field

[0001] The present invention relates to the technical field of video acquisition, coding and transmission, and in particular to a video ultra-low latency edge device solution and system based on WebRTC. Background Art

[0002] The WebRTC (Web Real-Time Communication) protocol is a technical standard widely used in video calls, video conferencing, real-time monitoring and other fields. The entire processing flow includes acquisition, encoding, transmission, decoding, rendering and display, etc. The processing flow of the native WebRTC system uses CPU processing, which has high requirements on the machine configuration used. Based on cost considerations, there is currently a demand for low-latency video transmission in low-configuration edge devices, such as video transmission in monitoring systems. However, this type of edge device has the following problems: (1) Insufficient edge hardware resources lead to high latency The hardware configuration of edge devices is poor. They use a 64-bit system (aarch64) based on the ARM architecture and have only 8 processors, so their processing power is relatively limited.

[0003] In the WebRTC processing flow, data format conversion in the acquisition link, timely processing of video with low-latency encoding, and maintenance of high frame rate encoding all have high requirements for computing resources. It is difficult to meet the low-latency requirements by relying solely on the CPU of the edge device to run WebRTC sending and receiving, and the overall delay exceeds 300 milliseconds. For example, in a video surveillance scenario, high latency will cause the surveillance screen to lag behind, making it impossible to capture changes in the on-site situation in a timely manner, affecting the effectiveness of surveillance.

[0004] (2) Audio and video synchronization mechanism causes video decoding delay In applications such as surveillance scenarios and AI video recognition, in many cases only video needs to be transmitted without audio.

[0005] In normal use, video and audio are transmitted at the same time, which not only increases the bandwidth, but also causes the video to wait for the audio to arrive before decoding in order to ensure the synchronization of sound and picture during playback. This waiting process increases the delay of video playback, affecting the timeliness of video data being quickly presented to users, and is not conducive to the rapid display of real-time monitoring images and event response.

[0006] (3) Software rendering performance bottleneck causes rendering delay Most existing WebRTC implementations rely on software for video rendering. Although this approach has good compatibility and flexibility, the software rendering performance is insufficient when rendering high-resolution and high-frame-rate videos, and it cannot quickly process large amounts of graphic image data, resulting in high rendering delays and unsmooth video playback. In high-definition video surveillance or high-quality video conferencing scenarios, this affects the viewing experience and the timeliness of information delivery.

[0007] (4) The native video playback delay setting limits the realization of ultra-low latency The native webrtc sets a minimum delay for video playback, which becomes an obstacle in the pursuit of ultra-low latency scenarios. This minimum delay increases the delay of actual video playback, which is contrary to the ultra-low latency requirement, limiting the application effect of WebRTC in scenarios with strict real-time requirements (such as remote real-time control, live broadcast of competitive events, etc.), and cannot fully realize its potential for fast transmission and video playback. Summary of the invention

[0008] In view of the above problems, the purpose of the present invention is to provide a video ultra-low latency edge device solution and system based on WebRTC, which involves a solution for realizing ultra-low latency in edge devices. Specifically, it is to realize an ultra-low latency solution of less than 100 milliseconds on a low-configuration edge device based on the WebRTC architecture.

[0009] The above-mentioned object of the present invention is achieved through the following technical solutions: A video ultra-low latency edge device solution based on WebRTC, including the following steps: S1: Setting a signaling service for forwarding signaling information between a sending end and a receiving end, wherein the signaling information includes session control information, codec information and other control information for interaction between the sending end and the receiving end; S2: The sending end collects video information, performs hardware encoding on the video information, and transmits the video information to the receiving end after the encoding is completed; S3: The receiving end uses hardware accelerated decoding and hardware accelerated rendering and display for the received video information.

[0010] Furthermore, in step S2, the transmitting end collects video information and performs hardware encoding on the video information, specifically: The transmitting end uses a camera to collect the original video data, a video acquisition module obtains video frames from the camera, and sends the video frames to a video encoding module, and the collected video data is stored in a YUV format; In order to run WebRTC in the edge device, the video encoding module uses the RK3588 chip to convert the data format of the collected video data into the I420 format required for encoding; The video data is hardware encoded on the RK3588 chip to reduce the bandwidth and storage space required for transmitting the video data.

[0011] Further, in step S2, after the encoding is completed, the video information is transmitted to the receiving end, specifically: After the encoding of the video information is completed, the encoded video information is encapsulated into a real-time transport protocol RTP packet, and the real-time transport protocol RTP packet is sent to the receiving end through the user datagram protocol UDP.

[0012] Furthermore, the ultra-low latency edge device solution for video based on WebRTC also includes: Using the frame rate priority mode in WebRTC, priority is given to ensuring high frame rate encoding when the sender performs hardware encoding, high frame rate transmission when the sender transmits the video information to the receiver, and high frame rate decoding when the receiver decodes the video information.

[0013] Further, in step S3, the receiving end uses hardware accelerated decoding and hardware accelerated rendering and display for the received video information, specifically: The receiving end parses the received protocol data and extracts video related information for parsing the video information; Then, the video is decoded. The receiving end uses a video decoder to restore the compressed video information encoded at the sending end to the original video frame, and in the edge device, the decoding process is accelerated by hardware to increase the decoding speed; The decoded video frames of the video information are transmitted to a rendering module, and the rendering module uses hardware acceleration to display the video frames on a screen.

[0014] Furthermore, the ultra-low latency edge device solution for video based on WebRTC also includes: Adding an audio flag to the session description protocol SDP of the signaling service; When the session description protocol SDP contains the audio flag, the sending end no longer sends audio information, the receiving end no longer receives the audio information, and if there is audio information, it is directly discarded, the video information is directly decoded, and rendered immediately after decoding is completed.

[0015] Furthermore, the ultra-low latency edge device solution for video based on WebRTC also includes: Set the minimum and maximum values ​​of the actual delay of the video playback of the video information, and set the playback waiting delay to 0 under ultra-low delay conditions; The minimum value is the shortest time interval allowed from the completion of video data reception to the start of operations including decoding and rendering when all conditions are in a relatively ideal state during video playback. The maximum value is an upper limit of video playback delay, which exists to limit the range that video playback delay will not exceed under any adverse circumstances; Setting the playback wait delay to 0 is to artificially and forcibly set the playback wait time to 0, which means that when the video information arrives at the receiving end, there will no longer be any deliberately reserved waiting buffer time, and the video frame will be immediately sent for decoding and rendering operations.

[0016] A video ultra-low latency edge device solution system based on WebRTC for executing the above-mentioned video ultra-low latency edge device solution based on WebRTC, comprising: A signaling service module is used to set a signaling service for forwarding signaling information between a sender and a receiver, wherein the signaling information includes session control information, codec information and other control information for interaction between the sender and the receiver. A sending module, used for collecting video information, performing hardware encoding on the video information, and transmitting the video information to the receiving end after encoding is completed; The receiving module is used to adopt hardware accelerated decoding and hardware accelerated rendering and display for the received video information.

[0017] A computer device comprises a memory and one or more processors, wherein the memory stores computer codes, and when the computer codes are executed by the one or more processors, the one or more processors execute the above method.

[0018] A computer-readable storage medium stores computer codes. When the computer codes are executed, the above method is executed.

[0019] The beneficial effects of the present invention are: (1) Excellent low-latency performance: It successfully breaks through traditional limitations on low-configuration edge devices and achieves remarkable low-latency video transmission based on WebRTC. The minimum latency can be controlled at 60 milliseconds. This remarkable achievement enables video data to be transmitted between the sender and the receiver in near real time. Whether it is the rapid detection of key events in remote monitoring scenarios or the realization of instant information exchange in real-time interactive applications, it greatly improves user experience and business processing efficiency, and opens up new possibilities for application areas with extremely high time sensitivity requirements.

[0020] (2) Adapt to the efficient use of low-configuration edge devices: fully tap the hardware potential of edge devices and optimize the design specifically for low-configuration edge devices. By cleverly using hardware resources such as the RK3588 chip to perform key operations such as data format conversion and encoding, the data processing speed is greatly improved and the overall computing burden of the device is reduced. This enables edge devices that originally performed poorly in video processing due to limited configuration to undertake complex video transmission tasks, achieve efficient resource utilization, and avoid the cost of large-scale hardware upgrades or equipment replacements due to insufficient device performance, providing enterprises and users with a cost-effective solution for video applications in the field of edge computing.

[0021] (3) Flexible and intelligent audio control: Innovatively introduce audio transmission switch settings to flexibly adjust audio transmission strategies according to actual application needs. In scenarios where audio is not required, such as some simple monitoring applications or interactive scenarios that only focus on video content, the audio transmission link can be decisively closed, thereby saving a lot of network bandwidth, device computing resources, and transmission time. This not only helps to further reduce the overall delay of video transmission, but also allows limited resources to be more concentrated on video data processing and transmission, ensuring that the core performance of video transmission is optimized to the greatest extent. At the same time, it also provides convenience for independent control of audio and video, meeting the needs of diverse application scenarios.

[0022] (4) Accurate playback delay optimization: By accurately setting the playback delay to zero, unnecessary waiting time during video playback is completely eliminated. Once the video frame arrives at the receiving end, it is immediately sent to the decoding and rendering process without any buffering waiting, which greatly speeds up the speed of video presentation. This extreme optimization of playback delay has achieved an unprecedented level of synchronization between video playback and real-life scenes. It has immeasurable value in application scenarios with extremely high real-time requirements such as remote real-time control and live broadcast of competitive events. It truly realizes an immersive video viewing experience for users and significantly improves the competitiveness of video transmission systems in the field of ultra-low latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1This is an overall flow chart of the video ultra-low latency edge device solution implemented based on WebRTC in the present invention; Figure 2 This is a detailed flow chart of the video ultra-low latency edge device solution implemented based on WebRTC in the present invention; Figure 3 This is an overall structural diagram of the video ultra-low latency edge device solution system implemented based on WebRTC in the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0026] First embodiment like Figure 1 and 2 As shown, this embodiment provides a video ultra-low latency edge device solution based on WebRTC, including the following steps: S1: Setting a signaling service for forwarding signaling information between a sending end and a receiving end, wherein the signaling information includes session control information, encoding and decoding information, and other control information for interaction between the sending end and the receiving end.

[0027] In this embodiment, the signaling service is used to establish and manage the connection between the sender and the receiver. The signaling information includes session control information, codec information, and other control information. The signaling service negotiates the video codec, network parameters, and other session-related settings by exchanging SDP (Session Description Protocol) information.

[0028] The signaling service plays a key role as the "command center" in the entire WebRTC-based video transmission system. Its main function is to forward various signaling information between the sender and the receiver, which covers multiple aspects such as session control information, codec information, and other control information. Through the transmission and interaction of this information, the signaling service can effectively establish a stable and reliable connection between the sender and the receiver, and comprehensively manage and coordinate the entire connection process.

[0029] For example, when initiating a video conference, the signaling service first receives a connection request from the sender, which contains preliminary information about the session, such as the sender's identity, the desired video quality parameters, etc. The signaling service then accurately forwards this information to the receiver, who responds based on its own conditions (such as device performance, network conditions, and user settings), and sends the response information back to the sender through the signaling service. This back-and-forth information exchange process is completed under the careful organization and scheduling of the signaling service, and ultimately achieves the successful establishment of a connection between the sender and the receiver.

[0030] 1. Session Control Information The session control information is like a "navigator" for video communication sessions. It dominates the coordination and management of key stages such as the start, progress and termination of the entire video interaction process. For example, in the session initiation phase, the session control information clarifies the sender's intention to initiate a connection request and carries parameters such as the unique identifier of the request to establish a session and the priority setting of the initiator. This information helps the receiver quickly identify and process the request and determine whether to accept the session connection. During the session, the session control information is also responsible for handling the transmission and execution of operation instructions such as session pause, resume, and switch video sources. For example, when a user needs to switch the camera view or pause the video playback in a video conference to focus on audio communication, the corresponding session pause or switch instructions will be communicated and responded between the sender and the receiver through the session control information carried by the signaling service, ensuring that the devices of both parties can perform these operations synchronously and maintain the continuity and consistency of the session. At the end of the session, the session control information is responsible for sending a request to terminate the session and ensuring the orderly release of resources on both sides, avoiding problems such as resource waste or data loss caused by sudden session interruption.

[0031] 2. Video Codec Negotiation In the field of video communication technology, multiple video codecs coexist, such as VP8, VP9, ​​H.264, H.265, etc., each of which has unique performance characteristics. VP8 is widely used in many scenarios with high adaptability requirements for network environments due to its good open source and real-time advantages; H.265 has excellent performance in balancing image quality and data volume with higher compression efficiency, but it has certain requirements for device computing power. The signaling service uses SDP information exchange to promote the negotiation process of video codecs. The sender and the receiver list the types of video codecs they support and related characteristic parameters in detail in the SDP information. The signaling service carefully selects the most suitable video codec from the list of codecs supported by both parties based on the real-time status of the current network (such as bandwidth fluctuation range, packet loss rate, etc.), the actual performance level of the devices at both ends (such as CPU computing speed, GPU graphics processing capability, etc.), and the specific requirements of the application scenario for video quality (such as the requirements for details in monitoring scenarios and the emphasis on fluency in live broadcast scenarios). For example, in remote monitoring scenarios where network bandwidth is relatively limited but real-time requirements are extremely high, the signaling service may tend to choose VP8 as the video codec to ensure that video data can be transmitted quickly and the monitoring screen can be presented in a timely manner; while in high-definition video live broadcast scenarios where network conditions are favorable and equipment performance is strong, H.265 may become the first choice due to its excellent compression ratio and image quality, thus bringing a clearer and more realistic visual experience to the audience. Through this scientific and reasonable negotiation mechanism, it is ensured that the sender and the receiver reach a high degree of agreement in the use of video codecs, laying a solid foundation for subsequent efficient video encoding, transmission and decoding.

[0032] 3. Network Parameter Negotiation Network parameters play a decisive role in the quality and efficiency of video transmission, including core elements such as network bandwidth, transmission protocol (UDP or TCP), IP address, and port number. The adequacy of network bandwidth is directly related to the quantity and quality of video data that can be transmitted. High bandwidth can provide strong support for high-resolution and high-frame-rate video transmission; the choice of transmission protocol profoundly affects the reliability and real-time performance of video transmission. The UDP protocol stands out in video transmission scenarios with strict real-time requirements due to its low latency and high efficiency, although its data integrity guarantee is relatively weak; the TCP protocol takes data reliability as the primary consideration. Although the latency is slightly higher, it can ensure that the data is delivered accurately. In the SDP information exchange process, the sender incorporates its own estimated network parameter recommendations into the SDP information and passes it to the receiver, including the recommended transmission protocol, its own IP address and port number information, and the approximate estimated network bandwidth data. The receiving end comprehensively evaluates and fine-tunes these network parameters based on the actual conditions of its own network (such as real-time available bandwidth, network stability, and latency, etc.) and personalized expectations for video quality, and uses signaling services to feed back the finalized network parameters to the sending end. Through such a rigorous negotiation mechanism, both parties can reach an optimal consensus on the network parameters, ensuring that video data can flow unimpeded in the most suitable network environment, significantly improving the quality and stability of video transmission, and effectively reducing the probability of problems such as freezes, delays, or image quality impairments caused by network parameter mismatches.

[0033] 4. Other session-related settings negotiation In addition to the negotiation of video codecs and network parameters, the signaling service is also deeply involved in the negotiation of a series of other session-related settings. For example, in the selection of audio codecs (if video communication involves audio transmission), it is necessary to fully consider factors such as audio quality requirements, device audio processing capabilities, and network bandwidth allocation. If in a remote teaching scenario, the teacher's voice needs to be clearly and accurately conveyed, the signaling service will select an audio codec that can minimize bandwidth usage while ensuring audio quality based on the audio codec capabilities of the sender and receiver devices and the network conditions, such as Opus and other high-efficiency audio encoding formats. In addition, the encryption method of the session is also one of the important negotiation contents. In scenarios such as corporate confidential video conferencing and video communication in the financial field that require extremely high security and privacy, the signaling service will actively coordinate the sender and receiver to jointly determine the use of high-strength encryption algorithms, such as encryption schemes based on the TLS (Transport Layer Security) encryption protocol, to effectively ensure the security and confidentiality of video and audio data during the entire transmission process, and effectively prevent security risks such as illegal theft, tampering, or malicious monitoring of data. At the same time, the signaling service will also accurately negotiate and set the direction of the media stream (one-way transmission or two-way communication) according to the needs of the specific application scenario. In remote monitoring scenarios, one-way video streaming from the monitoring end to the monitoring center is usually the main method; in video conferencing, remote collaboration and other scenarios, it is necessary to support two-way video and audio interaction to achieve full communication and interaction between participants.

[0034] S2: The sending end collects video information, performs hardware encoding on the video information, and transmits the video information to the receiving end after the encoding is completed.

[0035] Video information is sent directly from the sender to the receiver without passing through other services, reducing the transmission path.

[0036] In this embodiment, step S2 is specifically as follows: The transmitting end uses a camera to collect the original video data, and the video acquisition module obtains video frames from the camera, sends the video frames to the video encoding module, and the collected video data is stored in YUV format. In order to run WebRTC in the edge device, the video encoding module uses the RK3588 chip to convert the data format of the collected video data into the I420 format required for encoding, thereby improving the data format conversion speed. The video data is hardware encoded on the RK3588 chip to reduce the bandwidth and storage space required for transmitting the video data, and to improve the speed of video encoding. After the encoding of the video information is completed, the encoded video information is encapsulated into a real-time transport protocol RTP packet, and the real-time transport protocol RTP packet is sent to the receiving end via the user datagram protocol UDP.

[0037] In this embodiment, the above process may include the following solutions: 1. Video Collection The transmitter uses the camera as the source acquisition device of the video data to start the entire video transmission link. The camera is responsible for capturing the optical image information in the real scene and converting it into raw video data in the form of electrical signals. The video acquisition module is responsible for the key task of obtaining these video frames from the camera and transmitting them to the subsequent video encoding module in a timely manner. In this process, the collected video data is stored in the YUV format. The YUV format is a color space representation method widely used in the field of video processing. It stores brightness information (Y) and chrominance information (U and V) separately. The advantage of this format is that it can flexibly process brightness and chrominance information according to different needs in the subsequent video processing process. For example, when compressing and encoding the video, different encoding strategies can be used to process the Y, U, and V components respectively based on the characteristics that the human eye is more sensitive to brightness information, thereby improving the encoding efficiency while ensuring the video quality.

[0038] 2. Video encoding module based on RK3588 chip 1. Data format conversion In order to meet the specific requirements of subsequent video encoding, the video encoding module uses the powerful processing power of the RK3588 chip to convert the collected YUV format video data into the I420 format required for encoding. I420 is a specific sub-format of the YUV format. While ensuring video quality, it is more conducive to the efficient execution of the video encoding algorithm. The RK3588 chip can quickly and accurately complete this data format conversion operation through its specially designed hardware circuit and optimized algorithm instruction set, which greatly improves the speed of data format conversion compared to traditional software processing methods. This not only reduces the pre-processing time of data before encoding, improves the real-time performance of the entire video transmission system, but also provides a data foundation for subsequent encoding links that is more in line with encoding specifications and performance requirements.

[0039] 2. Hardware encoding process After completing the data format conversion, the RK3588 chip continues to play its core advantage of hardware encoding to encode the video data. Hardware encoding is achieved by using a dedicated video encoding engine inside the chip. These encoding engines are deeply optimized for common video encoding standards (such as H.264, H.265, etc.), and can compress and encode video data with extremely high efficiency. Through hardware encoding, the video data is greatly compressed while ensuring a certain video quality, thereby significantly reducing the bandwidth and storage space required to transmit video data. For example, under the same video content and image quality requirements, the amount of video data after hardware encoding may be only a fraction of the original data or even smaller, which is particularly important for edge device application scenarios with limited network bandwidth resources. At the same time, the speed of hardware encoding has made a qualitative leap compared to software encoding, which can meet the strict requirements for encoding efficiency during real-time video transmission, ensure that video data can be encoded and sent out in a timely manner, and avoid problems such as video freezes or transmission interruptions caused by encoding delays.

[0040] 3. Video Data Packaging and Transmission After the video information is encoded successfully, the encoded video information is encapsulated into a real-time transport protocol (RTP) packet. The RTP packet is a network data packet format designed specifically for real-time transmission of audio and video data. It contains important information such as timestamp, sequence number, and load type in the packet header. This information plays a key guiding role for the receiving end when performing video decoding, synchronous playback, and other operations. For example, the timestamp information can help the receiving end accurately restore the playback order and time interval of the video frames to ensure the smoothness of video playback; the sequence number is used by the receiving end to detect whether the data packet is lost and to handle the lost data packet accordingly. The video data encapsulated into the RTP packet is further sent to the receiving end through the user datagram protocol (UDP). The UDP protocol is an ideal choice for real-time video transmission with its low latency and high efficiency. Compared with the TCP protocol, the UDP protocol does not need to establish complex connections and maintain connection status. There is no cumbersome confirmation and retransmission mechanism during data transmission. Therefore, the RTP packet can be sent to the receiving end quickly, minimizing the delay of video data during network transmission. Although the UDP protocol does not guarantee absolute reliability of data transmission and packet loss may occur, it can still achieve stable video transmission in most network environments by combining the fault-tolerant design of video encoding and the packet loss handling mechanism at the receiving end, meeting the primary requirement of video real-time performance.

[0041] S3: The receiving end uses hardware accelerated decoding and hardware accelerated rendering and display for the received video information.

[0042] In this embodiment, step S3 is specifically as follows: The receiving end parses the received protocol data and extracts the video-related information for parsing the video information. Then, video decoding is performed. The receiving end uses a video decoder to restore the compressed video information encoded at the sending end to the original video frame, and in the edge device, hardware acceleration is used for the decoding process to increase the decoding speed. The decoded video frame of the video information is transmitted to the rendering module, and the rendering module uses hardware acceleration to display the video frame on the screen. This improves the rendering speed during playback, ensures the smoothness and low latency of video playback, and ensures that the video frame can be played in time to prevent freezes.

[0043] In this embodiment, the above process may include the following solutions: 1. Protocol data analysis The first task faced by the receiver is to parse the received protocol data. These protocol data contain rich information, including not only the video data itself, but also various control information, metadata, etc. During the parsing process, the receiver will accurately extract key information directly related to the video information parsing from the complex protocol data structure. For example, by parsing the Session Description Protocol (SDP) information, important parameters such as the type of video codec, video resolution, and frame rate are obtained. These parameters will provide necessary guidance for the subsequent video decoding process. At the same time, information such as timestamps and sequence numbers are also extracted to ensure the correct order and synchronous playback of video frames after decoding. This parsing process is like disassembling a letter with detailed instructions. The receiver extracts the key information about the video one by one according to specific rules and formats, laying the foundation for subsequent processing steps.

[0044] 2. Video Decoding After successfully extracting the video-related information, the receiving end enters the video decoding phase. At this point, a special video decoder is used to process the compressed video information that has been encoded at the sending end and restore it to the original video frame. Considering that the computing resources of the edge device may be relatively limited, in order to improve the decoding speed, the receiving end makes full use of hardware acceleration technology. Hardware acceleration performs decoding operations by calling special video decoding hardware modules (such as GPU or specific video decoding chips) in the edge device. These hardware modules are optimized for common video encoding standards (such as H.264, H.265, etc.) and have efficient decoding algorithms and parallel processing capabilities. Compared with relying solely on software decoding, hardware acceleration can significantly improve decoding efficiency and greatly shorten the time required for decoding. For example, when processing high-resolution, high-frame-rate video data, hardware acceleration can increase the decoding speed by several times or even more, effectively avoiding the video freeze caused by slow decoding speed, and ensuring that the video can be smoothly processed in the subsequent process.

[0045] 3. Video Rendering and Display When the video frame is successfully decoded, it is transmitted to the rendering module. The rendering module also uses hardware acceleration technology to display the video frame on the screen. In this process, hardware acceleration is mainly reflected in the rapid drawing and processing of graphics images. For example, the powerful graphics processing capabilities of the GPU are used to quickly convert the pixel data in the video frame into a format suitable for screen display and draw it in the correct position and order. Through hardware accelerated rendering, the rendering speed during playback can be greatly improved, ensuring that the video frame can be presented on the screen in a timely and accurate manner, preventing the occurrence of stuttering, and providing users with a smooth and low-latency video viewing experience. Whether watching the human image in a high-definition video conference or viewing real-time video images in a monitoring scene, this efficient rendering and display mechanism can ensure the smoothness and real-time nature of the video image, allowing users to experience visual effects that are almost synchronized with the actual scene.

[0046] Furthermore, in this embodiment, the frame rate priority mode in WebRTC can also be used. Under certain network conditions, high frame rate encoding when the sender performs hardware encoding, high frame rate transmission when the sender transmits the video information to the receiving end, and high frame rate decoding when the receiving end decodes the video information are prioritized to maintain overall low latency and smooth playback.

[0047] In this embodiment, the above process may include the following solutions: 1. Basic Concepts of Frame Rate Priority Mode Frame rate priority mode is an important strategy in WebRTC. It prioritizes the frame rate of the video when there are certain limitations on network bandwidth and other conditions. Frame rate refers to the number of frames per second in the video, measured in fps (frames per second). For example, common video frame rates include 24fps, 30fps, and 60fps. The higher the frame rate, the smoother the video image, and the more coherent the dynamic image seen by the human eye.

[0048] In frame rate priority mode, the system will prioritize resource allocation under certain network conditions to ensure that the video can maintain a high frame rate in all links of encoding, transmission and decoding. This is like scheduling in a traffic system with limited resources to ensure the fast passage (high frame rate) of vehicles (video frames).

[0049] 2. Frame rate priority mode in each link 1. Hardware encoding Coding strategy adjustment When hardware encoding is performed on the sender, the frame rate priority mode means that the encoder will try to maintain a high frame rate encoding as much as possible based on the network conditions and device performance. For example, when the network bandwidth fluctuates but can still support a certain amount of data transmission, the encoder will prioritize reducing the encoding of details for each frame (such as reducing the resolution or using a simpler encoding algorithm to process the picture details), and use more resources to ensure that more video frames can be encoded per unit time.

[0050] Hardware resource allocation Hardware encoding devices such as the RK3588 chip will allocate more computing resources to maintaining the frame rate. This may involve the time allocation and task priority adjustment of the computing units inside the chip (such as the CPU core, GPU core, or a dedicated video encoding unit) for video frame processing. For example, reduce the time used for other non-critical video processing tasks (such as color correction, etc.), and use more time to quickly process video frames to ensure that more video frames can be encoded per unit time, thereby maintaining a high frame rate.

[0051] (II) Transmission link Protocol Adaptation and Transmission Strategy When transmitting video information from the sender to the receiver, the frame rate priority mode affects the selection of transmission protocols and transmission strategies. For example, when transmitting based on the UDP protocol, the system will send video frames as quickly as possible within a certain acceptable range of packet loss rate to maintain a high frame rate. At the same time, the encapsulation and sending order of the real-time transport protocol (RTP) packets will also be optimized, giving priority to sending the latest video frames to ensure that the receiver can obtain enough video frames in time to maintain high frame rate playback.

[0052] Network bandwidth utilization In the case of limited network bandwidth, the frame rate priority mode will allocate bandwidth resources reasonably. For example, it may reduce the bandwidth used to transmit audio data (if audio is not the focus of the content) or other auxiliary data (such as video metadata), leaving more bandwidth for the transmission of video frames to ensure that the video frame rate is not significantly reduced due to bandwidth limitations. This is like in the "pipeline" of network bandwidth, giving priority to ensuring that the transportation channel of the "main cargo" of video frames is unobstructed.

[0053] (III) Decoding Hardware acceleration optimization When decoding video information at the receiving end, hardware acceleration technology plays a key role in achieving high frame rate decoding. The hardware device at the receiving end (such as a GPU or a dedicated video decoding chip) will optimize the decoding of high frame rate video streams. For example, by processing the decoding tasks of multiple video frames in parallel, the high-speed computing power of the hardware is used to complete the decoding operations of multiple video frames in a short time, ensuring that the video can be played smoothly at a high frame rate.

[0054] Cache and pre-decoding strategies The receiving end may also adopt caching and pre-decoding strategies to support high frame rate decoding. For example, a video frame buffer of appropriate size is established. When the network condition is good, a part of the video frames are cached in advance, and they are decoded quickly when needed. In this way, when there is a short fluctuation in the network or a slight delay in the decoding speed, the high frame rate of the video can still be guaranteed, just like reserving some "spare video frame resources" in advance to deal with emergencies.

[0055] In this embodiment, it also includes: When a lower delay is selected, in order to reduce bandwidth and increase the playback speed, an audio flag is added to the session description protocol SDP of the signaling service; when the audio flag is included in the session description protocol SDP, the sending end no longer sends audio information to reduce bandwidth, and the receiving end no longer receives the audio information. If there is audio information, it is directly discarded, and the video information is directly decoded, and rendered immediately after decoding is completed, so as to improve the real-time performance of the player playing the video. For native webrtc, the audio is initialized when the program is created, which may cause the program to crash when the acquisition device does not have an audio acquisition function. The present invention optimizes this part of the content, adds a control flag to the SDP, and initializes the audio processing and transmits the audio only when the audio is needed.

[0056] This embodiment also includes: in order to further reduce the overall delay of video transmission, the minimum and maximum values ​​of the actual delay of video playback of the video information are set, and the playback waiting delay is set to 0 under ultra-low delay conditions. To ensure that the video frames are decoded and rendered as soon as possible, the delay of video transmission is significantly reduced. The minimum value is the shortest time interval allowed for the video to complete data reception and start operations including decoding and rendering when all conditions are in a relatively ideal state during video playback. The maximum value is an upper limit of the video playback delay, which exists to limit the range that the video playback delay will not exceed under any unfavorable circumstances. Setting the playback waiting delay to 0 is to artificially set the playback waiting time to 0, which means that when the video information arrives at the receiving end, there will no longer be any deliberately reserved waiting buffer time, and the video frame will be sent immediately for decoding and rendering operations.

[0057] In this embodiment, the above process may specifically include the following explanations: 1. Minimum actual delay of video playback Latency benchmark under ideal conditions During video playback, the minimum value represents an ideal performance indicator. The ideal conditions here cover many aspects, including the network environment being in the best state, that is, having high bandwidth, extremely low packet loss rate, stable network connection, and no network jitter; at the same time, the performance of the device is also at its peak level, for example, the hardware resources such as CPU and GPU have enough spare power to process video data quickly and smoothly, and there are no other high-load tasks in the system that interfere with the video processing process. Under this ideal condition, the shortest time interval allowed from the completion of video data reception to the start of decoding, rendering and other operations is defined as the minimum value. For example, in an internally built high-speed LAN environment, when using a high-performance server to play a video that has been cached locally, the video playback delay can theoretically be close to this minimum value. The setting of this minimum value provides a goal for the entire video playback system, which motivates the system to optimize in various links to get as close to this ideal low-latency state as possible.

[0058] Guiding role for ultra-low latency transmission Clarifying the minimum value helps to specifically address factors that may affect the minimum latency during system design and optimization. For example, in the optimization of the network protocol stack, the time consumption from data reception to the decoding stage can be reduced by streamlining the data packet header processing process and adopting efficient cache management strategies; at the device hardware level, the driver of the video processing chip (such as the RK3588 chip) can be optimized to improve its interaction efficiency with the operating system and application programs, ensuring that video data can be quickly passed to the decoding and rendering modules. By continuously approaching this minimum value, the video transmission system can achieve better performance in ultra-low latency transmission and meet the needs of application scenarios with extremely high real-time requirements, such as real-time video assistance in telemedicine surgery. Every millisecond of latency reduction may have a positive impact on the success of the operation.

[0059] 2. Maximum actual delay of video playback Upper limit of delay to deal with adverse situations Corresponding to the minimum value, the maximum value is an upper limit of the video playback delay. Its core significance lies in ensuring the stability and availability of video playback. In actual video playback scenarios, it is inevitable to encounter interference from various unfavorable factors. From the network perspective, there may be network congestion, large bandwidth occupation, frequent packet loss, or unstable network signal leading to obvious jitter; from the device perspective, it may be due to running multiple high-load programs at the same time, causing its performance to be dispersed and the ability to process video data to decrease. In these cases, the video playback delay may increase, but by setting the maximum value, it can be ensured that even if these bad situations occur, the video playback delay will not increase indefinitely. For example, when watching mobile videos outdoors, the network signal may be good and bad, and the device may be heavily loaded due to too many background programs, but as long as the video playback delay is always within this maximum value, users can generally watch the video normally. Although there may be some slight freezes or image quality degradation, there will be no problems that are difficult for users to accept, such as video freezes caused by excessive delays, and serious asynchrony between the picture and the sound.

[0060] System robustness and user experience assurance Setting the maximum value helps improve the robustness of the entire video playback system. When the system detects that the video playback delay is close to the maximum value, a series of countermeasures can be taken, such as automatically adjusting the video resolution or frame rate to reduce the amount of data, thereby alleviating network pressure; or prompting the user to close some unnecessary background applications to release device resources. These measures can maintain the basic quality of video playback as much as possible under adverse conditions and ensure the lower limit of user experience. At the same time, the maximum value also provides an important reference indicator for system performance evaluation and optimization. If the video playback delay is often close to the maximum value during normal use, this prompts the system administrator or developer to conduct in-depth inspection and optimization of the network infrastructure, device configuration or application code to improve the overall performance of the system under various conditions.

[0061] 3. The waiting delay for playback is 0 under ultra-low latency conditions The ultimate low latency strategy Setting the playback wait delay to 0 under ultra-low latency conditions is a strategy for extreme real-time performance. This means that when the video information arrives at the receiving end, the normal small waiting buffer time is completely skipped, and the video frame is immediately sent for decoding and rendering operations. Unlike the minimum value, which is a relatively low latency state naturally achieved under ideal conditions, the playback wait delay of 0 is a forced setting that breaks the buffer waiting mechanism that may exist in the traditional video playback process. For example, in some remote monitoring scenarios with extremely high real-time requirements (such as security monitoring of key facilities, real-time image transmission at emergency rescue sites) or live broadcast scenarios of real-time competitive games, every second or even every frame of delay may have a significant impact on the results. Adopting this strategy of 0 playback wait delay can ensure that the video screen is presented at the fastest speed, just as if the user is watching it directly on the scene, so that the user can get the key information in the video at the first time and respond accordingly quickly.

[0062] Technical implementation and potential risk response In order to achieve zero waiting delay for playback, it is necessary to carry out technical optimization in many aspects of the entire video transmission system. At the receiving end, an efficient cache management mechanism is required to pass the video data to the decoding module as soon as it is received without additional waiting and queuing. At the same time, the decoding and rendering modules also need to have the ability to start and process quickly, such as using hardware acceleration technology to ensure that the video frames can be decoded and rendered and displayed on the screen in a very short time. However, this extreme low-latency strategy also has certain potential risks. Since the buffer waiting time is completely eliminated, if there is a momentary fluctuation in the network or a data transmission error, it may cause video frame loss or decoding error, resulting in a short-term picture abnormality. In order to cope with this risk, the system needs to adopt some fault-tolerant mechanisms, such as forward error correction coding (FEC) technology, which redundantly encodes the video data at the sending end so that the data transmission error can be corrected to a certain extent at the receiving end; or adopt a fast retransmission mechanism. When the receiving end detects the loss or error of the video frame, it can quickly notify the sending end to retransmit the relevant data to ensure the continuity and stability of the video playback.

[0063] Through experimental testing, after the above steps of this embodiment, ultra-low delay transmission of video on the RK3588 chip can be achieved, and the minimum delay is 60 milliseconds.

[0064] Second embodiment like Figure 3 As shown, this embodiment provides a video ultra-low latency edge device solution system based on WebRTC for executing the video ultra-low latency edge device solution based on WebRTC in the first embodiment, including: Signaling service module 1, used to set up a signaling service for forwarding signaling information between a sender and a receiver, wherein the signaling information includes session control information, encoding and decoding information and other control information for interaction between the sender and the receiver The sending module 2 is used to collect video information, perform hardware encoding on the video information, and transmit the video information to the receiving end after the encoding is completed; The receiving module 3 is used to adopt hardware accelerated decoding and hardware accelerated rendering and display for the received video information.

[0065] A computer-readable storage medium stores computer code. When the computer code is executed, the above method is executed. A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. The storage medium can include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0066] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

[0067] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0068] It should be noted that the above embodiments can be freely combined as needed. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A video ultra-low latency edge device solution based on WebRTC, characterized in that: The following steps are involved: S1: Setting a signaling service for forwarding signaling information between a sending end and a receiving end, wherein the signaling information includes session control information, codec information and other control information for interaction between the sending end and the receiving end; S2: The sending end collects video information, performs hardware encoding on the video information, and transmits the video information to the receiving end after the encoding is completed; S3: The receiving end uses hardware accelerated decoding and hardware accelerated rendering and display for the received video information.

2. The video ultra-low latency edge device solution based on WebRTC according to claim 1 is characterized in that: In step S2, the transmitting end collects video information and performs hardware encoding on the video information, specifically: The transmitting end uses a camera to collect the original video data, a video acquisition module obtains video frames from the camera, and sends the video frames to a video encoding module, and the collected video data is stored in a YUV format; In order to run WebRTC in the edge device, the video encoding module uses the RK3588 chip to convert the data format of the collected video data into the I420 format required for encoding; The video data is hardware encoded on the RK3588 chip to reduce the bandwidth and storage space required for transmitting the video data.

3. The video ultra-low latency edge device solution based on WebRTC according to claim 1 is characterized in that: In step S2, after the encoding is completed, the video information is transmitted to the receiving end, specifically: After the encoding of the video information is completed, the encoded video information is encapsulated into a real-time transport protocol RTP packet, and the real-time transport protocol RTP packet is sent to the receiving end through the user datagram protocol UDP.

4. The ultra-low latency edge device solution for video based on WebRTC according to claim 1 is characterized in that: Also includes: Using the frame rate priority mode in WebRTC, priority is given to ensuring high frame rate encoding when the sender performs hardware encoding, high frame rate transmission when the sender transmits the video information to the receiver, and high frame rate decoding when the receiver decodes the video information.

5. The video ultra-low latency edge device solution based on WebRTC according to claim 1 is characterized in that: In step S3, the receiving end uses hardware accelerated decoding and hardware accelerated rendering and display for the received video information, specifically: The receiving end parses the received protocol data and extracts video related information for parsing the video information; Then, the video is decoded. The receiving end uses a video decoder to restore the compressed video information encoded at the sending end to the original video frame, and in the edge device, the decoding process is accelerated by hardware to increase the decoding speed; The decoded video frames of the video information are transmitted to a rendering module, and the rendering module uses hardware acceleration to display the video frames on a screen.

6. The video ultra-low latency edge device solution based on WebRTC according to claim 1 is characterized in that: Also includes: Adding an audio flag to the session description protocol SDP of the signaling service; When the session description protocol SDP contains the audio flag, the sending end no longer sends audio information, the receiving end no longer receives the audio information, and if there is audio information, it is directly discarded, the video information is directly decoded, and rendered immediately after decoding is completed.

7. The video ultra-low latency edge device solution based on WebRTC according to claim 1 is characterized in that: Also includes: Set the minimum and maximum values ​​of the actual delay of the video playback of the video information, and set the playback waiting delay to 0 under ultra-low delay conditions; The minimum value is the shortest time interval allowed from the completion of video data reception to the start of operations including decoding and rendering when all conditions are in a relatively ideal state during video playback. The maximum value is an upper limit of video playback delay, which exists to limit the range that video playback delay will not exceed under any adverse circumstances; Setting the playback wait delay to 0 is to artificially and forcibly set the playback wait time to 0, which means that when the video information arrives at the receiving end, there will no longer be any deliberately reserved waiting buffer time, and the video frame will be immediately sent for decoding and rendering operations.

8. A video ultra-low latency edge device solution system based on WebRTC for executing the video ultra-low latency edge device solution based on WebRTC as described in any one of claims 1 to 7, characterized in that: include: A signaling service module is used to set a signaling service for forwarding signaling information between a sender and a receiver, wherein the signaling information includes session control information, codec information and other control information for interaction between the sender and the receiver. A sending module, used for collecting video information, performing hardware encoding on the video information, and transmitting the video information to the receiving end after encoding is completed; The receiving module is used to adopt hardware accelerated decoding and hardware accelerated rendering and display for the received video information.

9. A computer device comprising a memory and one or more processors, wherein the memory stores computer codes, and when the computer codes are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 7. 10 . A computer-readable storage medium storing a computer code. When the computer code is executed, the method according to claim 1 is executed.

Images