HTTP3-based gb / t 28181 monitoring video stream transmission and decoding method and electronic equipment

Abstract

This invention discloses a method and electronic device for efficient transmission and decoding of GB / T 28181 surveillance video streams based on HTTP / 3. This method utilizes the next-generation Web APIs WebTransport and WebCodecs to solve the high latency and high resource consumption problems of traditional WebSocket-based GB / T 28181 video surveillance web-based playback solutions. This invention achieves direct transmission of RTP data packets from the server to the web client through WebTransport's UDP transmission capability based on HTTP / 3, eliminating the need for protocol conversion; it employs WebCodecs for efficient hardware-accelerated decoding on the client side; and it provides a multi-channel multiplexed real-time monitoring and adaptive video quality control mechanism. Compared to traditional solutions, this invention significantly reduces end-to-end latency, decreases server resource consumption, improves concurrency capabilities, and can dynamically adjust video parameters according to network conditions, making it suitable for video surveillance applications in various network environments.

Description

Technical Field

[0001] This invention relates to the fields of computer networks and video surveillance technology, and in particular to a GB / T28181 method and electronic device for transmitting and decoding surveillance video streams based on HTTP3. Background Technology

[0002] Currently, monitoring platforms based on the GB / T28181 standard mainly use the SIP protocol for signaling interaction and the RTP protocol for transmitting video stream data. In the web client implementation, traditional solutions typically use WebSocket technology. The server converts the received RTP data into WebSocket+FMP4 format before transmitting it to the browser, which then plays it using the HTML5 video or audio tags. This solution has a significant performance bottleneck:

[0003] Protocol conversion is costly: SIP to WebSocket and RTP to FMP4 both require significant computing resources.

[0004] Low transmission efficiency: WebSocket is based on the TCP protocol, which is not suitable for scenarios involving real-time video transmission;

[0005] The decoding process is cumbersome: the FMP4 format requires the browser to complete the decompression and decoding, which increases the latency of the web client.

[0006] Although WebRTC technology is also a high-efficiency transport protocol based on UDP, enabling low-latency video transmission, it has the following problems:

[0007] Overly complex encapsulation: WebRTC integrates complete audio and video capture, encoding, transmission, and decoding functions, making it difficult to customize flexibly;

[0008] The server development cost is high: it requires the deployment of STUN / TURN servers to handle NAT traversal and implement complex signaling servers; when connecting to GB / T28181 equipment, an additional protocol conversion layer is required, which increases the system complexity.

[0009] Faced with the limitations of the aforementioned traditional technical solutions, the industry has been exploring more efficient solutions. Recently, with the gradual maturation of next-generation Web APIs such as WebTransport and WebCodecs launched by browser vendors, a new technical path has been provided for building high-performance, low-latency Web video surveillance clients. Therefore, there is an urgent need for a GB / T28181 method and system for monitoring video stream transmission and decoding based on HTTP3 to solve the existing technical problems. Summary of the Invention

[0010] The present invention aims to solve at least one of the technical problems existing in the prior art, and proposes a GB / T28181 monitoring video stream transmission and decoding method and electronic device based on HTTP3.

[0011] In a first aspect, embodiments of the present invention provide a GB / T28181 method for transmitting and decoding surveillance video streams based on HTTP3, including:

[0012] The web client receives the user's playback request, establishes a WebTransport connection with the transcoding gateway, and sends playback request parameters containing device information to the transcoding gateway.

[0013] The transcoding gateway parses the playback request parameters, generates a SIP INVITE message, and sends it to the GB / T28181 monitoring platform.

[0014] The GB / T28181 monitoring platform parses SIP INVITE messages and sends RTP data packets to the transcoding gateway;

[0015] The transcoding gateway forwards RTP packets to the web client via WebTransport. The web client processes the RTP data packets to obtain the original video and then displays it.

[0016] Furthermore, the user playback request includes single-channel video requests and multi-channel video requests. When the user playback request is a single-channel video request, the specific method for the Web client to establish a WebTransport connection with the transcoding gateway includes: the Web client creates a WebTransport instance and connects to the transcoding gateway; when the connection is successful, a bidirectional data stream is established, and a playback request containing device information is sent; when the connection fails, a network error is displayed and the process is terminated.

[0017] Furthermore, when a user's playback request is a single video request, the transcoding gateway forwards the RTP packet to the web client via WebTransport. Specifically, the transcoding gateway receives the RTP data packet sent by the device, looks up the corresponding WebTransport connection based on the SSRC identifier, and forwards the RTP packet to the web client via WebTransport.

[0018] Furthermore, when a user's playback request is a single-channel video request, the Web client processes the RTP data packets to obtain the original video and displays it. Specifically, the Web client receives RTP data packets through WebTransport, extracts encoded video frames using an RTP parser implemented in WASM, passes the extracted encoded frames to the VideoDecoder of WebCodecs for decoding, the VideoDecoder decodes the encoded frames into original video frames, and after successful decoding, renders and displays the frames using Canvas or WebGL.

[0019] Furthermore, when a user's playback request is a single video request, the web client establishes a multi-channel shared WebTransport connection with the transcoding gateway. The specific method includes: the web client creates a single WebTransport connection, and after the connection is successful, it creates a control data stream and multiple unidirectional receive data streams. The control stream is used for command transmission, and each video stream corresponds to a dedicated data stream. The client sends JSON format commands through the control data stream, and the transcoding gateway parses the control messages and executes the corresponding operations.

[0020] Furthermore, when a user requests multiple video streams, the web client and transcoding gateway process these requests. Specifically, the web client receives the user's multiple video stream requests, generates an ADD_STREAM request, assigns a unique identifier to each stream, and sends the request to the transcoding gateway. The transcoding gateway generates a SIPIVITE message for available devices and sends the request to the GB / T28181 monitoring platform. Upon receiving the INVITE request, the GB / T28181 monitoring platform sends an RTP stream. The transcoding gateway receives RTP data packets from different devices and identifies the different video streams based on the SSRC (Security Stream Controller Rating).

[0021] Furthermore, when a user requests multiple video streams, traffic control and priority management are implemented for each stream. Priorities are assigned based on video window size and activity status. When bandwidth is insufficient, adjustments are made according to priority, and the quality of each stream is restored in priority order after bandwidth is restored. Efficient transmission of the multiple video streams is achieved by creating a frame buffer for each stream and sorting packets. When a buffer overflows, the oldest non-critical frame is discarded, and RTP packets are sent through the corresponding WebTransport data stream. A streamId identifier is added to the data packet to distinguish different streams for the client. Parallel decoding of the multiple video streams is performed by simultaneously monitoring the arrival events of multiple data streams. Data is distributed to different decoding pipelines based on the streamId. Independent parser and decoder instances are created for each stream, and WebWorkers are used to implement multi-threaded decoding to avoid blocking the main thread. Collaborative rendering of the multiple video streams is performed by allocating rendering areas according to the user-defined layout. When rendering resources are insufficient, the quality is reduced or the number of parallel rendering streams is decreased, and requestAnimationFrame is used for synchronous rendering.

[0022] Furthermore, when the Web client establishes a WebTransport connection with the transcoding gateway, the Web client also performs bandwidth detection, determines the initial video parameters based on the bandwidth detection results, and sends a video request containing the initial video parameters to the transcoding gateway. The transcoding gateway generates a SIP INVITE message with adaptation parameters based on the video request. The GB / T28181 monitoring platform checks the device capabilities, determines the adaptation method, and the Web client performs real-time network monitoring, adjusts the parameters using an adaptive transmission strategy, manages the low-latency buffer, and optimizes the video image using frame rate adaptive technology and an I-frame request mechanism to obtain the original video, which is then displayed.

[0023] Furthermore, an adaptive transmission strategy is adopted to adjust parameters, specifically including: adjusting video parameters according to network conditions when bandwidth changes exceed a threshold, frame rate drops beyond a threshold, or user adjustment requests are received; managing low-latency buffers, specifically including: calculating the optimal buffer size based on network jitter; employing frame rate adaptive technology, specifically including monitoring decoding performance and frame processing time, reducing the target frame rate and decoding resolution when CPU utilization exceeds a threshold; and an I-frame request mechanism, specifically including: sending an I-frame request when a decoding error, abnormal image, video startup initialization phase, or resolution or quality level switch is detected; the server, upon receiving the request, sends an RTCP keyframe request to the device or inserts an I-frame in the transcoder; and resetting the decoder upon receiving the I-frame to restore normal display.

[0024] Secondly, the present invention also discloses an electronic device, comprising:

[0025] One or more processors;

[0026] Memory, used to store one or more programs;

[0027] When the one or more programs are executed by the one or more processors, the one or more processors implement the video stream transmission and decoding method.

[0028] This invention discloses a method and electronic device for efficient transmission and decoding of GB / T 28181 surveillance video streams based on HTTP / 3. This method utilizes the next-generation Web APIs WebTransport and WebCodecs to solve the high latency and high resource consumption problems of traditional WebSocket-based GB / T 28181 video surveillance web-based playback solutions. This invention achieves direct transmission of RTP data packets from the server to the web client through WebTransport's UDP transmission capability based on HTTP / 3, eliminating the need for protocol conversion; it employs WebCodecs for efficient hardware-accelerated decoding on the client side; and it provides a multi-channel multiplexed real-time monitoring and adaptive video quality control mechanism. Compared to traditional solutions, this invention significantly reduces end-to-end latency, decreases server resource consumption, improves concurrency capabilities, and can dynamically adjust video parameters according to network conditions, making it suitable for video surveillance applications in various network environments. Attached Figure Description

[0029] Figure 1 A flowchart illustrating a GB / T28181 monitoring video stream transmission and decoding method based on HTTP3, provided for an embodiment of the present invention;

[0030] Figure 2 This is a network topology diagram of a GB / T28181 surveillance video transmission and decoding system based on HTTP3, according to an embodiment of the present invention.

[0031] Figure 3 This is a flowchart of a single-channel monitoring video transmission method according to an embodiment of the present invention;

[0032] Figure 4 This is a flowchart of a multi-channel surveillance video transmission method according to an embodiment of the present invention;

[0033] Figure 5 This is a flowchart of an embodiment of the adaptive video quality and low-latency transmission method of the present invention;

[0034] Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0036] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0037] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0038] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0039] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0040] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0041] To address at least one of the technical problems existing in the aforementioned related technologies, the present invention provides a GB / T28181 monitoring video stream transmission and decoding method and electronic device based on HTTP3.

[0042] This implementation discloses a GB / T28181 method for monitoring video stream transmission and decoding based on HTTP3, such as... Figure 1 ,include:

[0043] The S100 Web client receives a user's playback request, establishes a WebTransport connection with the transcoding gateway, and sends playback request parameters containing device information to the transcoding gateway.

[0044] S200. The transcoding gateway parses the playback request parameters, generates a SIP INVITE message, and sends it to the GB / T28181 monitoring platform.

[0045] The S300.GB / T28181 monitoring platform parses SIP INVITE messages and sends RTP data packets to the transcoding gateway;

[0046] The S400 transcoding gateway forwards RTP packets to the web client via WebTransport. The web client processes the RTP data packets to obtain the original video and displays it.

[0047] This embodiment utilizes the next-generation Web APIs WebTransport and WebCodecs to solve the high latency and high resource consumption problems of traditional WebSocket-based GB / T 28181 video surveillance web-based playback solutions. This invention achieves direct transmission of RTP data packets from the server to the web client through WebTransport's HTTP / 3-based UDP transmission capabilities, eliminating the need for protocol conversion; it employs WebCodecs for efficient hardware-accelerated decoding on the client side; and it provides multiplexed real-time monitoring and adaptive video quality control mechanisms. Compared to traditional solutions, this invention significantly reduces end-to-end latency, decreases server resource consumption, improves concurrency capabilities, and can dynamically adjust video parameters according to network conditions, making it suitable for video surveillance applications in various network environments.

[0048] To better understand this implementation, this implementation first introduces a system based on a GB / T28181 method for monitoring video stream transmission and decoding based on HTTP3, such as... Figure 2In this embodiment, a system for a GB / T28181 surveillance video stream transmission and decoding method based on HTTP3 includes a GB / T28181 video surveillance server, a transcoding gateway, and a web client browser. The web client browser and the transcoding gateway have a one-to-many relationship, while the transcoding gateway and the GB / T28181 video surveillance server have a many-to-many relationship. Figure 1 This explanation uses only one-to-one video live streaming requests from a web client browser as an example.

[0049] The GB / T28181 video surveillance server, i.e., the implementation of the GB / T28181 protocol server, includes, but is not limited to, functions such as device registration, device management, user registration, user management, video storage, live streaming, and recorded playback. Specifically, it can receive registration requests from end users; after successful registration, users can request live or recorded media streams, obtain relevant device information, and control the devices. This embodiment only describes live video streaming; other functions are not described here.

[0050] The transcoding gateway server, which can also be a cluster, is the core of the entire system. It is responsible for signaling processing, media forwarding, and protocol adaptation. The gateway integrates a SIP server component to handle the registration and session management of GB / T28181 devices, maintain the online status of devices, and provide WebTransport services based on the HTTP / 3 protocol. It accepts connections from web clients and establishes a reliable bidirectional data channel. Depending on the network conditions and device capabilities of different clients, the server may perform adaptive transcoding operations or implement flow control strategies to ensure that data transmission adapts to the client environment. Unlike traditional solutions, the server in this system does not perform complex conversions such as RTP to FMP4. Instead, it directly forwards the received RTP packets to the web client through WebTransport, which greatly reduces the server's processing load.

[0051] In this embodiment, the user playback request includes single-channel video requests and multi-channel video requests. It can be understood that single-channel video requests and multi-channel video requests correspond to a user selecting a single monitoring device to watch and a user selecting multiple monitoring devices to watch, respectively. This embodiment first elaborates on the user playback request as a single-channel video request.

[0052] In this embodiment, as Figure 3When a user requests a single video stream, the system first initializes, starting the SIP server, media components, and WebTransport server components. If any component fails to initialize, an error log is logged, and the startup process terminates. Once all components initialize successfully, the system enters the running state. After initialization, the web client environment is checked, verifying browser support for WebTransport and WebCodecs APIs. If not supported, it automatically reverts to the traditional WebSocket+MSE solution; if supported, the relevant components are initialized, preparing to establish a connection. Finally, the user selects a monitoring device, and the web client retrieves the list of available devices. If retrieval fails, an error message is displayed, and the process terminates. If successful, the user selects the device and channel, clicks play, and initiates the WebTransport connection establishment.

[0053] In this embodiment, after the user selects a single monitoring device, the Web client establishes a WebTransport connection with the transcoding gateway. The specific method includes: the Web client creates a WebTransport instance and connects to the transcoding gateway. When the connection is successful, a bidirectional data stream is established, and a playback request containing device information is sent. When the connection fails, a network error is displayed and the process is terminated.

[0054] In this embodiment, after the Web client establishes a WebTransport connection with the transcoding gateway, the transcoding gateway forwards the RTP packets to the Web client through the WebTransport. The specific method includes: the transcoding gateway receives the RTP data packets sent by the device, looks up the corresponding WebTransport connection according to the SSRC identifier, and forwards the RTP packets to the Web client through the WebTransport.

[0055] In this embodiment, the transcoding gateway parses the playback request parameters, generates a SIP INVITE message, and sends it to the GB / T28181 monitoring platform. The specific method includes: the transcoding gateway parses the playback request parameters; if the parameters are invalid, it returns an error message and terminates the session; if the parameters are valid, it generates a SIP INVITE message and sends it to the GB / T28181 monitoring platform; it records the correspondence between the WebTransport connection and the GB / T28181 stream; the GB / T28181 monitoring platform parses the SIP INVITE message and initiates a media stream request to the corresponding front-end device; if the front-end device refuses or does not respond within 30 seconds, the transcoding gateway notifies the client of an error and terminates the process; if the front-end device accepts the request and returns a 200 OK response, it begins sending RTP data packets to the transcoding gateway.

[0056] In this embodiment, after sending RTP data packets to the transcoding gateway, the Web client processes the RTP data packets to obtain the original video and displays the original video. The specific method includes: the Web client receives RTP data packets through WebTransport, extracts encoded video frames using an RTP parser implemented in WASM, passes the extracted encoded frames to the VideoDecoder of WebCodecs for decoding, the VideoDecoder decodes the encoded frames into original video frames, and after successful decoding, renders and displays the frames using Canvas or WebGL.

[0057] In some preferred embodiments, the termination process is triggered when the user clicks "stop," leaves the page, or experiences a prolonged network interruption; the client sends a stop request; the server sends a SIP BYE message to the GB / T28181 device; resources are cleared and the WebTransport connection is closed; a "video has stopped" message is displayed, and the process ends normally.

[0058] This embodiment constructs a complete GB / T28181 surveillance video web-based playback solution, from system initialization, connection establishment, video request, data transmission to decoding and rendering. Compared with traditional solutions, this solution directly transmits RTP data via WebTransport, avoiding protocol conversion and significantly improving transmission efficiency and reducing latency.

[0059] In some preferred embodiments, when a user's playback request is a multi-channel video request, the web client establishes a multi-channel shared WebTransport connection with the transcoding gateway. The specific method includes: the web client creating a single WebTransport connection, and after the connection is successful, creating a control data stream and multiple unidirectional receive data streams. The control stream is used for command transmission, and each video stream corresponds to a dedicated data stream. The client sends JSON format commands through the control data stream, and the transcoding gateway parses the control messages and executes the corresponding operations.

[0060] In some preferred embodiments, when a user's playback request is a multi-channel video request, the web client and the transcoding gateway process the multi-channel video request. Specifically, the web client receives the user's multi-channel video request, generates an ADD_STREAM request, assigns a unique identifier to each stream, and sends the request to the transcoding gateway. The transcoding gateway generates a SIP INVITE message for available devices and sends the request to the GB / T28181 monitoring platform. Upon receiving the INVITE request, the GB / T28181 monitoring platform sends an RTP stream. The transcoding gateway receives RTP data packets from different devices and identifies different video streams based on the SSRC.

[0061] In some preferred embodiments, when a user requests multiple video streams, traffic control and priority management are performed on these streams. Priorities are assigned based on video window size and activity status. When bandwidth is insufficient, adjustments are made according to priority, and the quality of each stream is restored in priority order after bandwidth is restored. Efficient transmission of the multiple video streams is achieved by creating a frame buffer for each stream and sorting packets. When a buffer overflows, the oldest non-critical frame is discarded. RTP packets are sent through the corresponding WebTransport data stream, with a streamId identifier added to the packet for the client to distinguish between different streams. Parallel decoding of the multiple video streams is performed by simultaneously monitoring the arrival events of multiple data streams. Data is distributed to different decoding pipelines based on the streamId. Independent parser and decoder instances are created for each stream, and WebWorkers are used to implement multi-threaded decoding to avoid blocking the main thread. Collaborative rendering of the multiple video streams is performed by allocating rendering areas according to the user-defined layout. When rendering resources are insufficient, the quality is reduced or the number of parallel rendering streams is decreased. RequestAnimationFrame is used for synchronous rendering.

[0062] To better understand the processing of multiple video requests in this embodiment, this embodiment describes the detailed steps, such as... Figure 4 As shown:

[0063] 1. System initialization: Initialize basic components according to the single-channel video request process; add a media scheduler component to manage multiplexing; log errors and terminate the process when component initialization fails; after successful initialization, the client loads the page and checks the environment support.

[0064] 2. Loading the device list: The web client retrieves a list of available monitoring devices; it automatically retryes if the request fails, and prompts the user if it still fails after three attempts; after a successful request, the device list is displayed, and the user can select multiple devices to preview simultaneously.

[0065] 3. Establishment of a multi-channel shared connection: The client creates a single WebTransport connection; if the connection fails, it attempts to use a backup server, and if it continues to fail, it displays an error message; after a successful connection, it creates a control data stream and multiple one-way receive data streams; the control stream is used for command transmission, and each video stream corresponds to a dedicated data stream.

[0066] 4. Transmission Control Protocol Implementation: The client sends JSON-formatted commands through the control data stream; if sending fails, it retryes; if failures continue, a connection error is displayed; the transcoding gateway parses the control messages and executes the corresponding operations.

[0067] 5. Multi-stream video requests: After the user selects multiple cameras, the client generates an ADD_STREAM request to assign a unique identifier (streamId) to each stream; an error message is displayed when the request fails to be sent; the processing flow begins after the request is sent successfully.

[0068] 6. Transcoding Gateway Multi-Request Processing: Parse the device list and verify device availability; return corresponding error information for unavailable devices; generate SIP INVITE messages for available devices and send them in parallel to the GB / T28181 monitoring platform; maintain the mapping relationship between streamId and GB / T28181 session; return a global error when all device requests fail; process the video streams of available devices when some requests succeed.

[0069] 7. Multi-channel RTP data reception: After receiving the INVITE request, the GB / T28181 monitoring platform sends an RTP stream; the transcoding gateway receives RTP data packets from different devices; if a device does not send data within 5 seconds, it is marked as abnormal and the client is notified; the media scheduler identifies different video streams based on SSRC, and the normally received streams proceed to the next step.

[0070] 8. Flow control and priority management: Priority is assigned based on video window size and activity status; when bandwidth is insufficient, it is adjusted according to priority: high priority windows maintain full quality; medium priority windows reduce parameters; low priority windows may be paused or only send keyframes; after bandwidth is restored, the quality of each stream is restored in priority order.

[0071] 9. Efficient transmission of multiple data streams: Create frame buffers for each stream and sort packets; discard the oldest non-critical frames when the buffer overflows; send RTP packets through the corresponding WebTransport data stream; add a streamId identifier to the data packets for the client to distinguish between different streams; shut down and notify the client when a certain data stream fails to send continuously.

[0072] 10. Client-side multi-channel parallel decoding: Simultaneously listen for data arrival events from multiple data streams; distribute data to different decoding pipelines based on streamId; create independent parser and decoder instances for each stream; utilize WebWorkers to implement multi-threaded decoding to avoid blocking the main thread; prompt the user that the stream cannot be displayed when decoder creation fails.

[0073] 11. Multi-path collaborative rendering: Allocate rendering areas according to the layout set by the user; reduce quality or the number of parallel rendering streams when rendering resources are insufficient; use requestAnimationFrame for synchronous rendering; and achieve GPU-accelerated image compositing and rendering.

[0074] 12. Dynamic Stream Management: Users can add or remove monitored cameras at any time; the client sends ADD_STREAM or REMOVE_STREAM control messages; if the control message fails to send, an operation failure message is displayed and retry is allowed; the server responds to the control message, establishing or terminating the corresponding session; creating or closing the corresponding WebTransport data stream; the process continues after the interface is updated.

[0075] 13. Session terminated: Triggered when the user clicks "Close All" or leaves the page; the client sends a CLOSE_ALL control message; the transcoding gateway sends a SIP BYE message to all active devices and forwards it to the GB / T28181 monitoring platform; all resources are released and the WebTransport connection is closed; the message "All videos have stopped" is displayed, and the process ends normally.

[0076] This embodiment focuses on solving scenarios involving simultaneous viewing of multiple video streams. By leveraging the multiplexing capabilities of a single WebTransport connection, combined with priority management and parallel decoding, highly efficient multi-channel video monitoring is achieved. The system can dynamically adjust resource allocation for each video stream based on user behavior and network conditions, ensuring smooth display of critical footage.

[0077] In some preferred embodiments, since it is difficult to guarantee the quality of video surveillance under different network environments, when the Web client establishes a WebTransport connection with the transcoding gateway, the Web client will also perform bandwidth detection, determine the initial video parameters based on the bandwidth detection results, and send a video request containing the initial video parameters to the transcoding gateway. The transcoding gateway generates a SIP INVITE message with adaptation parameters based on the video request. The GB / T28181 monitoring platform checks the device capabilities and determines the adaptation method. The Web client performs real-time network monitoring, adjusts the parameters using an adaptive transmission strategy, manages the low-latency buffer, and optimizes the video image using frame rate adaptive technology and an I-frame request mechanism to obtain the original video, which is then displayed.

[0078] Specifically, an adaptive transmission strategy is adopted to adjust parameters, including: adjusting video parameters according to network conditions when bandwidth changes exceed a threshold, frame rate drops beyond a threshold, or user adjustment requests are received; managing low-latency buffers, including calculating the optimal buffer size based on network jitter; employing frame rate adaptive technology, including monitoring decoding performance and frame processing time, and reducing the target frame rate and decoding resolution when CPU utilization exceeds a threshold; and an I-frame request mechanism, including sending an I-frame request when a decoding error, abnormal image, video startup initialization phase, or resolution or quality level switch is detected. After receiving the request, the server sends an RTCP keyframe request to the device or inserts an I-frame in the transcoder. Upon receiving the I-frame, the decoder is reset, and normal display is restored.

[0079] To better understand how this embodiment addresses the issues of adaptive transmission and low-latency playback under different network environments, this embodiment describes the detailed steps, such as... Figure 5 As shown:

[0080] 1. System Initialization and Basic Connections: Complete the initialization of basic components according to the single-channel video request process; add traffic monitoring and adaptive transcoding components on the server side; fall back to basic mode if the adaptive component initialization fails; add network quality detection and buffer control modules on the client side; proceed to the next step after successful initialization.

[0081] 2. Network Status Probe: After the connection is established, perform a bandwidth probe; if the probe fails, use conservative default parameters; if the probe succeeds, record the round-trip time and bandwidth data; exchange network parameters and negotiate the initial transmission configuration.

[0082] 3. Adaptive quality control strategy: Determine the initial video parameters based on the detection results, specifically including using low quality mode when the bandwidth is extremely low (<500Kbps); using high quality mode when the bandwidth is sufficient (>2Mbps); using standard quality mode in other cases; sending a playback request containing parameters to the server; retrying if the request fails to send; terminating the process if it continues to fail; the server generates a SIP INVITE message with adapted parameters based on the request.

[0083] 4. Multi-level quality adaptation: Check device capabilities and determine the adaptation method: If the device supports parameter adjustment, directly request the adaptation bitstream; if the device does not support it but the server resources are sufficient, enable transcoding; if none of these are feasible, perform keyframe extraction; initialize the corresponding processing pipeline; if initialization fails, try alternative solutions; if all fail, terminate the process.

[0084] 5. Real-time network monitoring: The client calculates the receiving rate, packet loss rate, and jitter every 5 seconds; it immediately triggers re-adaptation when there is a severe network anomaly (packet loss rate > 30%); and it triggers reconnection when there is a network interruption (no data for > 10 seconds).

[0085] The server monitors uplink bandwidth usage; both parties exchange network status information; if the network is normal, the current transmission continues.

[0086] 6. Adaptive transmission strategy adjustment: Adjustment is triggered under the following conditions: 1. Bandwidth change > 30%; Insufficient decoding performance (frame rate reduction > 20%); User manually requests adjustment;

[0087] When the network improves, gradually increase the parameters (one level at a time); when the network deteriorates, quickly decrease the parameters (multiple levels can be decreased); the server may need to re-initiate a session with the device or modify the transcoding parameters; if the adjustment fails, revert to the previous stable configuration.

[0088] 7. Low-latency buffer management: Calculates the optimal buffer size based on network jitter: Uses the minimum buffer (50-100ms) when the network is stable (jitter <30ms); uses a medium buffer (100-300ms) when the network is moderate (jitter 30-100ms); uses a larger buffer (300-500ms) when the network is unstable (jitter >100ms); attempts a smooth transition when buffer adjustment causes playback interruption; prompts network instability if multiple adjustments still fail to stabilize the network.

[0089] 8. Adaptive Frame Rate Technology: Monitors decoding performance and frame processing time; performs the following actions when decoding latency is high: reduces the target frame rate when CPU utilization is too high (>80%); reduces the decoding resolution when GPU utilization is too high.

[0090] The server adjusts the sending strategy based on client feedback; if the performance is still insufficient after adjustment, it requests the server to perform frame rate downsampling; if all optimizations are ineffective, it prompts that the device performance is insufficient.

[0091] 9. I-frame request mechanism: I-frame requests are sent in the following situations: decoding error or abnormal picture is detected; during video startup initialization; after switching resolution or quality level; retrying when I-frame request transmission fails; after the server receives the request, it sends an RTCP keyframe request to the device or inserts an I-frame in the transcoder; when the device does not respond, it attempts to re-establish the session; after receiving an I-frame, the decoder is reset and normal display is restored.

[0092] 10. Quality Degradation for Smoothness Strategy: The "Quality Degradation for Smoothness" mode will be activated under the following conditions: frequent buffer exhaustion (>3 times within 10 seconds); irregular frame intervals (jitter > 100ms); continuous packet loss rate exceeding the threshold (>15%). The quality degradation strategy is executed in the following order of priority: first reduce resolution and image quality to maintain a stable frame rate; if the effect is not obvious, reduce the frame rate to ensure smooth playback; in extreme cases, reduce to static image mode (1-2 frames per second).

[0093] If the video quality is still not smooth after the quality is downgraded, an insufficient network condition is indicated; after the network is restored (stabilized for more than 10 seconds), the video quality will gradually be restored.

[0094] 11. Intelligent Frame Dropping and Catching-Up Playback: The corresponding compensation mechanism will be activated in the following situations: Catching-up playback will be activated when the transmission delay increases (>500ms); accelerated playback will be activated when the buffer is too large (>1000ms); the catching-up mechanism will perform the following operations: temporarily increase the playback speed (1.1-1.2 times) to reduce the delay; perform intelligent frame dropping on non-critical frames when transmission is congested; discard B-frames or some P-frames to reduce bandwidth pressure; automatically terminate when catching-up causes the picture to be unstable; and restore normal playback speed after the target delay is reached.

[0095] 12. Session End and Resource Release: The termination process is triggered in the following situations: user actively stops playback; network interruption for a long period of time (>60 seconds without data); browser page is closed or navigation leaves; server actively closes the session;

[0096] Perform the following cleanup operations: The client sends a stop request; the server sends a SIP BYE message to the GB / T28181 device; closes and releases all connections and data streams; releases decoders and rendering resources; after the resources are released, a "Video session has ended" message is displayed; the process ends normally.

[0097] This embodiment focuses on addressing the issue of ensuring video surveillance quality in complex and ever-changing network environments. Through various technologies such as network condition detection, adaptive quality control, and low-latency buffer management, the system can dynamically adjust transmission parameters based on real-time network conditions and device performance, providing the best monitoring experience in various environments.

[0098] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. For example... Figure 6 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0099] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0100] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0101] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0102] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described in the above embodiments. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0103] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described method.

[0104] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0105] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0106] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0107] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0108] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0109] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0110] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0111] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0113] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A GB / T28181 method for transmitting and decoding surveillance video streams based on HTTP3, characterized in that, include: The web client receives the user's playback request, establishes a WebTransport connection with the transcoding gateway, and sends playback request parameters containing device information to the transcoding gateway. The transcoding gateway parses the playback request parameters, generates a SIP INVITE message, and sends it to the GB / T28181 monitoring platform. The GB / T28181 monitoring platform parses SIP INVITE messages and sends RTP data packets to the transcoding gateway; The transcoding gateway forwards the received RTP data packets directly to the web client via WebTransport. The web client processes the RTP data packets to obtain the original video and displays it.

2. The video stream transmission and decoding method according to claim 1, characterized in that, The user playback request includes single-channel video requests and multi-channel video requests. When the user playback request is a single-channel video request, the specific method for the Web client to establish a WebTransport connection with the transcoding gateway includes: the Web client creates a WebTransport instance and connects to the transcoding gateway; when the connection is successful, a bidirectional data stream is established, and a playback request containing device information is sent; when the connection fails, a network error is displayed and the process is terminated.

3. The video stream transmission and decoding method according to claim 2, characterized in that, When a user requests a single video stream, the transcoding gateway forwards the RTP packet to the web client via WebTransport. The specific method includes: the transcoding gateway receives the RTP data packet sent by the device, looks up the corresponding WebTransport connection based on the SSRC identifier, and forwards the RTP packet to the web client via WebTransport.

4. The video stream transmission and decoding method according to claim 2, characterized in that, When a user requests a single video stream, the web client processes the RTP data packets to obtain the original video and displays it. Specifically, the web client receives RTP data packets through WebTransport, extracts encoded video frames using an RTP parser implemented in WASM, passes the extracted encoded frames to the VideoDecoder of WebCodecs for decoding, the VideoDecoder decodes the encoded frames into original video frames, and renders and displays the frames using Canvas or WebGL after successful decoding.

5. The video stream transmission and decoding method according to claim 2, characterized in that, When a user requests multiple video streams, the web client establishes a shared WebTransport connection with the transcoding gateway. Specifically, the web client creates a single WebTransport connection, and after the connection is successful, it creates a control data stream and multiple unidirectional receive data streams. The control data stream is used for command transmission, and each video stream corresponds to a dedicated data stream. The client sends JSON-formatted commands through the control data stream, and the transcoding gateway parses the control messages and executes the corresponding operations.

6. The video stream transmission and decoding method according to claim 2, characterized in that, When a user requests multiple video streams, the web client and transcoding gateway process these requests. Specifically, the web client receives the user's multiple video stream requests, generates an ADD_STREAM request, assigns a unique identifier to each stream, and sends the request to the transcoding gateway. The transcoding gateway generates a SIP INVITE message for available devices and sends the request to the GB / T28181 monitoring platform. Upon receiving the INVITE request, the GB / T28181 monitoring platform sends an RTP stream. The transcoding gateway receives RTP data packets from different devices and identifies the different video streams based on the SSRC (Security Stream Controller Rating).

7. The video stream transmission and decoding method according to claim 2, characterized in that, When a user requests multiple video streams, traffic control and priority management are implemented. Priorities are assigned based on video window size and activity status. When bandwidth is insufficient, adjustments are made according to priority, and the quality of each stream is restored in priority order after bandwidth is restored. Efficient transmission of multiple video streams is achieved by creating a frame buffer for each stream and sorting packets. When a buffer overflows, the oldest non-critical frame is discarded, and RTP packets are sent through the corresponding WebTransport data stream. A streamId identifier is added to the data packet for the client to distinguish between different streams. Parallel decoding of multiple video streams is performed by simultaneously monitoring the arrival events of multiple data streams. Data is distributed to different decoding pipelines based on the streamId. Independent parser and decoder instances are created for each stream, and WebWorkers are used to implement multi-threaded decoding to avoid blocking the main thread. Collaborative rendering of multiple video streams is performed by allocating rendering areas according to the user-defined layout. When rendering resources are insufficient, the quality is reduced or the number of parallel rendering streams is decreased, and requestAnimationFrame is used for synchronous rendering.

8. The video stream transmission and decoding method according to claim 1, characterized in that, When the web client establishes a WebTransport connection with the transcoding gateway, the web client also performs bandwidth detection, determines the initial video parameters based on the bandwidth detection results, and sends a video request containing the initial video parameters to the transcoding gateway. The transcoding gateway generates a SIP INVITE message with adaptation parameters based on the video request. The GB / T28181 monitoring platform checks the device capabilities, determines the adaptation method, and the web client performs real-time network monitoring, adjusts the parameters using an adaptive transmission strategy, manages the low-latency buffer, and optimizes the video image using frame rate adaptation technology and an I-frame request mechanism to obtain the original video, which is then displayed.

9. The video stream transmission and decoding method according to claim 8, characterized in that, An adaptive transmission strategy is employed to adjust parameters, specifically including: adjusting video parameters according to network conditions when bandwidth changes exceed a threshold, frame rate drops beyond a threshold, or user adjustment requests are received; managing low-latency buffers, specifically including: calculating the optimal buffer size based on network jitter; employing frame rate adaptive technology, specifically including monitoring decoding performance and frame processing time, reducing the target frame rate and decoding resolution when CPU utilization exceeds a threshold; and an I-frame request mechanism, specifically including: sending an I-frame request when a decoding error, image abnormality, video startup initialization phase, or resolution or quality level switch is detected; the server, upon receiving the request, sends an RTCP keyframe request to the device or inserts an I-frame in the transcoder; and resetting the decoder upon receiving the I-frame to restore normal display.

10. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video streaming and decoding method as described in any one of claims 1 to 9.

Images