Cross-platform video decoding and playing system and method

Abstract

The invention discloses a cross-platform video decoding and playing system and a cross-platform video decoding and playing method, which belong to the technical field of video monitoring and data processing, and are characterized in that a three-level degradation strategy is initiated, a single manufacturer plug-in fault does not influence the overall situation, and comprehensive compatibility of multi-brand equipment and old codes is realized by combining standardized interface mapping; through real-time hardware monitoring and a decision tree model, decoding resources are dynamically allocated, the GPU utilization rate is maximized, CPU occupation is reduced by more than 60%, different configuration terminals are self-adapted, and blurred screen jamming is thoroughly eliminated. The edge gateway integrates code stream adjustment, dual-code stream switching and caching, guarantees multi-channel concurrent stability, adopts synchronous ready, GPU texture synthesis and a compensation algorithm, breaks through the bandwidth bottleneck of multi-channel independent pushing, supports 16-channel synchronous smooth playing, abandons old plug-ins, realizes cross-platform undifferentiated access of a PC, a mobile phone and a browser, and flexibly adapts to internal and external network scenes through dual-mode deployment.

Description

Technical Field

[0001] This invention belongs to the field of video surveillance and data processing technology, specifically relating to a cross-platform video decoding and playback system and method. Background Technology

[0002] Currently, video surveillance systems are widely used in security, IoT, and other fields. Integrated security platforms are intelligent security hubs that integrate video surveillance, alarm linkage, access control, fire monitoring, and intelligent analysis. By integrating multi-dimensional security equipment and data, they achieve an upgrade from "passive monitoring" to "active early warning," and are widely used in government and enterprise, industrial parks, transportation, and community scenarios. Among these, compatibility with video surveillance equipment from multiple vendors is a core function and a major technical challenge. Currently, mainstream surveillance equipment manufacturers have their own video encoding and decoding solutions, which are only used on their respective platforms. When integrated security platforms integrate surveillance equipment from multiple vendors, they face technical bottlenecks such as poor compatibility, low decoding efficiency, video playback delays, stuttering, screen tearing, high network bandwidth requirements, and difficulties in cross-platform adaptation. Specific problems are as follows:

[0003] Compatibility limitations: Each manufacturer relies on proprietary protocols and SDKs, resulting in highly heterogeneous interfaces. During integration, specific adaptation code needs to be developed. Adding new devices or iterating the manufacturer's SDK can easily lead to interface incompatibility. Furthermore, existing solutions either rely solely on standard protocols, resulting in the loss of advanced manufacturer functions, or are bound to proprietary protocols, reducing universality. They also require the manufacturer's video management platform (VMS) for relaying, resulting in poor integration flexibility.

[0004] Insufficient decoding performance: Traditional server-side centralized decoding mode consumes a lot of server resources, and client playback is prone to screen tearing and stuttering. High CPU usage prevents the client from running other services normally. In addition, the existing decoding solution adopts a fixed mode and cannot dynamically allocate resources according to the client's GPU / CPU hardware configuration, resulting in poor adaptability.

[0005] Network transmission issues are prominent: there are problems such as high latency, stuttering, and interruption when transmitting across network segments and public networks, and the network bandwidth is under great pressure when multiple channels are played concurrently; existing technologies mostly optimize caching or bitstream individually without forming a systematic solution, and browser long connection occupation can easily cause performance bottlenecks and limit the number of concurrent channels;

[0006] Cross-platform adaptation is difficult: browser playback relies on outdated plugins such as ActiveX and NPAPI, which modern browsers have phased out; moreover, mobile (iOS / Android / Harmony OS) and PC versions require separate development and adaptation, resulting in high deployment costs and difficulty in achieving unified compatibility.

[0007] In summary, current integrated security platforms face four core pain points when integrating monitoring equipment from multiple vendors: heterogeneous protocols, low decoding efficiency, unstable network transmission, and difficulty in cross-platform adaptation. Existing solutions often employ single technical means: for example, accessing solely through the ONVIF / GB28181 standard protocol results in the loss of advanced vendor features; centralized server-side decoding causes CPU overload, screen tearing, and stuttering in high-concurrency scenarios; or reliance on ActiveX plugins for webpage playback, which is outdated by modern browsers. Furthermore, existing technologies lack a systematic architecture, with each module optimized in isolation, failing to collaboratively address issues such as dynamic fault tolerance, adaptive hardware resource scheduling, low-latency image synthesis, and consistent cross-terminal experience in multi-vendor environments. Summary of the Invention

[0008] To address the shortcomings of the existing technology, this application provides a cross-platform video decoding and playback system and method.

[0009] Firstly, this application proposes a cross-platform video decoding and playback system, applied to an integrated security platform to handle heterogeneous video streams from multiple vendors, including:

[0010] The protocol adaptation layer is configured to connect to monitoring devices from different manufacturers through a unified protocol adaptation module, converting heterogeneous video streams based on proprietary or standard protocols into standardized video streams with a unified format.

[0011] The client-side decoding layer, deployed on the user end, is configured to receive the standardized video stream and includes an intelligent decoding unit. The intelligent decoding unit is used to monitor the client hardware resources in real time and dynamically switch between hardware decoding and software decoding modes according to the hardware resource status.

[0012] The network transmission optimization layer is deployed in the device-side network environment and includes an edge gateway. The edge gateway is configured to aggregate multiple video streams and perform cross-network segment forwarding, while dynamically adjusting the bitstream parameters according to the network bandwidth.

[0013] The playback compositing layer, deployed on the user end, includes a GPU-accelerated compositing engine, configured to synchronize and align multiple decoded video frames and merge them at the pixel level to generate a single composite frame.

[0014] The cross-platform rendering layer is configured to receive the composite image frames on the browser side via the WebSocket protocol and call the WebGL interface for hardware-accelerated rendering.

[0015] In some embodiments, the protocol adaptation layer includes:

[0016] The abstract interface module defines standardized interfaces for stream retrieval, decoding, and control.

[0017] Multiple vendor-specific adapters, each corresponding to a vendor's private SDK, are used to map the calls to the standardized interfaces to the corresponding vendor's private API;

[0018] The dynamic plugin manager is used to dynamically load or unload the manufacturer-specific adapters and automatically trigger a three-level degradation strategy when an adapter decoding failure is detected: first, it switches to RTSP standard protocol decoding; if it still fails, it calls a general software decoder as a fallback.

[0019] In some embodiments, the intelligent decoding unit of the client decoding layer includes:

[0020] The encoding format normalization subunit is used to automatically identify the video stream encoding format and convert non-H.265 / H.264 standard encodings into standard formats in real time;

[0021] The hardware resource monitoring subunit is used to collect client GPU computing power and CPU utilization.

[0022] The decision tree model, based on the output of the hardware resource monitoring subunit, decides whether the current decoding task should be assigned to a hardware decoder or a software decoder.

[0023] In some embodiments, the edge gateway of the network transmission optimization layer is configured as follows:

[0024] Perform dual-stream logic switching: request the camera's sub-stream in real-time preview scenarios and request the main stream in video playback scenarios.

[0025] A built-in cache queue provides cached data to the client to maintain playback continuity when the network is interrupted.

[0026] In some embodiments, the GPU-accelerated compositing engine of the playback compositing layer includes:

[0027] The synchronization controller adopts a full-channel frame ready strategy to align the decoding progress of multiple video streams and enables a dynamic frame compensation algorithm for channels that time out during decoding.

[0028] The GPU memory compositer uses a unified buffer in GPU memory to call the graphics API to draw and stitch multiple YUV format frames as textures at the pixel level.

[0029] An adaptive encoder is used to perform secondary compression encoding on the synthesized single frame and dynamically adjust the encoding bitrate according to network conditions.

[0030] In some embodiments, the cross-platform rendering layer further includes:

[0031] The WebAssembly decoding module is used to perform software decoding on the browser side for received compressed data packets;

[0032] The WebGL rendering pipeline is used to upload decoded RGBA data as GPU textures, and to implement multi-screen splitting, adaptive scaling, and OSD information overlay through shaders.

[0033] In some embodiments, the system includes a dual-mode deployment:

[0034] In client mode, the client decoding layer, playback synthesis layer, and cross-platform rendering layer all run on the user terminal;

[0035] In server-side mode, the client decoding layer and playback synthesis layer run on a cloud server, while the cross-platform rendering layer runs on a mobile browser or a lightweight client.

[0036] Secondly, this application proposes a cross-platform video decoding and playback method, including the following steps:

[0037] By using a protocol adaptation layer to interface with devices from multiple vendors, heterogeneous protocols are converted into standardized video streams;

[0038] The client decoding layer receives the video stream and dynamically selects either hardware decoding or software decoding mode based on local hardware resources.

[0039] Edge gateways aggregate multiple streams and dynamically adjust the bitrate to optimize network transmission;

[0040] The playback compositing layer uses the GPU to synchronously combine multiple decoded frames into a single image.

[0041] The cross-platform rendering layer pushes the composited image to the browser via WebSocket and calls WebGL for rendering.

[0042] In some embodiments, the playback compositing layer synchronously composites multiple decoded frames into a single image using a GPU, including:

[0043] Detect the readiness status of each decoded frame and perform frame rate alignment based on the lowest frame rate channel;

[0044] For channels that are not ready due to timeout, a compensation frame is generated based on the difference between the previous frame and the next frame using an interpolation algorithm.

[0045] The aligned multi-channel YUV frames are uploaded to the GPU memory, and textures are drawn in the shader according to the preset layout to complete pixel-level fusion.

[0046] Thirdly, this application proposes an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0047] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0048] The beneficial effects of this invention are:

[0049] It pioneers a three-level degradation strategy, ensuring that a single vendor's plugin failure does not affect the entire system. Combined with standardized interface mapping, it achieves full compatibility with multiple brands of devices and outdated codecs. Through real-time hardware monitoring and a decision tree model, it dynamically allocates decoding resources, maximizing GPU utilization and reducing CPU usage by over 60%. It also adapts to different terminal configurations, completely eliminating screen tearing and stuttering. The edge gateway integrates bitstream adjustment, dual bitstream switching, and caching to ensure multi-channel concurrent stability. It adopts synchronous readiness, GPU texture synthesis, and compensation algorithms to overcome the bandwidth bottleneck of multi-channel independent push, supporting 16-channel synchronous smooth playback of the WebSocket+WebAssembly+WebGL rendering pipeline. It eliminates outdated plugins, enabling cross-platform, seamless access across PCs, mobile phones, and browsers. Its dual-mode deployment allows for flexible adaptation to internal and external network scenarios. Attached Figure Description

[0050] Figure 1 This is a system structure block diagram of the present invention.

[0051] Figure 2 This is a flowchart of the present invention. Detailed Implementation

[0052] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein; rather, these embodiments are provided so that a more thorough understanding of the invention can be achieved and that the full scope of the invention can be conveyed to those skilled in the art.

[0053] Firstly, this application proposes a cross-platform video decoding and playback system, applied to an integrated security platform to handle heterogeneous video streams from multiple vendors, such as... Figure 1 As shown, it includes:

[0054] The protocol adaptation layer is configured to connect to monitoring devices from different manufacturers through a unified protocol adaptation module, converting heterogeneous video streams based on proprietary or standard protocols into standardized video streams with a unified format.

[0055] In some embodiments, the protocol adaptation layer includes:

[0056] The abstract interface module defines standardized interfaces for stream retrieval, decoding, and control.

[0057] Multiple vendor-specific adapters, each corresponding to a vendor's private SDK, are used to map the calls to the standardized interfaces to the corresponding vendor's private API;

[0058] The dynamic plugin manager is used to dynamically load or unload the manufacturer-specific adapters and automatically trigger a three-level degradation strategy when an adapter decoding failure is detected: first, it switches to RTSP standard protocol decoding; if it still fails, it calls a general software decoder as a fallback.

[0059] The core of the abstract interface module is to design a unified abstract interface and data structure, and to map standard interfaces to vendor private interfaces through adapters, thereby breaking down the technical barriers of vendor private SDKs and achieving complete decoupling between the business layer and the vendor implementation.

[0060] (1) Standardized Abstract Interface Definition: The IVideoDecoder abstract interface class is designed to cover the core operations of the entire video decoding process, including decoder initialization, decoding startup, frame data acquisition, device control, and resource release. The interface parameters and return values ​​adopt standardized definitions and are independent of the manufacturer. The specific interface is as follows:

[0061] InitDecoder(): Receives a standardized DecoderParam structure parameter, completes decoder initialization, and returns a status code (0 = success, non-zero = error code).

[0062] StartDecode(): Starts the decoding task, triggers video stream fetching and decoding, and returns to the start status;

[0063] GetDecodedFrame(): Outputs a normalized FrameBuffer pointer to obtain the decoded video frame data;

[0064] ControlDevice(): Receives standardized DeviceCmd control commands to perform operations such as PTZ control and stream switching;

[0065] ReleaseDecoder(): Releases decoder resources, closes the stream connection, and completes resource reclamation.

[0066] (2) Standardized data structure design: Define the DecoderParam stream decoding parameter structure and the DeviceCmd device control instruction structure, unify parameter format and enumeration values, and cover key elements such as stream address, encoding type, decoding mode, callback function, and control instructions to ensure consistency in parameter transmission across vendors. Among them:

[0067] The DecoderParam structure contains the stream address (uniform format: protocol: / / IP:port / channel), encoding type (CODEC_H264 / CODEC_H265 / CODEC_SVAC enumeration), decoding mode (hardware decoding / software decoding), buffer queue length, frame callback function pointer, and error callback function pointer;

[0068] The DeviceCmd structure contains instruction types (PTZ control / stream switching / preset bit call), instruction parameters, and reserved extended fields to accommodate manufacturers' personalized needs.

[0069] (3) Vendor adapter implementation: For the private SDKs of various vendors such as Hikvision, Dahua, and Uniview, develop corresponding adapter classes (such as HikDecoderAdapter and DahuaDecoderAdapter). The adapter inherits the IVideoDecoder abstract interface and implements the specific mapping logic of the interface methods:

[0070] Parameter mapping: Convert DecoderParam standardized parameters into vendor-specific parameter structures (e.g., convert standard stream addresses into Hikvision NET_DVR_RTPADDR structures).

[0071] Interface mapping: Map the StartDecode() interface call to the vendor's SDK streaming function (such as Hikvision NET_DVR_RealPlay_V40()), and map the ControlDevice() instruction to the vendor's proprietary control function;

[0072] Callback mapping: Encapsulates standardized callback functions into the callback format required by the vendor's SDK to achieve unified reporting of decoded frame data and error information.

[0073] The dynamic plugin manager achieves automated management, version compatibility, and fault isolation of multi-vendor decoding plugins through "configuration driver + dynamic loading + fault tolerance mechanism", thereby improving system stability and scalability.

[0074] Plugin standardization specifications: All vendor decoding plugins exist in the form of dynamic link libraries (DLL / SO), follow a unified plugin specification, and must export fixed functions for the management module to call, including CreateDecoderAdapter() for creating adapter instances, DestroyDecoderAdapter() for destroying instances, and GetPluginInfo() for returning plugin information (including vendor name, version number, and supported encoding formats), to ensure that plugins can be uniformly identified and called.

[0075] Dynamic discovery and loading process: Employing a dual-drive model of "configuration file + directory scanning" to achieve automated plugin loading.

[0076] 1. Predefined configuration: In the plugin_config.xml configuration file, the paths, enabling status, and priorities of each vendor's plugins are preset (priority 1 is the highest and is loaded first).

[0077] 2. Directory Scan: After the system starts, the plugin management module scans the specified plugin directory, compares the consistency between the plugins in the directory and the configuration file, and automatically identifies new plugins that are not configured and marks them as "to be enabled".

[0078] 3. Dynamic loading: Based on configuration priority, plugins are loaded via dlopen() (Linux) / LoadLibrary() (Windows), dlsym() is called to obtain exported function pointers, adapter instances are created, and plugin compatibility is verified.

[0079] Version compatibility and fault tolerance mechanisms:

[0080] Version verification: The plugin management module maintains a list of compatible vendor SDK versions. When loading, it obtains the plugin version through GetPluginInfo() and compares it with the compatibility list. If the versions are incompatible, it automatically downgrades to the backup plugin version or switches to RTSP standard protocol decoding.

[0081] Fault isolation: A thread pool isolation mechanism is adopted, and each plugin runs in an independent thread. The failure of a single plugin (memory overflow, stream retrieval timeout) will not affect other plugins and the core system services.

[0082] Automatic degradation: When plugin decoding fails, a three-level backup strategy is triggered - priority is given to switching to the vendor's RTSP protocol decoding, and if RTSP fails, the platform's general software decoder is called as a backup. At the same time, information is reported through error callback, logs are recorded and alarms are triggered.

[0083] The client-side decoding layer, deployed on the user end, is configured to receive the standardized video stream and includes an intelligent decoding unit. The intelligent decoding unit is used to monitor the client hardware resources in real time and dynamically switch between hardware decoding and software decoding modes according to the hardware resource status.

[0084] In some embodiments, the intelligent decoding unit of the client decoding layer includes:

[0085] The encoding format normalization subunit is used to automatically identify the video stream encoding format and convert non-H.265 / H.264 standard encodings into standard formats in real time;

[0086] The hardware resource monitoring subunit is used to collect client GPU computing power and CPU utilization.

[0087] The decision tree model, based on the output of the hardware resource monitoring subunit, decides whether the current decoding task should be assigned to a hardware decoder or a software decoder.

[0088] Furthermore, the non-standard encoding is converted to the H.265 / H.264 standard format in real time through the encoding format normalization subunit, the dynamic decoding mode switching is realized based on the GPU hardware decoding priority strategy, and the resource allocation is optimized by using intelligent decoding to collect hardware data in real time.

[0089] Encoding format normalization subunit:

[0090] Real-time encoding detection: Employing an improved FFmpeg detection algorithm, the encoding format is quickly identified by analyzing the frame header information and data packet characteristics of the first 100 frames of the video stream. Algorithm optimizations include:

[0091] Parallel frame header analysis: Simultaneously detects multiple key fields, each corresponding to a specific feature in the video stream, and uses comprehensive analysis of these fields to assist in encoding format identification.

[0092] Weighted voting decision: A weighted voting mechanism is adopted to assign corresponding weights to each feature field, and the weighted sum is calculated based on the matching results of the fields to finally determine the encoding format.

[0093] Dynamic transcoding pipeline: Establishes a real-time transcoding channel for non-standard encodings (such as SVAC, MJPEG).

[0094] GPU hardware decoding priority strategy:

[0095] Hardware capability detection: Develop a Vulkan-based cross-platform detection module to evaluate GPU performance through the following steps:

[0096] Queries the set of decoding formats supported by the device;

[0097] Test the decoding latency and calculate the average decoding time of multiple frames as the decoding latency indicator;

[0098] The GPU performance score is calculated based on factors such as the types of supported decoding formats and decoding latency.

[0099] The specific process for dynamically switching decision trees is as follows:

[0100] Starting from the "current frame", the first step is to determine whether the "GPU performance score is higher than the threshold?".

[0101] If the condition is "yes", proceed to the "GPU decoding" step, and then continue to determine "5 consecutive frames timed out?".

[0102] If "Timeout for 5 consecutive frames?" is "yes", then "downgrade to software decoding" will be executed.

[0103] If "5 consecutive frames timed out?" is "no", then further check "Is the CPU load less than 50%?".

[0104] If "CPU load less than 50%?" is "No", then "Enable frame downsampling" will be executed.

[0105] If "CPU load less than 50%?" is "yes", no subsequent branches are shown in the graph.

[0106] If the initial judgment "GPU performance score is higher than the threshold?" is "No", no subsequent branches are shown in the graph.

[0107] This decision tree is used to dynamically adjust the decoding strategy based on GPU performance, decoding timeout, and CPU load.

[0108] Real-time monitoring indicator system:

[0109] Decoding queue depth: Real-time monitoring of the number of frames to be processed by each decoder, i.e., the decoding queue depth;

[0110] Hardware utilization matrix: Monitor hardware utilization to form a hardware resource utilization matrix, and record the resource utilization of each hardware unit.

[0111] Dynamic scheduling based on reinforcement learning:

[0112] A state space is defined, containing information such as the current hardware resource status and decoding queue depth; the action space includes operations such as switching decoding modes and adjusting resolution. The DQN algorithm is used to optimize the scheduling strategy, and a reward function is designed to guide model learning. The reward function comprehensively considers decoding latency and CPU load, providing a positive reward when the actual decoding time is close to the target time and the CPU utilization is below the maximum allowable value; otherwise, a penalty is imposed. Through continuous iterative optimization, the scheduling strategy can adaptively select the optimal decoding resource configuration.

[0113] Taking the access of Dahua 4K H.265 video stream as an example:

[0114] 1. The encoding detection module identifies the "DH265" proprietary encoding, triggering the transcoding process:

[0115] Start the FFmpeg filter chain: avcodec -> dh265 decoding -> x265 encoding;

[0116] Transcoding parameters: Maintain 4K resolution, CRF=23, preset=medium.

[0117] 2. Hardware testing shows that the NVIDIA RTX 3060 supports HEVC 4K decoding:

[0118] Calculate the GPU performance score, taking into account factors such as decoding latency and supported formats;

[0119] Select the GPU decoding mode and initialize the decoder via the CUVID interface.

[0120] 3. When decoding 8 video streams simultaneously:

[0121] The monitoring showed that the decoding queue depth reached 15, and the GPU load was close to 90%.

[0122] The DQN model decision converts the two channels to CPU decoding and adjusts the resolution to 1080p.

[0123] Calculate new reward values ​​in real time to verify the effectiveness of the strategy.

[0124] The network transmission optimization layer is deployed in the device-side network environment and includes an edge gateway. The edge gateway is configured to aggregate multiple video streams and perform cross-network segment forwarding, while dynamically adjusting the bitstream parameters according to the network bandwidth.

[0125] In some embodiments, the edge gateway of the network transmission optimization layer is configured as follows:

[0126] Perform dual-stream logic switching: request the camera's sub-stream in real-time preview scenarios and request the main stream in video playback scenarios.

[0127] A built-in cache queue provides cached data to the client to maintain playback continuity when the network is interrupted.

[0128] Among them, a dynamic adaptation mechanism for bitstream parameters is adopted in combination with dual bitstream function to distinguish application scenarios, transmission latency is reduced by convergence and forwarding through edge gateway, and video stream buffer queue is deployed to improve transmission stability.

[0129] The implementation scheme of the network transmission optimization layer is as follows:

[0130] Network status monitoring module:

[0131] Real-time round-trip latency is collected, and the average latency is calculated using a sliding window.

[0132] Packet loss rate is calculated using an exponentially weighted moving average method to smooth out network fluctuations;

[0133] Bandwidth estimation estimates the currently available bandwidth based on the amount of data successfully transmitted and the time intervals between transmissions.

[0134] Bitrate adjustment algorithm: An improved TCP-Friendly Rate Control (TFRC) algorithm is adopted to dynamically calculate the appropriate transmission bitrate based on parameters such as packet loss rate, round-trip time, and packet size. A QoE weighting factor is introduced to distinguish the priority of I-frames and P-frames. At the same time, a bandwidth prediction module is added to predict future bandwidth using the ARIMA model.

[0135] Dual-stream intelligent switching: Scene recognition:

[0136] Real-time analysis of inter-frame differences is performed by comparing the pixel value changes between the current frame and the previous frame to calculate the pixel difference ratio. Based on this, motion intensity is evaluated by smoothing the inter-frame differences using an exponential weighted average method to obtain the motion intensity value at the current moment, which is used to determine the intensity of motion in the scene.

[0137] Key technology implementation:

[0138] Intelligent routing selection: Intelligent routing selection is based on Dijkstra's algorithm. The cost function comprehensively considers the round-trip time, packet loss rate and jitter of the link, and adjusts the importance of each indicator through weight coefficients to select the path with the lowest overall cost for data transmission.

[0139] Protocol conversion: Enables real-time conversion from ONVIF / GB28181 to RTSP;

[0140] Cache management: The LRU-K algorithm is used to cache the video segments that have been accessed the most recently (K times).

[0141] Video stream caching mechanism: Hierarchical caching architecture:

[0142] Edge gateway cache: Stores the most recent 5 minutes of video using a circular buffer;

[0143] Client-side caching: Stores the most recent minute of video for fast seek;

[0144] Cache replacement algorithm: An improved ARC cache replacement algorithm is adopted, which maintains a recently accessed list and a frequently accessed list, and is supplemented by a ghost list to record evicted entries, and dynamically adjusts the caching strategy to improve the hit rate.

[0145] The playback compositing layer, deployed on the user end, includes a GPU-accelerated compositing engine, configured to synchronize and align multiple decoded video frames and merge them at the pixel level to generate a single composite frame.

[0146] In some embodiments, the GPU-accelerated compositing engine of the playback compositing layer includes:

[0147] The synchronization controller adopts a full-channel frame ready strategy to align the decoding progress of multiple video streams and enables a dynamic frame compensation algorithm for channels that time out during decoding.

[0148] The GPU memory compositer uses a unified buffer in GPU memory to call the graphics API to draw and stitch multiple YUV format frames as textures at the pixel level.

[0149] An adaptive encoder is used to perform secondary compression encoding on the synthesized single frame and dynamically adjust the encoding bitrate according to network conditions.

[0150] After achieving simultaneous decoding of 16 video streams, a GPU-accelerated compositing scheme is adopted. Through frame rate adaptive alignment and dynamic compensation technology, combined with WebSocket full-duplex transmission and WebGL accelerated rendering, smooth playback is ensured.

[0151] The implementation scheme for low-latency, high-smoothness playback optimization technology is as follows:

[0152] 1. Multi-channel video synchronous decoding: An improved frame synchronization scheduling algorithm is adopted. The synchronization deviation is evaluated by calculating the standard deviation of the decoding time of each channel, and a dynamic adjustment coefficient is introduced to adjust the synchronization waiting threshold in real time according to the fluctuation of decoding time, so as to achieve frame-level synchronization of multiple video channels.

[0153] Implementation steps:

[0154] Create a separate decoding thread for each video stream;

[0155] Real-time monitoring of decoding time for each thread;

[0156] Frame alignment is achieved through a synchronization fence.

[0157] Enable frame dropping policy for timeout channels.

[0158] 2. Frame Compression and Stitching / Merging: The decoded multi-channel video frame images are efficiently compressed, and then stitched together pixel-by-pixel according to a preset layout (e.g., 1×N, M×N grid) to form a single complete video frame. The size of the synthesized image is controlled to ≤100KB to reduce data transmission. As a core component for achieving high efficiency, energy saving, and high concurrency, this invention adopts an integrated solution of "synchronous ready stitching + GPU accelerated synthesis + adaptive secondary encoding." While ensuring low latency and no tearing, it merges multiple video frames into a single frame for transmission to reduce connection overhead. The specific implementation is as follows:

[0159] (1) Design of splicing timing, process and synchronization mechanism:

[0160] The system employs a "unified stitching after all channels are ready" strategy to avoid screen tearing caused by real-time stitching of single frames. It also balances waiting latency through dynamic buffering and frame rate adaptation mechanisms. In terms of workflow, each video stream is allocated an independent frame buffer queue. After each stream decodes one frame, it is stored in the corresponding queue and marked as "ready." The synchronization controller monitors the status of all channel buffer queues in real time. When all channels have ready frames, the stitching process is triggered, and all ready frames are extracted at once. After stitching is complete, the corresponding frame buffers are cleared, and the ready status is reset. To address the issue of varying frame rates across multiple video streams (e.g., some channels at 25fps and others at 30fps) and differences in decoding time, a two-level synchronous optimization mechanism is designed: First, adaptive frame rate alignment, using the lowest frame rate channel as a benchmark, achieves frame timing alignment through frame duplication or discarding strategies (discarding redundant frames in high frame rate channels and duplicating the latest frame in low frame rate channels) to avoid continuous waiting; Second, dynamic compensation for decoding time, predicting the decoding cycle of a single stream based on historical decoding time data, starting the decoding task in advance for slower channels, and setting a maximum waiting threshold (≤10ms). After the timeout, the latest ready frame is used to replace the unready frame, ensuring that the overall latency is controllable and there is no screen tearing.

[0161] (2) Specific algorithm integration and GPU acceleration optimization:

[0162] It prioritizes GPU texture manipulation combined with graphics APIs (DirectX 12 / OpenGL) to achieve efficient compositing, making full use of the client's GPU computing power, improving compositing efficiency by more than 70% compared to CPU memory copying, while reducing CPU usage. Specific memory and operation process: ① Decoded single-channel video frames (YUV420 format) are first quickly transferred from system memory to GPU memory via GPU memory mapping to avoid cross-memory copying losses; ② A unified compositing buffer is opened in GPU memory, and the coordinate regions of each video frame are divided according to a preset layout (such as 4×4 grid, 2×8 grid, and custom configuration is supported) to clarify the position and size of each frame in the composite image; ③ The GPU texture rendering pipeline is called to bind the frame data as texture resources to the rendering target, and the fragment shader draws the data to the compositing buffer one by one according to the coordinate regions, while simultaneously completing the format conversion of YUV components (adapting to GPU rendering format) and edge smoothing to avoid splicing gaps; ④ If the client's GPU computing power is insufficient (such as integrated graphics card), it automatically degrades to a CPU-optimized compositing scheme: a continuous large buffer is opened in CPU memory, and block copying is performed according to the three YUV components. First, the Y component is copied to the corresponding coordinate region, and then the U and V components are copied in sequence (utilizing the continuity of component data to optimize copying efficiency). At the same time, the SIMD instruction set is enabled to accelerate memory operations and reduce compositing time.

[0163] 3. Post-synthesis reprocessing strategy:

[0164] The stitched large-screen frames adopt an "adaptive secondary encoding + screen content optimization" scheme to achieve a balance between image quality and transmission latency, while also adapting to the characteristics of the stitched images. In terms of encoding, the default encoding standard is H.265 (HEVC). Considering the characteristics of composite images consisting of multiple independent sub-images with clear edge contours and concentrated color variations, Screen Content Coding (SCC) extension is enabled. This reuses repeated content within sub-images through intra-block copy (IBC) mode, improving the encoding compression ratio. Encoding parameters dynamically adapt to scene requirements: the bitrate is adaptively adjusted according to the composite image resolution (bitrate controlled at 2-3Mbps for 1080P composite images and 8-10Mbps for 4K composite images). The GOP structure adopts a short-cycle structure of "1 I-frame + 8 P-frames," with an I-frame interval ≤300ms, ensuring image fidelity (image quality loss ≤5%) while reducing transmission latency caused by inter-frame dependencies. For scenarios extremely sensitive to latency, such as emergency command, a one-click switch to no-encoding mode is supported. After splicing, the original composite frames are directly compressed without loss (LZ4 algorithm), achieving a compression ratio of up to 3:1, further shortening the processing cycle and ensuring minimal end-to-end latency.

[0165] The cross-platform rendering layer is configured to receive the composite image frames on the browser side via the WebSocket protocol and call the WebGL interface for hardware-accelerated rendering.

[0166] In some embodiments, the cross-platform rendering layer further includes:

[0167] The WebAssembly decoding module is used to perform software decoding on the browser side for received compressed data packets;

[0168] The WebGL rendering pipeline is used to upload decoded RGBA data as GPU textures, and to implement multi-screen splitting, adaptive scaling, and OSD information overlay through shaders.

[0169] WebSocket protocol optimization:

[0170] Packet segmentation strategy: An optimized packet segmentation strategy is adopted, which divides each frame of data into multiple data packets of appropriate size according to the network maximum transmission unit (MTU) for transmission, and marks I frames with high priority to ensure that key frames are delivered first.

[0171] Browser compatibility: The core decoder is encapsulated using WebAssembly technology to generate .wasm binary units;

[0172] Develop a JavaScript binding layer and implement the following functionalities using the Emscripten toolchain:

[0173] Memory management: Establish a SharedArrayBuffer shared memory area;

[0174] Thread communication: Multi-threaded decoding is implemented using Web Workers;

[0175] Hardware acceleration: Access GPU resources via the WebGL 2.0 API.

[0176] WebGL rendering pipeline: The specific steps are as follows:

[0177] Input video frames;

[0178] Preprocessing is performed using WebWorker;

[0179] Use WebSocket to transmit processed data;

[0180] Use WASM to decode video data;

[0181] Rendering is performed using WebGL;

[0182] The output is finally displayed on a Canvas.

[0183] In some embodiments, the system includes a dual-mode deployment:

[0184] In client mode, the client decoding layer, playback synthesis layer, and cross-platform rendering layer all run on the user terminal;

[0185] In server-side mode, the client decoding layer and playback synthesis layer run on a cloud server, while the cross-platform rendering layer runs on a mobile browser or a lightweight client.

[0186] The dual-mode deployment engine includes:

[0187] 1. Client-side mode:

[0188] The local installer contains a complete set of decoders;

[0189] Automatically detect hardware configuration and generate the optimal decoding scheme;

[0190] Communicates with the browser process via the IPC channel.

[0191] 2. Server-side mode:

[0192] Deploy a cloud-based transcoding cluster;

[0193] The bitrate is dynamically adjusted using ABR (Adaptive Bitrate Streaming) technology;

[0194] Implement the conversion between WebRTC signaling protocol and HLS / DASH streaming media protocol.

[0195] Unlike existing solutions that rely on separate development platforms and plugin dependencies, this invention develops a cross-platform suite through unified decoding kernel encapsulation. It is compatible with Windows / macOS / Linux desktop systems and mainstream browsers, completely eliminating outdated plugins. It supports dual deployment modes for clients and servers. Client-side deployment reduces the load on the server side, while server-side deployment performs decoding adaptation and format push for mobile devices, covering all mobile devices including iOS / Android / Harmony OS. Multi-platform deployment costs are reduced by 50%, and browser compatibility limitations are avoided.

[0196] Secondly, this application proposes a cross-platform video decoding and playback method, such as... Figure 2 As shown, it includes the following steps:

[0197] Step S1: Connect to devices from multiple vendors through the protocol adaptation layer to convert heterogeneous protocols into standardized video streams;

[0198] Step S2: The client decoding layer receives the video stream and dynamically selects either hardware decoding or software decoding mode based on local hardware resources;

[0199] Step S3: The edge gateway aggregates multiple streams and dynamically adjusts the bitrate to optimize network transmission;

[0200] Step S4: The playback composite layer uses the GPU to synchronously synthesize multiple decoded frames into a single image;

[0201] Step S5: The cross-platform rendering layer pushes the composited image to the browser via WebSocket and calls WebGL for rendering.

[0202] In some embodiments, the playback compositing layer synchronously composites multiple decoded frames into a single image using a GPU, including:

[0203] Detect the readiness status of each decoded frame and perform frame rate alignment based on the lowest frame rate channel;

[0204] For channels that are not ready due to timeout, a compensation frame is generated based on the difference between the previous frame and the next frame using an interpolation algorithm.

[0205] The aligned multi-channel YUV frames are uploaded to the GPU memory, and textures are drawn in the shader according to the preset layout to complete pixel-level fusion.

[0206] This embodiment uses the intelligent integrated security platform deployed at the Beijing Benz Shunyi plant as an example to detail the specific implementation process of the cross-platform video decoding and playback system and method provided by this invention. The plant covers a large area with a high density of monitoring equipment, including cameras from multiple manufacturers such as Huawei, Unisplendour, Hikvision, and Axis Communications, with each manufacturer's equipment using either proprietary protocols or a mix of standard protocols for access. The platform needs to achieve integrated management and control of all monitoring equipment throughout the plant, support users to simultaneously preview 16 video streams on their office PC browsers, and enable intelligent linkage for events such as perimeter intrusion and fire alarms.

[0207] In this embodiment, the system is deployed in a client-side mode. The protocol adaptation layer and network transmission optimization layer are deployed on the central server room and edge gateways within the factory, while the client-side decoding layer, playback synthesis layer, and cross-platform rendering layer all run on the user's office PC. Users can access the platform through the Chrome browser and enjoy a low-latency, high-quality video preview experience without installing any plugins.

[0208] Step 1: The protocol adaptation layer interfaces with devices from multiple vendors to convert heterogeneous video streams into standardized video streams;

[0209] Users log in to the platform via a browser and select 16 cameras (including several from Hikvision, Huawei, and Unisplendour) in area A of the factory for real-time preview. Upon receiving the request, the platform's backend protocol adaptation server initiates the protocol adaptation process.

[0210] First, the dynamic plugin manager dynamically loads the corresponding manufacturer-specific adapter from a pre-defined plugin library based on the camera manufacturer information. For example, for Hikvision cameras, the HikDecoderAdapter dynamic link library is loaded; for Huawei cameras, the HuaweiDecoderAdapter is loaded. Each adapter implements a unified IVideoDecoder abstract interface, mapping standardized streaming parameters to the parameter structure required by the manufacturer's private SDK.

[0211] Taking a Hikvision camera as an example, the adapter performs the following operations:

[0212] The decoder is initialized through the vendor's SDK by passing in standardized parameters (such as stream address, encoding type H.265, and decoding mode auto), and the SDK returns a decoding handle.

[0213] Calling StartDecode() initiates the streaming process, and the SDK returns the raw video frame data via a callback function.

[0214] The adapter encapsulates proprietary format frame data into a standardized FrameBuffer structure (containing timestamps, encoding formats, data pointers, etc.) and pushes it to subsequent processing modules.

[0215] If a camera fails to decode due to incompatibility with the manufacturer's SDK version, the dynamic plugin manager automatically triggers a three-level degradation strategy:

[0216] First, try switching to the manufacturer's RTSP standard protocol decoding and obtaining the stream via the ONVIF / GB28181 standard;

[0217] If RTSP still fails, the platform's built-in general-purpose FFmpeg software decoder will be used as a fallback.

[0218] At the same time, error logs are recorded and alerts are reported to the system so that maintenance personnel can handle the issue.

[0219] Through the above mechanism, the raw video streams from all 16 cameras are converted into real-time video streams in a unified H.264 / H.265 standard format and sent to the client decoding layer via an internal message queue.

[0220] Step 2: The client decoding layer receives the video stream and dynamically selects the decoding mode based on local hardware resources;

[0221] The user's office PC (configured with an Intel i7 CPU, NVIDIA RTX 3060 GPU, and 16GB of RAM) establishes a connection with the platform server via WebSocket and receives 16 standardized video streams. The intelligent decoding unit of the client decoding layer begins operation.

[0222] First, the encoding format normalization subunit performs rapid detection on each video stream: by analyzing the frame header information of the first 100 frames, it identifies the encoding format. If a non-standard encoding (such as a vendor's proprietary encoding) is detected, a real-time transcoding pipeline is started, using FFmpeg to convert it to the H.265 standard format, maintaining the original resolution with CRF=23 and preset=medium transcoding parameters.

[0223] Simultaneously, the hardware resource monitoring subunit collects GPU computing power and CPU utilization in real time. Initially, the GPU is idle, and the CPU utilization is approximately 15%. The decision tree model makes a judgment based on the current hardware state:

[0224] The GPU performance score P_GPU (calculated based on Vulkan detection of decoding latency and supported formats) is much higher than the threshold, so the decision is "GPU decoding".

[0225] Therefore, for most video streams, the intelligent decoding unit calls the NVIDIA CUVID hardware decoder for hardware decoding, the GPU load rises to 40%, and the CPU utilization remains low.

[0226] After all 16 video streams were started, the GPU load was detected to be close to 90%, and decoding timed out for 5 consecutive frames. The decision tree triggered a degradation strategy: switching 4 of the video streams with less motion to CPU software decoding, and simultaneously enabling frame downsampling (reducing the resolution to 720P) to balance overall resources. After the switch, the GPU load dropped to 70%, the CPU utilization rose to 30%, and the system ran stably.

[0227] Step 3: The network transmission optimization layer aggregates multiple streams through the edge gateway and dynamically adjusts the bitrate;

[0228] In the factory network environment, multiple edge gateways are deployed to aggregate camera video streams and forward them across network segments. In this embodiment, the 16 cameras requested by the user are distributed across different network segments. The edge gateways select the optimal path through intelligent routing (based on a cost function improved from Dijkstra's algorithm, taking into account latency, bandwidth, and packet loss rate), aggregate the camera streams, and forward them to the client.

[0229] The edge gateway has a built-in network status monitoring module that calculates round-trip time (RTT), packet loss rate (p), and available bandwidth (B_est) in real time. It also uses an improved TCP-Friendly Rate Control algorithm to dynamically adjust bitstream parameters.

[0230] When network conditions are good (RTT<50ms, p<0.1%), maintain a high bitrate for the main stream (e.g., 8Mbps for 4K video).

[0231] When network congestion is detected (RTT>150ms, p>1%), the bit rate is automatically reduced to 2Mbps, and I-frame transmission is prioritized.

[0232] Meanwhile, the edge gateway automatically switches between dual bitstreams based on the scenario: when the current scenario is real-time preview, the gateway requests a sub-bitstream (such as 720P) from the camera to reduce bandwidth usage; if the user switches to the video playback scenario, the gateway requests the main bitstream (such as 4K) to ensure playback quality.

[0233] To cope with network interruptions, the edge gateway has a built-in circular buffer that caches the most recent 5 minutes of video data. When the network is temporarily interrupted, the client reads data from the gateway's cache to maintain continuous playback and prevent the screen from freezing.

[0234] Step 4: The playback compositing layer uses the GPU to synchronously combine multiple decoded frames into a single image;

[0235] The 16 decoded frames (YUV420 format) output from the client decoding layer are sent to the playback compositing layer. The GPU-accelerated compositing engine then begins operation.

[0236] First, the synchronization controller employs a "full-channel frame-ready strategy" to align the decoding progress of each channel. An independent frame buffer queue is allocated to each video channel, and the status of each queue is monitored in real time. Since the decoding time varies among channels (hardware decoding is faster, software decoding is slower), the synchronization controller dynamically calculates a waiting threshold (≤10ms) based on the channel with the lowest frame rate. If a channel times out and is not ready, a dynamic frame compensation algorithm is activated: based on the difference between the previous and next frames of that channel, a compensation frame is generated through interpolation to ensure a tear-free composite image.

[0237] Once all 16 channels have ready frames (or timeout compensation frames), the GPU compositing process is triggered:

[0238] The 16 YUV frames are quickly uploaded to the GPU memory via GPU memory mapping, avoiding CPU memory copying.

[0239] A unified compositing buffer is created in the GPU memory, and a corresponding area is allocated to each frame according to the preset 4×4 grid layout.

[0240] The OpenGL graphics API is called to bind each frame as a texture, and the fragment shader draws it step by step according to the coordinate region. At the same time, the format conversion from YUV to RGBA and edge smoothing are completed.

[0241] The composited single frame (4K resolution) is stored in video memory.

[0242] Subsequently, the adaptive encoder performs secondary compression on the composite frame. Considering that the composite image consists of multiple sub-frames with clear edge contours, H.265 Screen Content Coding (SCC) extension is enabled, and an intra-block copy mode is used to improve the compression ratio. The encoding bitrate is dynamically adjusted according to the current network conditions: in this embodiment, the network is good, the bitrate is set to 8Mbps, the GOP structure is 1 I-frame + 8 P-frames, the I-frame interval is 300ms, ensuring that the image quality loss is ≤5%. For high real-time scenarios such as emergency command, the lossless compression mode (LZ4 algorithm) can be switched, with a compression ratio of approximately 3:1, further reducing latency.

[0243] Step 5: The cross-platform rendering layer pushes the composited image to the browser via WebSocket and calls WebGL for rendering;

[0244] The encoded composite image frames are pushed to the user's browser via the WebSocket protocol. The WebSocket connection employs an optimized packet segmentation strategy, dividing each frame of data into multiple packets of MTU size and marking I-frames with high priority to ensure that key frames are transmitted first.

[0245] The browser pre-loads a WebAssembly-based decoding module (compiled from a C++ decoding library using Emscripten). Upon receiving compressed data, the Web Worker thread starts the WASM decoder for software decoding, outputting raw frame data in RGBA format. The decoded frame is then passed to the main thread via a SharedArrayBuffer shared memory.

[0246] The main thread calls the WebGL 2.0 API to create the rendering pipeline:

[0247] Upload RGBA data as a GPU texture.

[0248] Multi-screen segmentation is achieved through vertex shaders and fragment shaders (adaptively scaling according to the actual layout), and OSD information (such as camera name, timestamp, alarm indicator, etc.) is overlaid.

[0249] Finally, the video is rendered onto a Canvas element and displayed as a smooth 16-segment video.

[0250] The actual end-to-end latency was less than 0.5ms, with no stuttering or screen tearing. Users can preview 16 video feeds simultaneously through a browser and perform operations such as PTZ control and video playback. When the perimeter intrusion detection system triggers an alarm, the platform automatically displays the camera feed for that area, achieving a closed loop of "alarm-video-response".

[0251] Dual-mode deployment alternatives:

[0252] This embodiment adopts a client-side mode, where all decoding, compositing, and rendering are completed on the user's PC, significantly reducing server load (by 60% compared to traditional solutions). For mobile access (such as iOS / Android devices), the platform can switch to a server-side mode: the decoding and compositing layers are deployed on a cloud server to generate a lightweight H.264 stream, which is then pushed to the mobile browser via WebRTC, achieving a consistent cross-platform experience and reducing multi-device deployment costs by more than 50%.

[0253] In summary, this embodiment fully demonstrates the workflow of the system and method of the present invention in the actual application at the Beijing Benz Shunyi plant. The collaborative work of each layer effectively solves core technical problems such as multi-vendor compatibility, decoding performance, network transmission and cross-platform rendering, and significantly improves the intelligence level of security management and user experience.

[0254] Thirdly, this application proposes an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0255] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0256] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0257] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0258] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0259] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0260] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0261] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0262] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0263] The above are merely preferred embodiments of the present invention. It should be noted that any modifications and improvements made by those skilled in the art without departing from the present technical solution should also be considered to fall within the scope of protection claimed by the present solution.

Claims

1. A cross-platform video decoding and playback system, applied to an integrated security platform to process heterogeneous video streams from multiple vendors, characterized in that, include: The protocol adaptation layer is configured to connect to monitoring devices from different manufacturers through a unified protocol adaptation module, converting heterogeneous video streams based on proprietary or standard protocols into standardized video streams with a unified format. The client-side decoding layer, deployed on the user end, is configured to receive the standardized video stream and includes an intelligent decoding unit. The intelligent decoding unit is used to monitor the client hardware resources in real time and dynamically switch between hardware decoding and software decoding modes according to the hardware resource status. The network transmission optimization layer is deployed in the device-side network environment and includes an edge gateway. The edge gateway is configured to aggregate multiple video streams and perform cross-network segment forwarding, while dynamically adjusting the bitstream parameters according to the network bandwidth. The playback compositing layer, deployed on the user end, includes a GPU-accelerated compositing engine, configured to synchronize and align multiple decoded video frames and merge them at the pixel level to generate a single composite frame. The cross-platform rendering layer is configured to receive the composite image frames on the browser side via the WebSocket protocol and call the WebGL interface for hardware-accelerated rendering.

2. The system according to claim 1, characterized in that: The protocol adaptation layer includes: The abstract interface module defines standardized interfaces for stream retrieval, decoding, and control. Multiple vendor-specific adapters, each corresponding to a vendor's private SDK, are used to map the calls to the standardized interfaces to the corresponding vendor's private API; The dynamic plugin manager is used to dynamically load or unload the manufacturer-specific adapters and automatically trigger a three-level degradation strategy when an adapter decoding failure is detected: first, it switches to RTSP standard protocol decoding; if it still fails, it calls a general software decoder as a fallback.

3. The system according to claim 2, characterized in that: The intelligent decoding unit of the client decoding layer includes: The encoding format normalization subunit is used to automatically identify the video stream encoding format and convert non-H.265 / H.264 standard encodings into standard formats in real time; The hardware resource monitoring subunit is used to collect client GPU computing power and CPU utilization. The decision tree model, based on the output of the hardware resource monitoring subunit, decides whether the current decoding task should be assigned to a hardware decoder or a software decoder.

4. The system according to claim 3, characterized in that: The edge gateway of the network transmission optimization layer is configured as follows: Perform dual-stream logic switching: request the camera's sub-stream in real-time preview scenarios and request the main stream in video playback scenarios. A built-in cache queue provides cached data to the client to maintain playback continuity when the network is interrupted.

5. The system according to claim 4, characterized in that: The GPU-accelerated compositing engine for the playback compositing layer includes: The synchronization controller adopts a full-channel frame ready strategy to align the decoding progress of multiple video streams and enables a dynamic frame compensation algorithm for channels that time out during decoding. The GPU memory compositer uses a unified buffer in GPU memory to call the graphics API to draw and stitch multiple YUV format frames as textures at the pixel level. An adaptive encoder is used to perform secondary compression encoding on the synthesized single frame and dynamically adjust the encoding bitrate according to network conditions.

6. The system according to claim 5, characterized in that: The cross-platform rendering layer further includes: The WebAssembly decoding module is used to perform software decoding on the browser side for received compressed data packets; The WebGL rendering pipeline is used to upload decoded RGBA data as GPU textures, and to implement multi-screen splitting, adaptive scaling, and OSD information overlay through shaders.

7. The system according to claim 6, characterized in that: The system includes a dual-mode deployment: In client mode, the client decoding layer, playback synthesis layer, and cross-platform rendering layer all run on the user terminal; In server-side mode, the client decoding layer and playback synthesis layer run on a cloud server, while the cross-platform rendering layer runs on a mobile browser or a lightweight client.

8. A cross-platform video decoding and playback method based on the system according to any one of claims 1 to 7, characterized in that: Includes the following steps: By using a protocol adaptation layer to interface with devices from multiple vendors, heterogeneous protocols are converted into standardized video streams; The client decoding layer receives the video stream and dynamically selects either hardware decoding or software decoding mode based on local hardware resources. Edge gateways aggregate multiple streams and dynamically adjust the bitrate to optimize network transmission; The playback compositing layer uses the GPU to synchronously combine multiple decoded frames into a single image. The cross-platform rendering layer pushes the composited image to the browser via WebSocket and calls WebGL for rendering.

9. The method according to claim 8, characterized in that: The playback compositing layer uses the GPU to synchronously synthesize multiple decoded frames into a single image, including: Detect the readiness status of each decoded frame and perform frame rate alignment based on the lowest frame rate channel; For channels that are not ready due to timeout, a compensation frame is generated based on the difference between the previous frame and the next frame using an interpolation algorithm. The aligned multi-channel YUV frames are uploaded to the GPU memory, and textures are drawn in the shader according to the preset layout to complete pixel-level fusion.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in claim 8 or 9.

Images