A homologous multiprotocol multiplexing distribution system, method, device and storage medium

Abstract

The application provides a homologous multi-protocol multiplexing distribution system, method, device and storage medium, comprising: a device access module, configured to interact with a streaming media source device, pull an original audio and video stream, and maintain a unique data source instance; a plurality of protocol access micro-modules, each corresponding to a media protocol, independently deployed, configured to subscribe to the data source instance of the device access module, and distribute the original audio and video data received from the device access module to a client after encapsulation according to the respective corresponding media protocol; and a media gateway module, configured to provide an application program interface to the outside, to perform user authentication in response to a request of the client, and return an access address of the protocol access micro-module corresponding to the requested media protocol to the client. The above scheme can improve the running stability, scalability and management efficiency of the system.

Description

Technical Field

[0001] This application relates to the field of streaming media technology, and more specifically, to a homogeneous multi-protocol multiplexing and distribution system, method, device, and storage medium. Background Technology

[0002] This invention relates to the field of streaming media transmission technology. With the popularization of real-time audio and video services such as security monitoring and live streaming, streaming media systems need to be compatible with multiple media transmission protocols to adapt to the playback needs of multiple terminals and multiple scenarios, which places higher demands on the stability and scalability of the system.

[0003] Currently, mainstream streaming media services adopt a monolithic integrated architecture, represented by open-source services such as Nginx-RTMP and SRS. These services integrate all the functions of the entire chain, such as device connection, source stream retrieval, and multi-protocol encapsulation and distribution, into a single service process, and all client playback requests are handled by this single process.

[0004] The core flaw of this architecture lies in the deep coupling of protocol modules. Upgrading or failing a single protocol module can affect the stable operation of the entire service. It cannot achieve protocol-level fault isolation, nor can it scale up or down independently for different protocol loads, resulting in insufficient system availability and scalability. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a system, method, device and storage medium for multiplexing and distributing multiple protocols from the same source, which can improve the system's operational stability, scalability and management efficiency.

[0006] In a first aspect, embodiments of this application provide a homogeneous multi-protocol multiplexing and distribution system, including: The device access module is used to interact with streaming media source devices, retrieve raw audio and video streams, and maintain a unique data source instance; Multiple protocol access micro-modules are deployed independently, each corresponding to a media protocol. They are used to subscribe to the data source instance from the device access module and encapsulate the raw audio and video data received from the device access module according to their respective media protocols before distributing it to the client. The media gateway module is used to provide an application programming interface to respond to client requests for user authentication and return to the client the access address of the protocol access micro-module corresponding to the requested media protocol. The device access module includes: A circular buffer is used to store raw audio and video frames for a predetermined duration; The subscription management module is used to maintain a list of subscribers, where each subscriber in the list corresponds to a session of a protocol access micro-module; The data distribution module is used to traverse the subscriber list maintained by the subscription management module and distribute the new data to the protocol access micro-module corresponding to each subscriber when new data arrives from the data source.

[0007] Optionally, the device access module is configured to check whether a complete I-frame exists in the circular buffer when a new session is subscribed; The device access module is configured to, if present, prioritize pushing the complete I-frame and subsequent frames to the corresponding protocol access micro-module; if not present, trigger the forced I-frame request module.

[0008] Optionally, the forced I-frame request module is used to: When there is no complete I-frame in the circular buffer, a forced I-frame command is sent to the streaming media source device, and the received forced I-frame is stored in the circular buffer.

[0009] Optionally, the media gateway module is further configured to: Receive a protocol switching request initiated by the client, carrying the original session identifier; The corresponding data source instance is located based on the original session identifier, and a new access address under the new protocol is generated and returned to the client; The new access address allows the client to continue receiving data from the original playback progress.

[0010] Optionally, the data source instance maintained by the device access module is uniquely identified by a combination of device serial number, channel number, and stream type; The device access module is used to determine whether the requested data source already exists based on the unique identifier. If it does not exist, a new instance is created. If it already exists, the existing instance is reused, and the latest keyframe in the circular buffer is pushed to the newly subscribed protocol access micro-module.

[0011] Optionally, the plurality of protocol access micro-modules include at least two of the following: FLV micro-module, HLS micro-module, RTSP micro-module, and WebSocket micro-module; The FLV micro-module is used to package the raw audio and video data into FLV format; The HLS micro-module is used to package the data into TS or fMP4 slice format; The RTSP micro-module is used to package it into an RTP packet; The WebSocket micro-module is used for distribution after encapsulation via the WebSocket protocol.

[0012] Secondly, embodiments of this application provide a method for homogeneous multi-protocol multiplexing and distribution, applied to a homogeneous multi-protocol multiplexing and distribution system. The system includes a device access module, multiple protocol access micro-modules, and a media gateway module. Each protocol access micro-module corresponds to a media protocol and is deployed independently. The method includes: The device access module interacts with the streaming media source device, pulls the original audio and video streams, and maintains a unique data source instance; Each protocol access micro-module subscribes to the data source instance from the device access module, and encapsulates the raw audio and video data received from the device access module according to its corresponding media protocol before distributing it to the client; The media gateway module provides an application programming interface to respond to client requests for user authentication and return to the client the access address of the protocol access micro-module corresponding to the requested media protocol. The device access module includes a ring buffer, a subscription management module, and a data distribution module; the method further includes: The circular buffer stores raw audio and video frames for a predetermined time period; The subscription management module maintains a subscriber list, and each subscriber in the subscriber list corresponds to a session of a protocol access micro-module; When new data arrives from the data source, the data distribution module iterates through the subscriber list maintained by the subscription management module and distributes the new data to the protocol access micro-module corresponding to each subscriber.

[0013] Optionally, the method further includes: When a new session is subscribed, the device access module checks whether a complete I-frame exists in the circular buffer. If the block exists, the device access module will push the complete I-frame and subsequent frames to the corresponding protocol access micro-module first; if it does not exist, the forced I-frame request module will be triggered.

[0014] Optionally, the method further includes: When there is no complete I-frame in the circular buffer, the forced I-frame request module sends a forced I-frame command to the streaming media source device and stores the received forced I-frame into the circular buffer.

[0015] Optionally, the method further includes: The media gateway module receives a protocol switching request initiated by the client, carrying the original session identifier; The corresponding data source instance is located based on the original session identifier, and a new access address under the new protocol is generated and returned to the client; The new access address allows the client to continue receiving data from the original playback progress.

[0016] Optionally, the data source instance maintained by the device access module is uniquely identified by a combination of device serial number, channel number, and stream type; the method further includes: The device access module determines whether the requested data source already exists based on the unique identifier. If it does not exist, a new instance is created. If it already exists, the existing instance is reused, and the latest keyframe in the circular buffer is pushed to the newly subscribed protocol access micro-module.

[0017] Optionally, the plurality of protocol access micro-modules include at least two of FLV micro-modules, HLS micro-modules, RTSP micro-modules, and WebSocket micro-modules; the method further includes: The FLV micro-module packages the raw audio and video data into FLV format; The HLS micro-modules are packaged into TS or fMP4 slice format; The RTSP micro-module is packaged into an RTP packet; The WebSocket micro-module is distributed after being encapsulated using the WebSocket protocol.

[0018] Thirdly, embodiments of this application provide a computer device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the same-source multi-protocol multiplexing and distribution method described in any of the optional embodiments of the second aspect above are performed.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the same-origin multiplexed distribution method described in any of the optional embodiments of the second aspect above.

[0020] The technical solution provided in this application includes, but is not limited to, the following beneficial effects: The device access module interacts with the streaming media source device to pull the original audio and video streams and maintains a unique data source instance. This enables unified access control of audio and video sources, avoids multiple protocol services from repeatedly pulling data streams from the streaming media source device, effectively reduces the access load of the streaming media source device, reduces network bandwidth redundancy, and ensures the uniqueness and consistency of the global audio and video data source.

[0021] Each protocol access micro-module corresponds to a single media protocol and is deployed independently. Protocol modules can be expanded and iterated as needed. Modules are isolated from each other and do not interfere with each other. By subscribing to a unique data source instance to obtain raw audio and video data, and then encapsulating and distributing it according to its own protocol rules, it can achieve the output of multiple media protocols for audio and video streams from the same source, and can be compatible with the access needs of clients with different protocol types.

[0022] The media gateway module exposes an application programming interface (API) that can centrally receive client access requests and uniformly complete user authentication and verification, thereby strengthening system access security and access control. At the same time, it can accurately match and return the access address of the corresponding protocol access micro-module based on the media protocol type requested by the client, realizing intelligent routing access for the client, simplifying the client configuration process, and improving the convenience of user access.

[0023] The device access module has a built-in circular buffer for storing raw audio and video frames for a predetermined time period. This enables short-term caching and retention of audio and video frames, allowing newly connected protocol subscription nodes to quickly replenish historical frame data. Furthermore, the circular buffer can reuse storage space in a cyclical manner, avoiding the problem of continuous memory growth caused by linear caching, effectively saving system memory resources, and ensuring the stable operation of audio and video data caching.

[0024] The subscription management module maintains a subscriber list and binds the micro-module sessions of each protocol access, which can centrally manage the subscription relationships of all protocol micro-modules. This facilitates the full lifecycle management of subscription sessions, such as adding, canceling, and monitoring their status. It also clearly outlines the relationship logic between data sources and multi-protocol distribution nodes, improving the orderliness and maintainability of the system's subscription relationship management.

[0025] When new data arrives from the data source, the data distribution module automatically traverses the subscriber list and distributes the data to each corresponding micro-module. This enables one-to-many proactive distribution of single-source data to multi-protocol access micro-modules, eliminating the need to establish separate data transmission channels for each protocol node. This simplifies the internal data distribution logic and improves the real-time performance of audio and video data forwarding and overall distribution efficiency.

[0026] In summary, this application employs a modular, layered design. It utilizes a device access module to achieve unified source stream retrieval and instance management, relies on an independent protocol access micro-module to complete multi-protocol heterogeneous adaptation and distribution, and leverages a media gateway module to achieve unified authentication and intelligent address scheduling. Furthermore, through internal mechanisms such as ring caching, subscription list management, and targeted traversal distribution, it optimizes audio and video data caching, subscription management, and distribution capabilities while reducing source device and bandwidth resource consumption, supporting flexible multi-protocol expansion, and ensuring secure and convenient access. This overall improvement enhances the stability, real-time performance, and scalable maintenance capabilities of the same-source multi-protocol audio and video distribution system.

[0027] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This diagram illustrates the structure of a homogeneous multi-protocol multiplexing and distribution system provided in Embodiment 1 of this application. Figure 2 This diagram illustrates the overall layered architecture of a microservice system for multi-protocol conversion, multiplexing, and distribution of same-source streaming media data, as provided in Embodiment 1 of this application. Figure 3 This paper illustrates a user interaction method with a streaming media system and a related interaction flowchart within the system, as provided in Embodiment 1 of this application. Figure 4 This paper shows a schematic diagram of the structure of a device access module provided in Embodiment 1 of this application; Figure 5 This illustration shows a schematic diagram of an architecture for same-origin multi-protocol distribution provided in Embodiment 1 of this application; Figure 6 The flowchart of a method for multiplexing and distributing multiple protocols from the same source provided in Embodiment 2 of this application is shown. Figure 7 A schematic diagram of the structure of a computer device provided in Embodiment 3 of this application is shown. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0031] Example 1 To facilitate understanding of this application, the following is combined with... Figure 1 The schematic diagram of the structure of a homogeneous multi-protocol multiplexing and distribution system provided in Embodiment 1 of this application illustrates Embodiment 1 of this application in detail.

[0032] See Figure 1 As shown, Figure 1 This illustration shows a schematic diagram of a homogeneous multi-protocol multiplexing and distribution system provided in Embodiment 1 of this application, which includes: The device access module 101 is used to interact with the streaming media source device, pull the original audio and video streams, and maintain a unique data source instance.

[0033] Specifically, this device access module corresponds to the XMTS_SDK (X Media Transport Service Software Development Kit) device access service, which is the core of the entire system's media data source management. Streaming media source devices specifically refer to audio and video acquisition devices such as cameras. The interaction between the module and the device includes a complete login verification process, which is handled through the RPS (Remote Proxy Service) client service. Private or standard protocols are sent through the device port. After receiving the data, the device verifies the account and password and responds. Once verification is successful, the module establishes a stable connection with the device and retrieves the raw audio and video streams acquired by the device.

[0034] The module maintains data source instances for full lifecycle management using unique device identifiers. When the first subscriber of the source stream arrives, the streaming process is triggered and a data source instance is created. When the last subscriber leaves, the module delays the release of streaming connections, caches, and other resources corresponding to the instance to avoid device pressure and playback delays caused by frequent starting and stopping of the streaming process.

[0035] The module maintains all active data source instances using a hash table structure. The key of the hash table is a unique identifier for the device, which is composed of the device serial number, channel number, and stream type. Each data source object contains core information such as stream status (currently streaming / idle), circular buffer, subscriber list, and transcoder instance reference, ensuring that the same stream from the same device will only share one data source instance, even if it is subscribed to by different protocols, thus avoiding the waste of resources from repeated streaming from the root.

[0036] Multiple protocol access micro-modules 102 are deployed independently, each corresponding to a media protocol. They are used to subscribe to the data source instance from the device access module and encapsulate the raw audio and video data received from the device access module according to their respective media protocols before distributing it to the client.

[0037] Specifically, this series of protocol access micro-modules corresponds to the XMedia series of media services. They are independent execution units for media protocol encapsulation and distribution, including independent micro-services corresponding to various protocols such as XMedia_FLV, XMedia_HLS, XMedia_RTSP, XMedia_RTMP, XMedia_WS, and XMedia_WebRTC. Each service is deployed completely independently, achieving complete decoupling between protocols.

[0038] Each protocol access micro-module can independently scale up or down according to its corresponding protocol's business load without adjusting the deployment resources of other protocol modules. At the same time, upgrades or crashes of a single protocol service will not affect the normal operation of other protocol modules, achieving fault isolation between protocols and solving the defects of severe protocol coupling and poor overall service stability in the existing monolithic architecture.

[0039] Each protocol access micro-module initiates a data source subscription request to the device access module through the system's internal interface and establishes a session corresponding to its own protocol type, including HTTPSession, RTSPSession, WSSession, etc. After receiving the raw audio and video data distributed by the device access module, the module strictly completes the data encapsulation according to the specifications of the corresponding protocol and finally distributes it to various client players that support the corresponding protocol, such as ffplay, VLC, PotPlayer, and web browsers.

[0040] The media gateway module 103 is used to provide an application programming interface to respond to client requests for user authentication and return to the client the access address of the protocol access micro-module corresponding to the requested media protocol.

[0041] Specifically, this media gateway module corresponds to GWM (Gateway Media Service), which is the only window for the entire system to interact with the outside world. It also undertakes the core responsibilities of service discovery, configuration center, and system monitoring and management. Internally, it can monitor and manage the running status of all sub-services within the system.

[0042] The module provides a standard HTTP (Hypertext Transfer Protocol) application interface. When a client initiates a request to obtain a media playback address, it must include the access device information, device account password, and required media encapsulation format parameters in the request. After receiving the request, the module forwards the verification request to the device access module, which then completes the device login verification. After receiving the verification result, the module executes the subsequent process.

[0043] Once the module receives the login success verification result returned by the device access module, it will generate the playback URL (Uniform Resource Locator) of the corresponding protocol access micro-module as the access address based on the media protocol type requested by the client, and return it to the client. If the verification fails, it will return the corresponding error message to the client and refuse to provide the playback address. This achieves unified authentication, encryption, and traffic statistics management across the entire system, avoiding the duplication of related logic development for each protocol module.

[0044] See Figure 2 As shown, Figure 2 This document illustrates the overall layered architecture of a microservice system for multi-protocol conversion, multiplexing, and distribution of homogeneous streaming media data, as provided in Embodiment 1 of this application. It fully presents the core module composition and hierarchical relationship of the system. The overall title is "Streaming Media System," and the architecture is divided into three layers from top to bottom.

[0045] The top layer is the GWM module, which corresponds to the media gateway module in this application. The functional description on the right side of the module is that it provides interface interaction to the outside world and manages and monitors the entire microservice system internally. This module is the only window for the entire system to interact with the outside world. It provides an HTTP application interface to respond to client requests and undertakes the core responsibilities of service discovery, configuration center, and monitoring of the running status of sub-services in the entire system. At the same time, it completes core operations such as user authentication, device account verification, and playback address generation.

[0046] The middle layer is the media protocol service layer, horizontally arranged with six independent protocol access micro-modules: XMedia_FLV (X Media Flash Video), XMedia_HLS (X Media HTTP Live Streaming), XMedia_RTSP (X Media Real Time Streaming Protocol), XMedia_RTMP (X Media Real-Time Messaging Protocol), XMedia_WS (X Media WebSocket), and XMedia_WebRTC (XMedia Web Real-Time Communication). The function description on the right side of this layer is media service, which packages the raw audio and video streams into the corresponding format and sends them to the user. Each module corresponds to an independent media protocol, is deployed completely independently, and operates decoupled. It is responsible for subscribing to data sources from the underlying device access modules, encapsulating the received raw audio and video data according to the corresponding protocol specifications, and distributing it to the adapted client, thus achieving complete separation of protocol processing logic and source stream data management.

[0047] The underlying layer is the XMTS_SDK module, which corresponds to the device access module of this application. The functional description on the right side of the module is device access service, which interacts with the device to manage the data source. This module is the core of the media data source management of the entire system. It is responsible for interacting with streaming media source devices such as cameras, logging in, verifying and pulling raw audio and video streams. It maintains a globally unique data source instance for each stream and undertakes core capabilities such as audio and video data caching, subscriber management, multi-channel data distribution, and forced I-frame requests. It is the carrier of the core design of this application of one-time streaming and multi-protocol multiplexing.

[0048] In this embodiment of the application, the end-to-end process from the client initiating a playback request to completing the streaming media playback can be achieved through... Figure 3 The complete timing sequence is shown below. See [link / reference]. Figure 3 As shown, Figure 3 The diagram illustrates a user interaction method with a streaming media system and a related interaction flowchart within the system, as provided in Embodiment 1 of this application. The interaction participants involved in the diagram include User (user / client), GWM (media gateway module), XMedia (protocol access micro-module), XMTS_SDK (device access module), and Device (streaming media source device / camera). The entire process is divided into two core stages: obtaining the URL and playing the URL.

[0049] The URL acquisition stage is the authentication and playback address acquisition process before playback. First, the User sends a request to GWM to obtain the media URL for device preview or playback. After receiving the user's request, GWM forwards a verification request to XMTS_SDK to verify whether the device account password is correct. After receiving the verification request, XMTS_SDK initiates a device access operation to the target Device to log in to the device. After the Device completes the account password verification, it returns the device login result to XMTS_SDK. Based on the login result returned by the device, XMTS_SDK synchronously returns the verification result to GWM. After the verification is successful, GWM returns the device playback URL to the User.

[0050] The playback URL stage is the core process from client-initiated playback to streaming media data reception, decoding, and playback. The user initiates a playback request for the playback URL to XMedia through the player. After receiving the playback request, XMedia sends a subscription request to XMTS_SDK to obtain (subscribe to) the media data source. After receiving the subscription request, if the corresponding data source instance does not exist, XMTS_SDK sends a start streaming command to the Device. After receiving the start streaming command, the Device returns the media data to XMTS_SDK. After maintaining a unified data source instance, XMTS_SDK returns the media data to the subscribed XMedia in parallel. After receiving the raw audio and video data, XMedia performs protocol encapsulation processing to package it into the corresponding encapsulation format (e.g., XMedia_FLV is packaged into FLV format). Finally, XMedia sends the encapsulated media data to the user's playback client.

[0051] See Figure 4 As shown, Figure 4 This illustration shows a structural diagram of a device access module provided in Embodiment 1 of this application, wherein the device access module includes: The circular buffer 1011 is used to store raw audio and video frames for a predetermined duration.

[0052] Specifically, the circular buffer is the core unit inside the device access module that enables fast first-frame output. It is implemented using a circular queue structure with a predetermined time length of 500 milliseconds, which is specifically used to store the original audio and video frames within this time length. It must include the video key frame (I-frame) to provide data support for the fast decoding and output of new sessions.

[0053] The buffer size is calculated based on the frame rate, with a maximum capacity pre-allocated for the number of frames corresponding to 500 milliseconds. Different bitrates have different frame rates, and the corresponding number of frames and data size are also different, and are not dynamically adjusted during operation. When new audio and video frame data arrives, the buffer will automatically overwrite the oldest frame data, always storing the latest 500 milliseconds of audio and video data.

[0054] The subscription management module 1012 is used to maintain a subscriber list, where each subscriber in the subscriber list corresponds to a session of a protocol access micro-module.

[0055] Specifically, the subscription management module is the core scheduling unit for the device access module to realize multi-protocol multiplexing and distribution. The session information corresponding to each subscriber includes a unique session ID, protocol type, and required transcoding parameters (such as target resolution and encoding format), which can accurately distinguish the session requirements of different protocols and different clients.

[0056] The subscription management module maintains the validity of the subscriber list in real time. When new audio and video frame data arrives from the data source, it provides the data distribution module with a complete subscriber traversal list to ensure that the data can be distributed to all legitimate subscribers. At the same time, when a subscriber actively cancels their subscription, the client disconnects, or the session times out abnormally, the corresponding subscriber is promptly removed from the list to avoid resource waste caused by invalid data distribution.

[0057] The data distribution module 1013 is used to traverse the subscriber list maintained by the subscription management module and distribute the new data to the protocol access micro-module corresponding to each subscriber when new data arrives from the data source.

[0058] Specifically, the data distribution module is the core execution unit of the device access module to achieve "one-time streaming, multiple protocol reuse". Based on the subscriber list maintained by the subscription management module, it pushes the same original audio and video frame data in parallel to all protocol access micro-modules that have subscribed to the data source, without having to establish a separate streaming connection between each protocol access micro-module and the device.

[0059] This distribution mechanism only needs to pull the original audio and video stream from the device side once to support the reuse of any number of protocol sessions. It completely solves the problem of serious waste of CPU (Central Processing Unit) and memory resources caused by the independent streaming, transcoding and encapsulation of each protocol service in the existing siloed architecture. At the same time, it greatly reduces the uplink bandwidth pressure and hardware performance requirements of the device and improves the overall system stability.

[0060] The core mechanism of this application for same-origin multi-protocol distribution can be achieved through... Figure 5 The complete architecture diagram shown is available in the image. Figure 5 As shown,Figure 5 The diagram illustrates a single-source multi-protocol distribution architecture provided in Embodiment 1 of this application. This architecture is divided into four layers from source stream access to client playback: source stream layer, data source management layer, protocol encapsulation layer, and client layer.

[0061] The core of the source layer is the Device (camera), i.e., the streaming media source device, which is the source of the original audio and video data for the entire system and is responsible for capturing audio and video images and outputting the original bitstream. The core of the data source management layer is the XMTS_SDK module, which internally maintains a media data source (sn_channel_stream). This data source uses the device serial number_channel number_stream type as a globally unique identifier and is a unique data source instance for each stream in this application. The original audio and video streams output by the Device (camera) only need to be pulled once and stored in this unified media data source for reuse by all upper-layer protocol modules, thus fundamentally avoiding the waste of server resources and device bandwidth caused by repeated streaming and decoding.

[0062] The protocol encapsulation layer consists of multiple independently deployed XMedia series protocol access micro-modules. Each module corresponds to an independent media protocol and dedicated session management, including the XMedia_FLV module, which internally maintains an HTTPSession (Hyper Text Transfer Protocol Session) responsible for encapsulating raw audio and video data into FLV format; the XMedia_HLS module, which internally maintains an HTTPSession responsible for encapsulating raw audio and video data into HLS segment format; the XMedia_RTSP module, which internally maintains an RTSPSession (Real Time Streaming Protocol Session) responsible for encapsulating raw audio and video data into RTP (Real-time Transport Protocol) packet format; and the XMedia_WS module, which internally maintains a WSSession (WebSocket... Session (WebSocket protocol session) is responsible for encapsulating and distributing audio and video data via the WebSocket protocol; and the XMedia... extension module internally maintains a general Session, supports flexible access and extension of new media protocols, and all protocol modules subscribe to the same media data source in XMTS_SDK, receive raw audio and video data distributed in parallel by the receiving module, and independently complete the encapsulation and processing of the corresponding protocol without interference.

[0063] The client layer consists of different types of playback terminals (USER), including USER (ffplay), USER (vlc), USER (potplayer), and USER (web). Each terminal receives the encapsulated streaming media data distributed by the corresponding XMedia module according to the protocol type it supports, and completes video playback. This architecture clearly presents the core invention logic of this application, which uses XMTS_SDK as the core to build a unified media data source management center, realizes one-time retrieval of the same source stream, parallel multiplexing and distribution of multiple protocols, and supports flexible expansion of new protocols.

[0064] In an optional implementation, the device access module is configured to check whether a complete I-frame exists in the circular buffer when a new session is subscribed.

[0065] Specifically, a new session subscription refers to a new protocol access micro-module initiating a legitimate data source subscription request to the device access module. After receiving the request, the device access module will first read the latest audio and video frame data stored in the circular buffer, and verify whether there is a complete I-frame in the data that can be used as the starting point for video decoding, so as to determine whether it can directly achieve millisecond-level fast first frame output to the client.

[0066] The device access module is configured to, if present, prioritize pushing the complete I-frame and subsequent frames to the corresponding protocol access micro-module; if not present, trigger the forced I-frame request module.

[0067] Specifically, when a complete I-frame is detected in the circular buffer, the device access module will immediately retrieve the complete I-frame from the buffer, as well as all subsequent frames between the I-frame and the latest frame, and push them to the newly subscribed protocol access micro-modules. This allows the client player to quickly complete video decoding and display the image, achieving a fast image output effect with the first frame of a new session taking less than 500 milliseconds.

[0068] When no complete I-frame is detected in the circular buffer, the device access module will immediately trigger the forced I-frame request module. This module sends a forced I-frame command to the streaming media source device to quickly obtain key frames that can be used for decoding, thus avoiding long black screen waiting times for the client and improving the playback experience.

[0069] In an optional implementation, the forced I-frame request module is used to: When there is no complete I-frame in the circular buffer, a forced I-frame command is sent to the streaming media source device, and the received forced I-frame is stored in the circular buffer.

[0070] Specifically, the forced I-frame command is mainly sent using a proprietary protocol supported by the streaming media source device. For standard protocols such as national standards, it can be adapted and sent according to the device's protocol support capabilities. If the device does not support the forced I-frame command, it will not respond, which will only cause the I-frame output to slow down and the outflow speed to decrease, thereby affecting the fast image output effect, without causing other serious abnormalities in the system or device.

[0071] The forced I-frame request module records the sending time of the most recent forced I-frame request and sets a request interval threshold of 500 milliseconds. This threshold is a pre-configured fixed parameter to avoid frequent sending of commands to the device, which could cause device processing abnormalities. When the forced I-frame returned by the device arrives at the device access module, the module will immediately store the I-frame in the circular buffer to complete the cache update. At the same time, it will notify all subscribers that have subscribed to this data source to switch the decoding synchronization point to ensure the consistency of playback screens across all sessions.

[0072] In an optional implementation, the media gateway module is further configured to: Receive a protocol switching request initiated by the client, carrying the original session identifier.

[0073] Specifically, this capability enables seamless switching between protocols on the client side, solving the experience defect in the existing architecture where the client must re-pull the stream when switching between different protocols, causing playback interruption; the original session identifier contains core information such as the unique device identifier, original playback progress, and source stream parameters, which can uniquely lock the corresponding data source instance and playback status.

[0074] The corresponding data source instance is located based on the original session identifier, and a new access address under the new protocol is generated and returned to the client.

[0075] Specifically, after receiving a protocol switching request, the media gateway module accurately locates the corresponding data source instance through the original session identifier. Without requiring the client to re-initiate device login verification and source stream retrieval requests, it directly generates a new access address for the corresponding protocol access micro-module based on the new protocol type requested by the client and returns it to the client.

[0076] The new access address allows the client to continue receiving data from the original playback progress.

[0077] Specifically, the new protocol session corresponding to the new access address will inherit the playback progress of the original session through the session inheritance mechanism. The device access module will continue to send audio and video data to the new protocol access micro-module from the playback position where the original session was interrupted. This allows the client to continue receiving data and playing from the original playback progress without re-buffering, achieving a seamless playback effect without lag or interruption when switching across protocols.

[0078] In an optional implementation, the data source instance maintained by the device access module is uniquely identified by a combination of device serial number, channel number, and stream type.

[0079] Specifically, the standard combination format of this unique identifier is "device serial number_channel number_stream type". The device serial number is a globally unique number for each streaming media source device. The channel number is used to distinguish different video capture channels of the same device. The stream type specifically refers to the main stream (HD) and auxiliary stream (SD) of the camera device, which is used to distinguish video streams of different resolutions under the same channel of the same device.

[0080] The device access module maintains all active data source instances in the system using a hash table data structure with this unique identifier as the key. The lookup feature of the hash table enables millisecond-level retrieval of data source instances, ensuring the response speed of subscription requests in high-concurrency scenarios.

[0081] The device access module is used to determine whether the requested data source already exists based on the unique identifier. If it does not exist, a new instance is created. If it already exists, the existing instance is reused, and the latest keyframe in the circular buffer is pushed to the newly subscribed protocol access micro-module.

[0082] Specifically, when the device access module receives a data source subscription request from the protocol access micro-module, it first parses the device serial number, channel number, and stream type parameters carried in the request, generates a corresponding unique identifier, and searches the hash table to see if the data source instance corresponding to the identifier exists.

[0083] When there is no corresponding data source instance in the hash table, the request is determined to be the first subscription request for the source stream. The device access module will immediately trigger the interaction and streaming process with the corresponding streaming media source device. After completing the device login verification, the original audio and video streams are pulled, the corresponding data source instance is created, and added to the hash table for management.

[0084] When a corresponding data source instance already exists in the hash table, the existing instance can be reused directly without having to repeatedly establish a pull connection with the device or repeatedly pull source stream data. At the same time, the latest complete I-frame stored in the circular buffer, as well as all subsequent frames from that I-frame to the latest frame, are immediately pushed to the newly subscribed protocol access micro-module, enabling rapid output of the new session while ensuring the consistency of source stream data in multi-protocol sessions.

[0085] In one alternative implementation, the plurality of protocol access micromodules include at least two of the following: FLV micromodule, HLS micromodule, RTSP micromodule, and WebSocket micromodule.

[0086] Specifically, each protocol access micro-module is a completely independent microservice node, which can flexibly add or remove supported protocol types according to business needs. When adding a new protocol, only the corresponding independent microservice needs to be developed, and the existing underlying logic such as source stream access, data source subscription, and authentication management can be reused without repeated development, which greatly shortens the access cycle of new protocols and solves the defects of high cost and long cycle of new protocol access in the existing architecture.

[0087] The system can independently expand or shrink the resources of the corresponding protocol access micro-modules according to the actual business access volume of each protocol. For example, when the web access volume surges, only the WebSocket micro-module and FLV micro-module need to be expanded, and other protocol modules do not need to be adjusted, which greatly improves the resource utilization rate and solves the problem that the existing monolithic architecture cannot be expanded or shrunk on demand and has low resource utilization.

[0088] The FLV micro-module is used to package raw audio and video data into FLV format.

[0089] Specifically, the FLV micro-module corresponds to the XMedia_FLV service, which encapsulates the received raw audio and video frame data into FLV data in audio tag, video tag, and script tag formats according to the FLV protocol specification, and distributes it to client players that support FLV format playback via HTTP protocol. It is suitable for low-latency live streaming playback scenarios on PC (Personal Computer) and Web.

[0090] The HLS micro-module is used to package the data into TS or fMP4 slice format.

[0091] Specifically, the HLS micro-module corresponds to the XMedia_HLS service, which encapsulates continuous raw audio and video data into fixed-length slice files in TS (Transport Stream) or fMP4 (fragmented Moving Picture Experts Group 4) format according to the HLS protocol specification, and simultaneously generates corresponding m3u8 index files. The client retrieves the m3u8 index file and downloads the slice files in index order to complete the playback. It is suitable for playback scenarios with cross-platform compatibility and high network adaptability requirements.

[0092] The RTSP micro-module is used to package it into an RTP packet.

[0093] Specifically, the RTSP micro-module corresponds to the XMedia_RTSP service. It first completes RTSP signaling interaction with the client, including the complete process of OPTIONS, DESCRIBE, SETUP, PLAY, etc., to negotiate media transmission parameters. Then, it packages and encapsulates the original audio and video frame data according to the RTP specification and transmits it to the client via UDP (User Datagram Protocol) or TCP (Transmission Control Protocol) to achieve low-latency real-time transmission of streaming media data, which is suitable for professional real-time playback scenarios such as security monitoring and device preview.

[0094] The WebSocket micro-module is used for distribution after encapsulation via the WebSocket protocol.

[0095] Specifically, the WebSocket micro-module corresponds to the XMedia_WS service, which is based on the WebSocket full-duplex communication protocol. It establishes a persistent long connection with the web browser client and pushes the encapsulated audio and video data to the client in real time. This eliminates the need for the client to frequently initiate HTTP polling requests, significantly reducing playback latency and network overhead. It is suitable for real-time live streaming and low-latency preview playback scenarios on the web browser.

[0096] Example 2 See Figure 6 As shown, Figure 6 The flowchart shown is a method for homogeneous multi-protocol multiplexing and distribution provided in Embodiment 2 of this application. The method is applied to a homogeneous multi-protocol multiplexing and distribution system, which includes a device access module, multiple protocol access micro-modules, and a media gateway module. Each protocol access micro-module corresponds to a media protocol and is deployed independently. The method includes steps S601-S603: S601: The device access module interacts with the streaming media source device, pulls the original audio and video streams, and maintains a unique data source instance; S602: Each protocol access micro-module subscribes to the data source instance from the device access module, and encapsulates the raw audio and video data received from the device access module according to its corresponding media protocol and distributes it to the client; S603: The media gateway module provides an application programming interface to respond to client requests for user authentication and return to the client the access address of the protocol access micro-module corresponding to the requested media protocol.

[0097] In an optional implementation, the device access module includes a ring buffer, a subscription management module, and a data distribution module; the method further includes: The circular buffer stores raw audio and video frames for a predetermined time period; The subscription management module maintains a subscriber list, and each subscriber in the subscriber list corresponds to a session of a protocol access micro-module; When new data arrives from the data source, the data distribution module iterates through the subscriber list maintained by the subscription management module and distributes the new data to the protocol access micro-module corresponding to each subscriber. The device access module includes a ring buffer, a subscription management module, and a data distribution module; the method further includes: The circular buffer stores raw audio and video frames for a predetermined time period; The subscription management module maintains a subscriber list, and each subscriber in the subscriber list corresponds to a session of a protocol access micro-module; When new data arrives from the data source, the data distribution module iterates through the subscriber list maintained by the subscription management module and distributes the new data to the protocol access micro-module corresponding to each subscriber.

[0098] In an optional implementation, the method further includes: When a new session is subscribed, the device access module checks whether a complete I-frame exists in the circular buffer. If the block exists, the device access module will push the complete I-frame and subsequent frames to the corresponding protocol access micro-module first; if it does not exist, the forced I-frame request module will be triggered.

[0099] In an optional implementation, the method further includes: When there is no complete I-frame in the circular buffer, the forced I-frame request module sends a forced I-frame command to the streaming media source device and stores the received forced I-frame into the circular buffer.

[0100] In an optional implementation, the method further includes: The media gateway module receives a protocol switching request initiated by the client, carrying the original session identifier; The corresponding data source instance is located based on the original session identifier, and a new access address under the new protocol is generated and returned to the client; The new access address allows the client to continue receiving data from the original playback progress.

[0101] In an optional implementation, the data source instance maintained by the device access module is uniquely identified by a combination of device serial number, channel number, and stream type; the method further includes: The device access module determines whether the requested data source already exists based on the unique identifier. If it does not exist, a new instance is created. If it already exists, the existing instance is reused, and the latest keyframe in the circular buffer is pushed to the newly subscribed protocol access micro-module.

[0102] In an optional implementation, the plurality of protocol access micro-modules include at least two of FLV micro-modules, HLS micro-modules, RTSP micro-modules, and WebSocket micro-modules; the method further includes: The FLV micro-module packages the raw audio and video data into FLV format; The HLS micro-modules are packaged into TS or fMP4 slice format; The RTSP micro-module is packaged into an RTP packet; The WebSocket micro-module is distributed after being encapsulated using the WebSocket protocol.

[0103] Example 3 Based on the same application concept, see [link / reference] Figure 7 As shown, Figure 7 This illustration shows a structural schematic diagram of a computer device provided in Embodiment 3 of this application, wherein, as shown... Figure 7 As shown, the computer device 700 provided in Embodiment 3 of this application includes: The computer device 700 includes a processor 701, a memory 702, and a bus 703. The memory 702 stores machine-readable instructions that can be executed by the processor 701. When the computer device 700 is running, the processor 701 communicates with the memory 702 through the bus 703. When the machine-readable instructions are executed by the processor 701, the steps of the same-source multi-protocol multiplexing and distribution method shown in Embodiment 2 are executed.

[0104] Example 4 Based on the same concept, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the same-origin multiplexed distribution method described in any of the above embodiments.

[0105] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0106] The computer program product for same-source multi-protocol multiplexing and distribution provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0107] The same-origin multi-protocol multiplexing and distribution system provided in this application embodiment can be specific hardware on the device or software or firmware installed on the device. The method provided in this application embodiment has the same implementation principle and technical effects as the aforementioned system embodiment. For the sake of brevity, any parts not mentioned in the method embodiment can be referred to the corresponding content in the aforementioned system embodiment.

[0108] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0109] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0111] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0113] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A homogeneous multi-protocol multiplexing and distribution system, characterized in that, include: The device access module is used to interact with streaming media source devices, pull raw audio and video streams, and maintain a unique data source instance; Multiple protocol access micro-modules are deployed independently, each corresponding to a media protocol. They are used to subscribe to the data source instance from the device access module and encapsulate the raw audio and video data received from the device access module according to their respective media protocols before distributing it to the client. The media gateway module is used to provide an application programming interface to respond to client requests for user authentication and return to the client the access address of the protocol access micro-module corresponding to the requested media protocol. The device access module includes: A circular buffer is used to store raw audio and video frames for a predetermined duration; The subscription management module is used to maintain a list of subscribers, where each subscriber in the list corresponds to a session of a protocol access micro-module; The data distribution module is used to traverse the subscriber list maintained by the subscription management module and distribute the new data to the protocol access micro-module corresponding to each subscriber when new data arrives from the data source.

2. The system according to claim 1, characterized in that, The device access module is used to check whether a complete I-frame exists in the circular buffer when a new session is subscribed; The device access module is configured to, if present, prioritize pushing the complete I-frame and subsequent frames to the corresponding protocol access micro-module; if not present, trigger the forced I-frame request module.

3. The system according to claim 2, characterized in that, The forced I-frame request module is used for: When there is no complete I-frame in the circular buffer, a forced I-frame command is sent to the streaming media source device, and the received forced I-frame is stored in the circular buffer.

4. The system according to claim 1, characterized in that, The media gateway module is also used for: Receive a protocol switching request initiated by the client, carrying the original session identifier; The corresponding data source instance is located based on the original session identifier, and a new access address under the new protocol is generated and returned to the client; The new access address allows the client to continue receiving data from the original playback progress.

5. The system according to claim 1, characterized in that, The data source instance maintained by the device access module is uniquely identified by a combination of device serial number, channel number, and stream type. The device access module is used to determine whether the requested data source already exists based on the unique identifier. If it does not exist, a new instance is created. If it already exists, the existing instance is reused, and the latest keyframe in the circular buffer is pushed to the newly subscribed protocol access micro-module.

6. The system according to claim 1, characterized in that, The multiple protocol access micro-modules include at least two of the following: FLV micro-module, HLS micro-module, RTSP micro-module, and WebSocket micro-module; The FLV micro-module is used to package the raw audio and video data into FLV format; The HLS micro-module is used to package the data into TS or fMP4 slice format; The RTSP micro-module is used to package it into an RTP packet; The WebSocket micro-module is used for distribution after encapsulation via the WebSocket protocol.

7. A method for multiplexing and distributing multiple protocols from the same source, characterized in that, An application is made in a homogeneous multi-protocol multiplexing and distribution system, the system comprising a device access module, multiple protocol access micro-modules, and a media gateway module, each protocol access micro-module corresponding to a media protocol and deployed independently; the method includes: The device access module interacts with the streaming media source device, pulls the original audio and video streams, and maintains a unique data source instance; Each protocol access micro-module subscribes to the data source instance from the device access module, and encapsulates the raw audio and video data received from the device access module according to its corresponding media protocol before distributing it to the client; The media gateway module provides an application programming interface (API) to respond to client requests for user authentication and return to the client the access address of the protocol access micro-module corresponding to the requested media protocol.

8. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the same-source multi-protocol multiplexing and distribution method as described in claim 7 are performed.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the same-source multi-protocol multiplexing and distribution method as described in claim 7.

Images