A streaming media server

Abstract

The application provides a streaming media server, comprising the following independent components: an API for quickly creating a browser-based Webrtc audio and video interaction client and performing related operations on a media server; a Worker bound to and running on a certain core of a CPU, responsible for media data processing and forwarding a C++ subprocess; and the API and the Worker interact through inter-process communication. The streaming media server of the application conforms to the JTT-1078 protocol vehicle-mounted device and solves the problems of poor real-time performance, high delay and poor performance of existing open-source general media servers by taking advantage of the Webrtc real-time communication technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of streaming media servers, in particular to a streaming media server based on Webrtc real-time communication technology and JTT-1078 road transport vehicle satellite positioning system video communication protocol. BACKGROUND

[0002] With the rapid development of China's road transport industry, the number of "two passenger and one dangerous" road special vehicles has increased significantly. Therefore, while improving the vehicle transportation efficiency and strengthening the vehicle carrying capacity, the demand for monitoring the driving state of the vehicle through audio and video, standardizing the driving habits of the driver, etc. is increasing. Due to the network environment firewall and other reasons, audio and video data often encounter difficulties in network transmission, at this time the intervention of the streaming media server is needed to send the audio and video data of the vehicle-mounted access device to the front-end monitoring personnel.

[0003] However, there is no streaming media server specially developed for JTT-1078 protocol vehicle-mounted equipment at present. The usual practice is to modify other general streaming media servers to support this application scenario. These general streaming media servers have poor applicability and are usually developed for live and on-demand application scenarios. Some media server technologies are relatively backward, usually using RTSP, RTMP and other protocols for development, but the real-time performance is poor, the concurrency is not high, the performance is poor, the maintenance and modification are difficult, and the developers have high requirements in audio and video screen technology and network technology. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a streaming media server that meets the JTT-1078 protocol vehicle-mounted equipment and uses the advantages of Webrtc real-time communication technology to solve the problems of poor real-time performance, high delay and poor performance of existing open source general media servers.

[0005] The present application provides a streaming media server, comprising the following independent components:

[0006] API, for quickly creating a browser-based Webrtc audio and video interaction client and performing related operations on the media server;

[0007] Worker, bound and running on a certain core of CPU, responsible for media data processing and forwarding C++ subprocess;

[0008] The API and the Worker interact through inter-process communication.

[0009] Further, the Worker includes the following four channels:

[0010] Jtt1078Transport: for accepting TCP connection of vehicle-mounted device, for accepting sending JTT-1078 media data;

[0011] WebrtcTransport: for establishing network connection with Webrtc client, for carrying out DTLS data verification encryption, for receiving and sending SRTP media data;

[0012] PipeTransport: for carrying out data transmission between different Workers through pipe;

[0013] PlainTransport: for general TCP / UDP channel, for being used for data transmission between different Hosts, and for being used for communication with external device.

[0014] Further, each of the channels can create a plurality of data producers Producer and a plurality of data consumers Consumer, and each media stream corresponds to a data producer Producer or a data consumer Consumer; a data producer Producer can be connected with a plurality of data consumers Consumer of the same channel or different channels; data in the data producer Producer of any channel is forwarded by a Router module to all the data consumers Consumer connected with the data producer Producer, and then the data is sent out by the channel where the data consumer Consumer is located;

[0015] Wherein:

[0016] the data consumer Consumer is used for sending specified data through the channel;

[0017] the data producer Producer is used for obtaining specified data from the channel;

[0018] the Router module is used for forwarding the data obtained by the data producer Producer to a specified data consumer Consumer, and one data producer Producer can correspond to a plurality of data consumers Consumer;

[0019] Further, data forwarding includes the following cases:

[0020] (1) data forwarding between the same Workers in the Host;

[0021] The platform where the vehicle-mounted device is located accesses a certain Worker by calling an API and applies for a media channel WebrtcTransport and Jtt1078Transport from the Worker, and then the WebrtcTransport creates a data producer Producer for receiving data from the Webrtc client and a data consumer Consumer for sending data to the Webrtc client; and the Jtt1078Transport creates a data producer Producer for receiving data from the vehicle-mounted device and a data consumer Consumer for sending data to the vehicle-mounted device;

[0022] After the channel is established, the Consumers of the Workers are connected with the Producers of the opposite Workers respectively, and the data forwarding channel between the WebrtcTransport and the Jtt1078Transport channels is established;

[0023] The Webrtc client and the vehicle-mounted device forward audio and video data;

[0024] (2) Data forwarding between different Workers in the Host:

[0025] Multiple Workers in the same Host use PipTransport to perform cross-Worker data interaction, Worker 1 creates PipTransport 1 and creates a data producer Producer 1 and a consumer Consumer 1, Worker 2 creates PipTransport 2 and creates a data producer Producer 2 and a consumer Consumer 2, and after the forwarding association of Producer 1 to Consumer 2 and Producer 2 to Consumer 1 is established, data forwarding between Worker 1 and Worker 2 can be realized;

[0026] (3) Cross-Host data forwarding:

[0027] Different Hosts' Workers can utilize PlainTransport to interact data cross Hosts, Worker1 on Host1 creates PlainTransport1 and creates data producer Producer1 and consumer Consumer1, Worker2 on Host2 creates PlainTransport2 and creates data producer Producer2 and consumer Consumer2, and establishes the forwarding association of Producer1 to Consumer2 and Producer2 to Consumer1, then the media server will forward data between Host1 and Host2 through network.

[0028] Further, the Worker further comprises the following modules:

[0029] JTT-1078: used to obtain audio and video raw data from JTT-1078 format media;

[0030] ICE: auxiliary network penetration, collects interface IP information;

[0031] DTLS: data packet transport layer security protocol, negotiates secret key for encrypting UDP data;

[0032] RTP: used to package media data into RTP format;

[0033] RTCP: used for real-time transport control protocol, provides feedback of media service quality;

[0034] SRTP: used to encrypt RTP data into SRTP format;

[0035] Logger: a log module, used for log management;

[0036] Libuv: a network IO module;

[0037] Utils: a tool set;

[0038] EnhancedEventEmitter: used for dispatching and distributing messages and events between modules;

[0039] TransportCC: a congestion control module.

[0040] Further, the Worker is a single-process single-thread lock-free design.

[0041] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: the streaming media server is developed based on the Webrtc real-time communication technology, and the low-delay and high-reliability characteristics of Webrtc are fully utilized; the server is divided into two independent components, one is a low-level client API written by using a JavaScript script language based on nodejs, which greatly reduces the development difficulty of programmers, and the programmers can easily establish a Webrtc audio and video interaction client based on a browser by using the API; the other is a C / C++ sub-process for processing a media layer (JTT-1078, ICE, DTLS, RTP, etc.) called a Worker, which is used for processing, packaging, forwarding, etc. of media data. The two independent components interact through inter-process communication. When the media server starts, the same number of Worker processes as the number of CPU cores of the Host (physical host or cloud server) are started and bound to the CPU, so as to avoid the performance degradation caused by the frequent CPU scheduling of the processes jumping between different CPUs. The multiple Workers can be based on pipes and network communication to realize cluster deployment. The streaming media server uses libuv event-driven asynchronous network IO, which greatly improves the network IO performance. Each Worker is designed as a single-process single-thread lock-free, which improves the robustness, avoids the data structure damage caused by the lock mechanism, greatly improves the performance of the media service, and multiple Workers can be started according to the number of CPU cores when used, so as to realize high concurrency of the server. The developers only need to care about the use of the API, and do not need to care about the specific implementation of the server.

[0042] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0043] The present application will be further described below with reference to the accompanying drawings and embodiments.

[0044] Figure 1 The architecture schematic diagram of the streaming media server of the first embodiment of the present application is shown in the figure.

[0045] Figure 2 The architecture schematic diagram of the streaming media server of the second embodiment of the present application is shown in the figure.

[0046] Figure 3 The principle schematic diagram of data forwarding of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0047] The embodiment of the application provides a streaming media server, conforms to JTT-1078 protocol vehicle-mounted equipment, and solves the problems of poor real-time performance, high delay, poor performance and the like of an existing open source general media server by using the advantages of Webrtc real-time communication technology, so that high concurrency, high availability of the server, reduction of server cost, and reduction of development and maintenance cost of the media server are realized.

[0048] The technical scheme in the embodiment of the application has the following general idea: the streaming media server of the application is developed based on Webrtc real-time communication technology, and the low-delay and high-reliability characteristics of Webrtc are fully utilized; the server is divided into two independent components, one is a low-level client API written by using a JavaScript script language based on nodejs, which greatly reduces the development difficulty of programmers, and the programmers can easily establish a Webrtc audio and video interaction client based on a browser by using the API; the other is a C / C++ sub-process called Worker for processing a media layer (JTT-1078, ICE, DTLS, RTP, etc.), which is used for processing, packaging, forwarding and the like of media data. The two independent components interact through inter-process communication. When the media server is started, the same number of Worker processes as the number of CPU cores of a Host (a physical host or a cloud server) is started and is bound to the CPU, so that the performance decline caused by frequent CPU scheduling due to the jumping of processes between different CPUs is avoided. The plurality of Workers can be based on pipes and network communication, so that cluster deployment is realized. The streaming media server uses libuv event-driven asynchronous network IO, so that the network IO performance is greatly improved. Each Worker is designed as a single-process single-thread lock-free, so that the robustness is improved, the data structure damage caused by the lock mechanism is avoided, the performance of the media service is greatly improved, a plurality of Workers can be started according to the number of CPU cores when used, high concurrency of the server is realized, and the developers only need to care about the use of the API and do not need to care about the specific implementation of the server.

[0049] In the formula, R represents a C1-C20 alkyl group, X represents a C1-C20 alkyl group or a C1-C20 alkoxy group, and n represents an integer of 1 to 10.

[0050] WebRTC (Web Real-Time Communications) is a real-time communication technology without any plug-in or third-party software, which allows a network application or a site to establish a point-to-point (Peer-to-Peer) connection between browsers without the aid of an intermediate medium, so as to realize the low-delay and high-reliability transmission of video stream and / or audio stream or other arbitrary data.

[0051] Libuv is a high-performance, event-driven asynchronous network I / O library that significantly improves network I / O efficiency. Written in C, it boasts high portability. Libuv encapsulates the underlying asynchronous I / O model implementations for different platforms, thus enabling cross-platform compatibility with Windows and Linux.

[0052] Example 1

[0053] like Figure 1 As shown, this embodiment provides a streaming media server that conforms to the JTT-1078 protocol for in-vehicle devices, and utilizes the advantages of Webrtc real-time communication technology to solve the problems of poor real-time performance, high latency, and poor performance of existing open-source general-purpose media servers.

[0054] This invention provides a streaming media server, comprising the following independent components:

[0055] API for quickly creating browser-based WebRTC audio and video interactive clients and performing related operations on media servers;

[0056] Worker is a C++ subprocess that is bound to and runs on a specific CPU core and is responsible for media data processing and forwarding.

[0057] The API and the Worker interact through inter-process communication.

[0058] In this embodiment, the media server consists of two independent components: an API and a Worker. Most mainstream browsers support WebRTC real-time communication technology, allowing users to create WebRTC clients in just a few simple steps. Users can quickly create browser-based WebRTC clients using the provided API and operate the media server via the API, including requesting media channels, closing channels, checking channel status, and checking server status. A Worker represents a C++ subprocess running on a single-core CPU, responsible for media data processing and forwarding.

[0059] This embodiment uses mature WebRTC real-time communication technology, which reduces development difficulty and cycle, achieves full platform compatibility, and provides low-latency audio and video data transmission.

[0060] Example 2

[0061] This embodiment provides a streaming media server, which is a further improvement on the first embodiment.

[0062] like Figure 2 As shown, the Worker includes the following four channels:

[0063] Jtt1078Transport: for accepting TCP connection of vehicle-mounted device, for accepting sending JTT-1078 media data;

[0064] WebrtcTransport: for establishing network connection with Webrtc client, for carrying out DTLS data verification encryption, for receiving and sending SRTP media data;

[0065] PipeTransport: for carrying out data transmission between different Workers through pipe;

[0066] PlainTransport: for general TCP / UDP channel, for data transmission between different Hosts, and for communication with external device.

[0067] Further, each of the channels can create a plurality of data producers Producer and a plurality of data consumers Consumer, each media stream corresponds to a data producer Producer or a data consumer Consumer; a data producer Producer can be connected with a plurality of data consumers Consumer of the same channel or different channels; the data in the data producer Producer of any channel is forwarded to all the data consumers Consumer connected with the data producer Producer by the Router module, and then the data is sent out by the channel where the data consumer Consumer is located;

[0068] Wherein:

[0069] The data consumer Consumer is used for sending specified data through the channel;

[0070] The data producer Producer is used for obtaining specified data from the channel;

[0071] The Router module is used for forwarding the data obtained by the data producer Producer to the specified data consumer Consumer, and one data producer Producer can correspond to a plurality of data consumers Consumer;

[0072] The Worker further includes the following modules:

[0073] JTT-1078: for obtaining audio and video raw data from JTT-1078 format media;

[0074] ICE: auxiliary network penetration, collects interface IP information;

[0075] DTLS: Data Transport Layer Security protocol, negotiate secret key for encrypting UDP data;

[0076] RTP: Real Time Protocol, used for packing media data into RTP format;

[0077] RTCP: Real Time Transport Control Protocol, provide feedback for media quality of service;

[0078] SRTP: Secure Real Time Protocol, used for encrypting RTP data into SRTP format;

[0079] Logger: Log module, used for log management;

[0080] Libuv: Network IO module;

[0081] Utils: Tool set;

[0082] EnhancedEventEmitter: Message and event dispatching between modules;

[0083] TransportCC: Congestion control module.

[0084] Further, as shown in Figure 3 , data forwarding includes the following cases:

[0085] (1) Host intra-Worker data forwarding:

[0086] The platform where the vehicle-mounted device is located accesses a certain Worker by calling an API and applies for a media channel WebrtcTransport and Jtt1078Transport from the Worker. The WebrtcTransport creates a data producer Producer for receiving data from the Webrtc client and a data consumer Consumer for sending data to the Webrtc client. The Jtt1078Transport creates a data producer Producer for receiving data from the vehicle-mounted device and a data consumer Consumer for sending data to the vehicle-mounted device.

[0087] After the channel is established, the Consumers of the Workers are connected to the Producers of the opposite Workers, and the data forwarding channel between the WebrtcTransport and the Jtt1078Transport channels is established.

[0088] The Webrtc client and the vehicle-mounted device forward audio and video data, and the platform staff can communicate with the vehicle-mounted device in the vehicle via audio and video.

[0089] (2) Data forwarding between different Workers in the same Host:

[0090] Data interaction between multiple Workers in the same Host is implemented by using PipTransport, for example, Worker 1 in the same Host forwards data to Worker 2, Worker 1 creates PipTransport 1 and creates data producer Producer 1 and consumer Consumer 1, Worker 2 creates PipTransport 2 and creates data producer Producer 2 and consumer Consumer 2, and establishes forwarding association of Producer 1 to Consumer 2 and Producer 2 to Consumer 1, so as to realize data forwarding between Worker 1 and Worker 2;

[0091] (3) Data forwarding across Hosts:

[0092] Data interaction between Workers in different Hosts can be implemented by using PlainTransport, for example, Worker 1 in Host 1 forwards data to Worker 2 in Host 2, Worker 1 in Host 1 creates PlainTransport 1 and creates data producer Producer 1 and consumer Consumer 1, Worker 2 in Host 2 creates PlainTransport 2 and creates data producer Producer 2 and consumer Consumer 2, and establishes forwarding association of Producer 1 to Consumer 2 and Producer 2 to Consumer 1, so that the media server forwards data between Host 1 and Host 2 through the network. Through the data forwarding capability across Hosts, the media server can implement cluster deployment function.

[0093] Further, the Worker is a single-process single-thread lock-free design. The single-process mode fully utilizes a certain core of the CPU on the server, and then the number of processes is controlled according to the number of CPU cores through the business layer to improve the concurrent capability of the server. The lock-free design can improve the robustness, avoid the data structure damage caused by the lock mechanism, and reduce the idle time of the CPU to improve the overall efficiency.

[0094] The technical scheme provided in the embodiments of the present application has at least the following technical effects or advantages: using the mature Webrtc real-time communication technology, reducing the development difficulty and development cycle, realizing full-platform compatibility, and providing low-delay audio and video data transmission. Using the mature congestion control algorithm and packet loss compensation algorithm ensures the reliability of data transmission. The single-process single-thread lock-free architecture design improves the robustness and stability of the program. The high concurrency capability of the server can be implemented by starting multiple processes. Using event-driven asynchronous network IO greatly improves the network IO performance. The data can be encrypted for transmission to ensure data security. A set of low-level APIs are provided for developers to call, reducing the development difficulty.

[0095] Although the specific embodiments of the present application are described above, those skilled in the art should understand that the specific embodiments described are only illustrative, and are not intended to limit the scope of the present application, and equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present application should be covered within the scope of the claims of the present application.

Claims

1. A streaming media server, characterized by: Comprise the following independent components: API, for quickly creating a browser-based Webrtc audio and video interaction client, and performing related operations on the media server; Worker, bound and running on a certain core of the CPU, responsible for media data processing and forwarding C++ subprocesses; The API and the Worker interact through inter-process communication; The Worker includes the following four channels: Jtt1078Transport: used to accept TCP connections of the vehicle-mounted device and accept and send JTT-1078 media data; WebrtcTransport: used to establish a network connection with the Webrtc client, perform DTLS data verification and encryption, and receive and send SRTP media data; PipeTransport: used for data transmission between different Workers through a pipe; PlainTransport: a general TCP / UDP channel used for data transmission between different Hosts and communication with external devices; Data forwarding includes the following cases: (1) Data forwarding between the same Worker in the Host: The platform where the vehicle-mounted device is located accesses a certain Worker by calling the API and applies for media channels WebrctTransport and Jtt1078Transport from the Worker, and then the WebrctTransport creates a data producer Producer for receiving data from the Webrtc client and a data consumer Consumer for sending data to the Webrtc client; and the Jtt1078Transport creates a data producer Producer for receiving data from the vehicle-mounted device and a data consumer Consumer for sending data to the vehicle-mounted device; After the channel is established, the Consumers of the Workers are connected to the Producers of the opposite Workers, and the data forwarding channel between the WebrctTransport and the Jtt1078Transport channels is established; The Webrtc client and the vehicle-mounted device perform audio and video data forwarding; (2) Data forwarding between different Workers in the Host: Multiple Workers in the same Host use PipeTransport for cross-Worker data interaction, Worker1 creates PipeTransport1 and creates data producer Producer1 and consumer Consumer1, Worker2 creates PipeTransport2 and creates data producer Producer2 and consumer Consumer2, and establishes forwarding association from Producer1 to Consumer2 and from Producer2 to Consumer1, so as to realize data forwarding between Woker1 and Woker2; (3) Cross-Host data forwarding: Different Hosts' Workers can utilize PlainTransport to interact data cross Hosts, Worker1 on Host1 creates PlainTransport1 and creates data producer Producer1 and consumer Consumer1, Worker2 on Host2 creates PlainTransport2 and creates data producer Producer2 and consumer Consumer2, and establishes the forwarding association of Producer1 to Consumer2 and Producer2 to Consumer1, then the media server will forward data between Host1 and Host2 through network.

2. The streaming media server of claim 1, wherein: Each of the channels can create multiple data producers Producer and multiple data consumers Consumer, each media stream corresponds to a data producer Producer or a data consumer Consumer; a data producer Producer can be connected with multiple data consumers Consumer of the same channel or different channels; the data in the data producer Producer of any channel is forwarded to all data consumers Consumer connected with the data producer Producer by the Router module, and then the data is sent out by the channel where the data consumer Consumer is located; Wherein: the data consumer Consumer is configured to send the specified data through the channel; the data producer Producer is configured to obtain the specified data from the channel; the Router module is configured to forward the data obtained by the data producer Producer to the specified data consumer Consumer, and one data producer Producer can correspond to multiple data consumers Consumer.

3. The streaming media server of claim 2, wherein: When the media server starts, it will start the same number of Worker processes as the number of CPU cores and bind them to the CPU.

4. The streaming media server of claim 1, wherein: The Worker further comprises the following modules: JTT-1078: configured to obtain audio and video raw data from media in JTT-1078 format; ICE: auxiliary network penetration, collects interface IP information; DTLS: data packet transmission layer security protocol, negotiates secret keys for encrypting UDP data; RTP: configured to package media data into RTP format; RTCP: configured to provide real-time transmission control protocol, provides feedback of media service quality; SRTP: configured to encrypt RTP data into SRTP format; Logger: a log module, configured to manage logs; Libuv: a network IO module, so that libuv event-driven asynchronous network IO can be used; Utils: a tool set; EnhancedEventEmitter: configured to dispatch and distribute messages and events between modules; TransportCC: a congestion control module.

5. The streaming media server of claim 1, wherein: The Worker is a single-process single-thread lock-free design.

Images