A real-time data communication system

Abstract

The present invention relates to a real-time data communication system, which is characterized in that: it includes a communication gateway and a media mixing and relay module; the communication gateway server can support the operator's telephone network and the Internet; the first terminal device establishes a first communication link with the communication gateway through the operator's telephone network, so as to establish a traditional CS audio telephone call between the first terminal device and the communication gateway; the first terminal device establishes a second communication link with the communication gateway through the Internet, so as to establish a video call between the first terminal device and the communication gateway; the second terminal device is communicatively connected to the communication gateway through the media mixing and relay module, thereby realizing real-time audio and video calls between the first terminal device and the second terminal device. Compared with the prior art, the advantages of the present invention are: realizing audio and video communication in heterogeneous networks, enabling the first terminal device and the second terminal device to simultaneously conduct video calls during the process of not interrupting the traditional CS audio telephone call.

Description

Technical Field

[0001] The present invention relates to the field of communications, and particularly to a real-time data communication system. Background Art

[0002] A real-time data communication system, also abbreviated as a real-time audio and video data communication system and also simply called an RTC system. In recent years, with the continuous development of network and audio-visual technologies, RTC systems have been widely used in scenarios such as remote finance, distance education, telemedicine, and Internet social networking.

[0003] Existing real-time data communication systems include the following types:

[0004] First, a PSTN telephone system based on the CS domain or PS domain;

[0005] Second, a WebRTC audio and video call system;

[0006] Third, WeChat video call communication based on Tencent's proprietary RTMP protocol or TRTC protocol;

[0007] Fourth, a SIP audio and video call system;

[0008] Fifth, various converged communication systems, such as Cisco Webex, Zoom, Tencent Video Conference, Huawei Video Conference, and so on.

[0009] A real-time data communication system with a single network architecture is easily restricted by device or network conditions, and its applicability is not wide enough. For example: a simple PSTN telephone can only communicate audio; the operator's PS domain VoLTE video telephone has requirements for mobile phones (such as iPhone does not support); a simple WebRTC call cannot achieve highly reliable notifications, and the call success rate is extremely low; WeChat video calls require the other party to be a friend, and the call success rate is not high either, which cannot meet business needs; various converged communication systems require all parties to install specific software, and the applicable population is not wide.

[0010] Therefore, it is very necessary to establish a real-time audio and video communication system and its communication method under heterogeneous networks, reduce call latency, improve the call's resistance to weak networks, and ensure the time synchronization of audio and video media streams. Summary of the Invention

[0011] The technical problem to be solved by the present invention is to provide a real-time data communication system that can simultaneously conduct a video call during the process of a traditional CS audio telephone call without interruption in view of the above-mentioned prior art.

[0012] The technical solution adopted by the present invention to solve the above technical problems is as follows: A real-time data communication system, characterized in that it includes a communication gateway capable of supporting different communication networks, and a media mixing and transit module communicatively connected to the communication gateway; wherein the different communication networks that the communication gateway server can support at least include the operator telephone network and the Internet;

[0013] In the above real-time data communication system, the first terminal device and the second terminal device perform a video call simultaneously during the traditional CS audio telephone call without interruption in the following manner:

[0014] The first terminal device establishes a first communication link with the communication gateway through the operator telephone network, so as to establish a traditional CS audio telephone call between the first terminal device and the communication gateway, and the first terminal device sends voice data to the communication gateway through the first communication link; then the first terminal device establishes a second communication link with the communication gateway through the Internet, so as to establish a video call between the first terminal device and the communication gateway, and the first terminal device sends video data to the communication gateway through the second communication link;

[0015] The second terminal device is communicatively connected to the communication gateway through the media mixing and transit module; the communication gateway mixes the voice data and video data of the first terminal device through the media mixing and transit module and sends them to the second terminal device; the second terminal device sends its own generated voice data and video data to the communication gateway through the media mixing and transit module, and the communication gateway then sends the voice data of the second terminal device to the first terminal device through the first communication link, and sends the video data of the second terminal device to the first terminal device through the second communication link, thereby realizing the real-time audio and video call between the first terminal and the second terminal.

[0016] As an improvement, the system provided by the present invention further includes a third terminal device, and the third terminal device is also communicatively connected to the communication gateway through the media mixing and transit module; after the real-time audio and video call starts between the first terminal device and the second terminal device, the media mixing and transit module mixes the voice data and video data exchanged between the first terminal device and the second terminal device and sends them to the third terminal device, and the third terminal device records the real-time audio and video communication content between the first terminal device and the second terminal device to generate an audio and video file.

[0017] After a traditional CS audio phone call and a video call are established between a first terminal device and a communication gateway, one problem to be solved is the audio-video synchronization problem, and at the same time, the communication quality in a weak Internet network needs to be improved through QoS technology. Further improvement: after a traditional CS audio phone call and a video call are established between the first terminal device and the communication gateway, the communication gateway closes positive feedback and turns on forward error correction on the second communication link, and uses low-latency QoS technology; on the first communication link between the communication gateway and the operator telephone network, NACK technology is used; for RTP communication between the communication gateway and the media mixing and transit module, NACK technology is also used.

[0018] After a traditional CS audio phone call and a video call are established between the first terminal device and the communication gateway, the communication gateway collects the network delay rtt and the ideal time deviation data jitter on the second communication link in real time, and then estimates the one-way network delay delay_t according to the following formula:

[0019] delay_t = rtt / 2 + a * jitter;

[0020] Where the network delay rtt and the ideal time deviation data jitter are calculated according to the RTP protocol standard; a is a preset constant, and the value range is 2 to 5, and the preferred value is 3;

[0021] Then update the delay synchronization variable delta_t:

[0022] delta_t = max(0, min(delay_t - ct, b)), where ct is a constant, meaning the fixed delay of the audio call, and the default value is set to 20ms; b is a preset constant, and the value range is 450 to 550, and the preferred value is 500;

[0023] Finally, the communication gateway sets the buffer length of NACK in the first communication link to delta_t, and realizes the synchronization of voice data and video data through the increased audio delay of NACK.

[0024] Compared with the prior art, the advantages of the present invention are: by implementing this solution, audio-video communication is realized in a heterogeneous network, so that the first terminal device and the second terminal device can simultaneously conduct a video call without interrupting the traditional CS audio phone call process. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a block diagram of a real-time data communication system in an embodiment of the present invention.

[0026] Figure 2 It is a processing flowchart of a communication gateway in an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0027] The present invention will be further described in detail below in conjunction with the embodiments and the accompanying drawings.

[0028] As Figure 1 shown, in this embodiment, the real-time data communication system includes a communication gateway capable of supporting different communication networks, and a media mixing and relaying module communicatively connected to the communication gateway, abbreviated as Mixer in the accompanying drawings. The function of the media mixing and relaying module is to mix voice data and video data and perform relaying. This media mixing and relaying module adopts existing conventional technologies and will not be elaborated in detail here; among them, the different communication networks that the communication gateway server can support at least include the operator telephone network and the Internet;

[0029] In the above real-time data communication system, the first terminal device T1 and the second terminal device T2 perform a video call while not interrupting the traditional CS audio telephone call in the following manner:

[0030] The first terminal device establishes a first communication link with the communication gateway through the operator telephone network, so as to establish a traditional CS audio telephone call between the first terminal device and the communication gateway. The first terminal device sends the voice data to the communication gateway through the first communication link; then the first terminal device establishes a second communication link with the communication gateway through the Internet, so as to establish a video call between the first terminal device and the communication gateway, abbreviated as PS video in the accompanying drawings. The first terminal device T1 sends the video data to the communication gateway through the second communication link;

[0031] The second terminal device T2 is communicatively connected to the communication gateway through the media mixing and relaying module Mixer; the communication gateway mixes the voice data and video data of the first terminal device through the media mixing and relaying module Mixer and sends them to the second terminal device T2; the second terminal device T2 sends its own generated voice data and video data to the communication gateway through the media mixing and relaying module Mixer, and the communication gateway then sends the voice data of the second terminal device to the first terminal device T1 through the first communication link and sends the video data of the second terminal device T2 to the first terminal device T1 through the second communication link, thereby realizing the real-time audio and video call between the first terminal and the second terminal. In this embodiment, the second terminal device T2 and the media mixing and relaying module Mixer can be communicatively connected through a local area network or the Internet, and the voice data and video data generated by the second terminal device T2 are also sent to the media mixing and relaying module Mixer through two different communication links respectively.

[0032] In addition, it further includes a third terminal device T3, and the third terminal device T3 is also communicatively connected to the communication gateway through the media mixing and relay module; after a real-time audio and video call starts between the first terminal device T1 and the second terminal device T2, the media mixing and relay module Mixer mixes the voice data and video data exchanged between the first terminal device T1 and the second terminal device T2 and then sends them to the third terminal device T3, and the third terminal device T3 records the real-time audio and video communication content between the first terminal device T1 and the second terminal device T2 to generate an audio and video file such as an MP4 file.

[0033] The first terminal device T1, the second terminal device T2, and the third terminal device T3 can be mobile phones, computers, or any other form of terminal device.

[0034] After a traditional CS audio call and video call are established between the first terminal device and the communication gateway, one problem to be solved is the audio and video synchronization problem, and at the same time, the communication quality in the case of a weak Internet network needs to be improved through QoS technology. For example Figure 2 , the core QoS control process of the communication gateway. After a traditional CS audio call and video call are established between the first terminal device and the communication gateway, first, the communication gateway closes NACK and enables forward error correction on the second communication link, and uses low-latency QoS technology. In the attached figure, forward error correction is abbreviated as SWRLC-FEC; on the first communication link between the communication gateway and the operator's telephone network, NACK technology is used; for the RTP communication between the communication gateway and the media mixing and relay module, NACK technology is also used; that is, the NACK function needs to be closed for the video link and the NACK function needs to be enabled for the audio link; then the communication gateway collects the network delay rtt and the ideal time deviation data jitter on the second communication link in real time, and then estimates the one-way network delay delay_t according to the following formula:

[0035] delay_t = rtt / 2 + a * jitter;

[0036] Among them, the network delay rtt and the ideal time deviation data jitter are calculated according to the RTP protocol standard, a is a preset constant, and the value range is 2.5 to 3.5, and the preferred value is 3;

[0037] Then update the delay synchronization variable delta_t:

[0038] delta_t = max(0, min(delay_t – ct, b)), where ct is a constant, which means the fixed delay of the audio call, and the default value is set to 20ms; b is a preset constant, and the value range is 450 to 550, and the preferred value is 500;

[0039] Finally, the communication gateway sets the buffer length of the positive feedback in the first communication link to delta_t, and realizes the synchronization of voice data and video data by increasing the delay of audio forwarding.

[0040] In this embodiment, NACK, Negative Acknowledgement, that is, "negative feedback", is a type of message in the communication protocol, indicating that the data packet is delayed or lost during transmission. When the sender sends the original data packet and the receiver does not receive it, a positive feedback message will be sent to the receiver, indicating that the receiver has not correctly received the data packet. The receiver opens a buffer area buffer and determines the waiting time t according to a certain strategy (such as rtt). If the retransmitted packet is received within the waiting time, the lost data can be restored. Therefore, this technology can be used to combat network packet loss, but the side effect is that it will increase the delay (t). Forward Error Correction: Sliding Window Random Linear Code (RLC) Forward Erasure Correction (FEC) Schemes for FEC FRAME, abbreviated as SWRLC-FEC, represents sliding window random linear coding, which is a forward error correction technology standard that can reduce delay in the TCP / IP protocol.

[0041] The RTP (Real-Time Transport Protocol) protocol is defined by RFC3550. RTP provides support for streaming media data such as audio and video streams, allowing multiple users to perform real-time transmission on the Internet. The main task of RTP is to provide real-time RTC interactive services, and provide bitstream statistical information and reliability mechanisms. The system consists of three parts: the RTP header, the RTP data packet, and RTCP (Real-Time Transport Control Protocol). The RTP header is used to transmit data on the network, and the RTP data packet is used to store media information and video packets; while RTCP is responsible for processing the traffic statistical information between data transmissions, as well as the monitoring and management of network quality. RTP supports most multimedia data formats and uses UDP to improve transmission efficiency. In this embodiment, the network delay rtt and the ideal time deviation data jitter can be directly calculated through the RTP protocol standard, and this technology adopts existing conventional technologies.

Claims

1. A real-time data communication system, characterized in that: It includes a communication gateway that can support different communication networks, and a media mixing and transfer module communicatively connected to the communication gateway; among them, the different communication networks that the communication gateway server can support at least include the operator telephone network and the Internet; In the above real-time data communication system, the first terminal device and the second terminal device perform a video call simultaneously without interrupting the traditional CS audio telephone call in the following manner: The first terminal device establishes a first communication link with the communication gateway through the operator telephone network, so as to establish a traditional CS audio telephone call between the first terminal device and the communication gateway, and the first terminal device sends voice data to the communication gateway through the first communication link; Then the first terminal device establishes a second communication link with the communication gateway through the Internet, so as to establish a video call between the first terminal device and the communication gateway, and the first terminal device sends video data to the communication gateway through the second communication link; The second terminal device is communicatively connected to the communication gateway through the media mixing and transfer module; the communication gateway mixes the voice data and video data of the first terminal device through the media mixing and transfer module and then sends them to the second terminal device; the second terminal device sends the voice data and video data generated by itself to the communication gateway through the media mixing and transfer module, and the communication gateway then sends the voice data of the second terminal device to the first terminal device through the first communication link, and sends the video data of the second terminal device to the first terminal device through the second communication link, so as to realize the real-time audio and video call between the first terminal and the second terminal.

2. The real-time data communication system according to claim 1, wherein: It further includes a third terminal device, and the third terminal device is also communicatively connected to the communication gateway through the media mixing and transfer module; after the real-time audio and video call starts between the first terminal device and the second terminal device, the media mixing and transfer module mixes the voice data and video data exchanged between the first terminal device and the second terminal device and then sends them to the third terminal device, and the third terminal device records the real-time audio and video communication content between the first terminal device and the second terminal device to generate an audio and video file.

3. The real-time data communication system according to claim 1, wherein: After a traditional CS audio telephone call and a video call are established between the first terminal device and the communication gateway, the communication gateway closes NACK and turns on forward error correction on the second communication link, and uses low-latency QoS technology; On the first communication link between the communication gateway and the operator telephone network, the NACK technology is used; for the RTP communication between the communication gateway and the media mixing and transfer module, the positive feedback technology is also used.

4. The real-time data communication system according to claim 3, characterized in that: After a traditional CS audio telephone call and a video call are established between the first terminal device and the communication gateway, the communication gateway real-time collects the network delay rtt and the ideal time deviation data jitter on the second communication link, and then estimates the one-way network delay delay_t according to the following formula: delay_t = rtt / 2 + a * jitter; Wherein the network delay rtt and the ideal time deviation data jitter are calculated according to the RTP protocol standard, and a is a preset constant; Then update the delay synchronization variable delta_t: delta_t = max(0, min(delay_t – ct, b)), where ct is a preset constant, meaning the fixed delay of an audio call, and b is a preset constant; Finally, the communication gateway sets the buffer length of the positive feedback in the first communication link to delta_t, and realizes the synchronization of voice data and video data by increasing the delay of audio forwarding.

Images