Point-to-point audio and video call system based on custom signaling

Abstract

The invention discloses a point-to-point audio and video call system based on custom signaling. A media plane adopts a mature WebRTC protocol, and a signaling plane designs a load domain of the MQTT protocol by taking the MQTT protocol as a carrier to form a custom signaling. The signaling naming mode refers to the SIP protocol. An MQTT server needs to be set up to transmit signaling, and the server only carries out forwarding and does not carry out any processing on a signaling message. A client needs to customize a signaling state machine to realize session management so as to support the functions of call initiation, automatic hanging up when an opposite party is busy and the like. Different from the SIP protocol, the self-defined signaling does not need a special protocol server for protocol processing, so that the call establishment time delay is shortened. Because the MQTT protocol is a binary protocol, compared with the SIP protocol, the MQTT-based self-defined signaling has the advantages that the flow consumption of hundreds of bytes is reduced on the signaling surface, and the bandwidth waste is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time audio and video calls, and in particular to a point-to-point audio and video call system based on user-defined signaling. Background Art

[0002] WebRTC (Web Real-Time Communications) is a technology for implementing real-time audio, video, and data transmission between browsers. It allows developers to use JavaScript APIs and HTML5 to create real-time communication applications such as voice calls, video chats, and file sharing. It is based on the network transmission protocols UDP and TCP, uses STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relay NAT) protocols to traverse various types of network address translation (NAT) devices, and implements end-to-end encryption to protect the security of communication content. WebRTC can run on multiple platforms, including operating systems such as Windows, macOS, Linux, Android, and iOS.

[0003] MQTT is a lightweight message transmission protocol based on the publish / subscribe model, which has the following advantages: simple and easy to implement, supports QoS, is lightweight and saves bandwidth, is data-independent (does not care about the Payload data format), and has continuous session awareness. It supports efficient data communication through the publish / subscribe model and realizes real-time data push without the need for devices to constantly poll the server, thereby reducing network overhead and power consumption. Summary of the invention

[0004] The present invention provides a point-to-point audio and video call system based on custom signaling, aiming to take advantage of the characteristics of the custom signaling protocol, such as light weight, flexibility and easy expansion, to achieve the purpose of shortening the call establishment delay and reducing the waste of signaling plane bandwidth.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0006] A point-to-point audio and video call method based on custom signaling includes a calling end and a called end, wherein the calling end performs the following steps:

[0007] Select the call object to initiate an audio or video call, set the caller state machine state switch, store the call object user name, create a PeerConnection; create a local session description object SDP and put it in the Invite message, and send the Invite message to the called party;

[0008] When receiving the 100Trying message, the ice candidate object IceCandidate is placed in the Candidate message and sent to the called party to start the exchange of ice candidate objects;

[0009] When the 100Trying message is received and the ICE and SDP are negotiated successfully, the calling state machine state is switched;

[0010] After receiving the 180Ringing message, the calling state machine switches to a new state, completes call preparation, and waits for the called party to answer the call.

[0011] The calling state machine includes five states: IDLE, WAITING, PROGRESSING, CONNECTED, and CALLING. The state switching is as follows:

[0012] IDLE sends an Invite message to WAITING;

[0013] WAITING switches to IDLE after two timeouts or receiving a 486Busy message;

[0014] WAITING receives a 100Trying message and both ICE and SDP negotiation are successful and switches to PROGRESSING;

[0015] PROGRESSING receives 180Ringing message and moves to CONNECTED;

[0016] After receiving the 200OK (Type 4) message from CONNECTED, the message goes to CALLING.

[0017] CALLING sends a BYE message or receives a BYE message and goes to IDLE;

[0018] CONNECTED, PROGRESSING or WAITING sends a BYE message back to IDLE;

[0019] CONNECTED, PROGRESSING or WAITING returns to IDLE after receiving 603Decline message.

[0020] Only when both state machines of the calling party are in the IDLE state can a new call be initiated or a new call request be accepted.

[0021] A point-to-point audio and video call method based on custom signaling includes a calling end and a called end, wherein the called end performs the following steps:

[0022] When receiving the Invite message, if the called party's state machine is in the IDLE state, set the called party's state machine state switch, store the call object's user name, and create a PeerConnection; set the remote SDP, create a local SDP and put it in the 100Trying message, send the 100Trying message to the calling party, and set the called party's state machine state switch;

[0023] After setting the local SDP, put the IceCandidate into the Candidate message and send it to the calling party to start the exchange of ice candidates;

[0024] After ICE and SDP are successfully negotiated, a 180Ringing message is sent, and the called party's state machine is switched. At the same time, click Answer to make an audio or video call.

[0025] The called state machine includes five states: IDLE, INITIAL, PROGRESSING, CONNECTED, and CALLING. The state switching is as follows:

[0026] IDLE receives the Invite message and goes to INITIAL;

[0027] INITIAL sends 100Trying message to PROGRESSING;

[0028] PROGRESSING switches to CONNECTED after both ICE and SDP are successfully negotiated and a 180Ringing message is sent;

[0029] CONNECTED sends a 200OK (Type 4) message and then moves to CALLING;

[0030] CALLING sends a BYE message or receives a BYE message and goes to IDLE;

[0031] CONNECTED, PROGRESSING or INITIAL sends a 603Decline message back to IDLE;

[0032] CONNECTED, PROGRESSING, or INITIAL receives BYE and returns to IDLE.

[0033] Only when both state machines of the called party are in the IDLE state can a new call be initiated or a new call request be accepted.

[0034] A point-to-point audio and video call system based on custom signaling includes a server and two clients, the two clients serving as a calling end and a called end respectively, wherein:

[0035] The client is used to perform signaling interaction with the server to realize audio and video calls;

[0036] The server is used to forward signaling messages and IceCandidate between two clients.

[0037] The server comprises:

[0038] MQTT server, used to forward signaling messages between two clients;

[0039] ICE server, used to provide IceCandidates to two clients.

[0040] The client comprises:

[0041] MQTT client module, used to interact with the WebRTC client module and call the custom signaling encapsulation and parsing module to encapsulate and parse custom signaling;

[0042] Custom signaling encapsulation and parsing module, used to encode, send, receive and decode the MQTT payload field according to the custom signaling format;

[0043] WebRTC client module, used to interact with the MQTT client module;

[0044] The state machine module is used to perform client state transition according to the received and sent signaling.

[0045] The present invention has the following beneficial effects and advantages:

[0046] 1. A lightweight point-to-point audio and video call system is implemented. Due to the custom signaling, a dedicated protocol server is not required for protocol processing, which reduces the complexity of the system.

[0047] 2. The calling state machine and called state machine are implemented on the client to maintain the client's session status, thereby supporting functions such as detecting that the other party is busy and detecting that the other party has not answered the call for a long time.

[0048] 3. Implement custom signaling based on the MQTT protocol. MQTT is a binary protocol, which makes custom signaling naturally encrypted. Compared with the SIP protocol, MQTT has a very simple header, and the payload field is encoded in binary form, so that custom signaling can save hundreds of bytes of traffic and reduce bandwidth waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a state transition diagram of the calling state machine;

[0050] Figure 2It is a state transition diagram of the called state machine;

[0051] Figure 3 It is a schematic diagram of the system module composition. DETAILED DESCRIPTION

[0052] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0053] A point-to-point audio and video call system based on custom signaling, comprising:

[0054] The caller (hereinafter referred to as the caller) selects the callee to initiate an audio or video call, sets the caller state machine state switching, stores the callee user name, and creates a PeerConnection; creates a local SDP (Session Description Protocol) and puts it into a custom Invite message, and sends the Invite message to the peer end;

[0055] The called party (hereinafter referred to as the called party) receives the custom Invite message. If the called party's state machine is in the IDLE state, the called party's state machine state switch is set, the call object user name is stored, and PeerConnection is created; the remote SDP is set, the local SDP is created and put into the custom 100Trying message, the 100Trying message is sent to the other party, and the called party's state machine state switch is set; after setting the local SDP, the called party puts the IceCandidate (ice candidate object) into the custom Candidate message and sends it to the other party (i.e., the ice candidate object exchange process is started);

[0056] The caller receives the custom 100Trying message, puts the IceCandidate into the custom Candidate message and sends it to the peer (starting the ice candidate exchange process). After receiving the 100Trying message and ICE and SDP negotiation succeeds, the caller state machine switches to the state;

[0057] After the ICE and sdp negotiation is successful, the called party sends a custom 180Ringing message and switches the called party's state machine; after receiving the custom 180Ringing message, the calling party switches the calling party's state machine. At this point, all preparations for the call have been completed, and the two parties can have an audio and video call as long as the called party clicks the answer button.

[0058] It also includes: the client contains two state machines, the calling state machine and the called state machine. When the user logs in to the client, both the calling state machine and the called state machine are in the IDLE state. When the caller initiates a call, the calling state machine starts a series of state switching; when the called party receives a call request, the called state machine starts a series of state switching. Only when both state machines of the client are in the IDLE state can a new call be initiated; only when both state machines of the client are in the IDLE state can a new call request be accepted, otherwise the other party will be prompted to be busy.

[0059] The calling state machine includes five states: IDLE, WAITING, PROGRESSING, CONNECTED, and CALLING.

[0060] The transitions between the five states of the calling state machine are as follows: IDLE sends a custom Invite message (a message type will be mentioned later, and the word "custom" will not be repeated) to WAITING, WAITING goes to IDLE after two timeouts or receives a 486 Busy message, WAITING receives a 100 Trying message and the ICE negotiation is successful and the sdp negotiation is successful to PROGRESSING, PROGRESSING receives a 180 Ringing message to CONNECTED, CONNECTED receives a 200 OK (Type 4) message to CALLING, CALLING sends a BYE message or receives a BYE message to IDLE; in addition, CONNECTED / PROGRESSING / WAITING sends a BYE message back to IDLE; CONNECTED / PROGRESSING / WAITING receives a 603 Decline message to return to IDLE.

[0061] The called state machine includes five states: IDLE, INITIAL, PROGRESSING, CONNECTED, and CALLING.

[0062] The transitions between the five states of the called state machine are as follows: IDLE receives an Invite message and switches to INITIAL; INITIAL sends a 100Trying message and switches to PROGRESSING; PROGRESSING switches to CONNECTED after ICE and sdp negotiation succeed and a 180Ringing message is sent; CONNECTED switches to CALLING after sending a 200OK (Type 4) message; CALLING sends a BYE message or receives a BYE message and switches to IDLE; CONNECTED / PROGRESSING / INITIAL sends a 603Decline message and returns to IDLE; CONNECTED / PROGRESSING / INITIAL receives BYE and returns to IDLE.

[0063] A point-to-point audio and video call system based on custom signaling includes: a client and a server; the client includes an MQTT client module, a WebRTC client module, a custom signaling encapsulation and parsing module, and a state machine module; the server includes an MQTT server and an ICE server, the MQTT server only forwards signaling messages and does not make any modifications to the signaling messages, and the ICE server uses a public server and does not need to be built by itself.

[0064] The MQTT client module will automatically subscribe to the topic / signaling / custom / USERNAME after the user logs in to the client. USERNAME is the user name when the user logs in to the client. When user A needs to send a signaling message to user B, he only needs to send the message to the corresponding topic.

[0065] The WebRTC client modules need to exchange SDP objects and IceCandidate objects. The SDP object is used to describe the audio and video codecs, transmission protocols and other parameters supported by the client, and the IceCandidate object describes the protocols and routes required for WebRTC to communicate with the other end. When both SDP and ICE are successfully negotiated, WebRTC is ready to transmit media streams.

[0066] The custom signaling encapsulation and parsing module encodes and sends and receives and decodes the MQTT payload field according to the custom signaling format. The naming method of the custom signaling message refers to the naming of the SIP protocol.

[0067] The state machine module initializes the calling state machine and the called state machine after the user logs in to the client. At this time, both state machines are in the IDLE state. When the calling party initiates a call, the state of the calling state machine changes accordingly; when the called party accepts a call, the state of the called state machine changes accordingly.

[0068] An embodiment of the present invention provides a method and steps for state conversion of a calling state machine. Figure 1 As shown in the example, the following steps are included:

[0069] Step 1.0: The calling state machine is initialized and is in the IDLE state.

[0070] Step 1.1: In the IDLE state, the caller sends an Invite message and switches to the WAITING state.

[0071] Step 1.2: In the WAITING state, if the caller receives the 100Trying message and the ICE and sdp negotiation succeeds, the caller switches to the PROGRESSING state. If the ICE negotiation fails or the sdp negotiation fails, the caller switches to the IDLE state.

[0072] Step 1.3: In the PROGRESSING state, the caller receives the 180Ringing message and switches to the CONNECTED state.

[0073] Step 1.4: In the CONNECTED state, the calling party receives a 200 OK (Type 4) message and switches to the CALLING state. At this time, the calling party and the called party start talking.

[0074] Step 1.5: In the CALLING state, the caller sends or receives a BYE message and switches to the IDLE state.

[0075] Supplementary explanation: The caller sends a BYE message during CONNECTED, PROGRESSING or WAITING to switch to IDLE state; the caller receives a 603Decline message during CONNECTED, PROGRESSING or WAITING to switch to IDLE state.

[0076] An embodiment of the present invention provides a method and steps for converting the state of a called state machine. Figure 2 As shown in the example, the following steps are included:

[0077] Step 2.0: The called party state machine is initialized and is in the IDLE state.

[0078] Step 2.1: In the IDLE state, the called party receives the Invite message and switches to the INITIAL state.

[0079] Step 2.2: In the INITIAL state, the called party sends a 100Trying message and switches to the PROGRESSING state.

[0080] Step 2.3: In the PROGRESSING state, if the called party's ICE and sdp negotiation succeeds and sends a 180Ringing message, the state changes to the CONNECTED state; if the ICE or sdp negotiation fails, the state changes to the IDLE state.

[0081] Step 2.4: In the CONNECTED state, the called party sends a 200OK (Type 4) message and switches to the CALLING state. At this time, the calling and called party start talking.

[0082] Step 2.5: In the CALLING state, the called party sends a BYE message or receives a BYE message and switches to the IDLE state.

[0083] Supplementary explanation: The called party receives a BYE message during CONNECTED, PROGRESSING or INITIAL to switch to IDLE state; the called party sends a 603Decline message during CONNECTED, PROGRESSING or INITIAL to switch to IDLE state.

[0084] The present invention provides a custom signaling format based on the MQTT protocol. The signaling message Payload format is defined as shown in Table 1:

[0085] Table 1

[0086]

[0087] The first four bytes represent the creation time of the Sig Msg message, and the fifth byte contains the type and reservation of the Sig Msg message. The message supports up to 16 types and has good scalability. When Sig Msg Type is 0, it indicates a point-to-point call signaling message. When the service needs to be expanded, the message type can be expanded accordingly. The sixth byte is the sender name length, and the seventh to nth bytes are the sender name. The n+1 to ... bytes are the Sig Msg message content.

[0088] The specific content encoding format of the Sig Msg message is shown in Table 2, and the encoding, meaning and corresponding Content of Content Type are shown in Table 3 (the naming method of the message type refers to the SIP protocol).

[0089] Table 2

[0090]

[0091] Table 3

[0092]

[0093]

[0094] The present invention provides an audio and video call system based on custom signaling, such as Figure 3 , its module composition and interaction between modules are as follows:

[0095] The client includes the MQTT client module, the custom signaling encapsulation and parsing module, the WebRTC client module, and the state machine module. The server includes the MQTT server and the ICE server.

[0096] The MQTT client module and the WebRTC client module exchange information such as sdp and IceCandidate. The MQTT client module calls the custom signaling encapsulation and parsing module to encapsulate and parse custom signaling.

[0097] The two MQTT client modules forward custom signaling through the MQTT server, and the ICE server provides IceCandidate information to the WebRTC client module.

[0098] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should be regarded as within the scope of protection of the present invention.

Claims

1. A point-to-point audio and video call method based on custom signaling, including a calling end and a called end, characterized in that: The calling end performs the following steps: Select the call object to initiate an audio or video call, set the caller state machine state switch, store the call object user name, create a PeerConnection; create a local session description object SDP and put it in the Invite message, and send the Invite message to the called party; When receiving the 100Trying message, the ice candidate object IceCandidate is placed in the Candidate message and sent to the called party to start the exchange of ice candidate objects; When the 100Trying message is received and the ICE and SDP are negotiated successfully, the calling state machine state is switched; After receiving the 180Ringing message, the calling state machine switches to a new state, completes call preparation, and waits for the called party to answer the call.

2. According to claim 1, a point-to-point audio and video call method based on custom signaling is characterized in that: The calling state machine includes five states: IDLE, WAITING, PROGRESSING, CONNECTED, and CALLING. The state switching is as follows: IDLE sends an Invite message to WAITING; WAITING switches to IDLE after two timeouts or receiving a 486Busy message; WAITING receives a 100Trying message and ICE and SDP negotiation are successful, then switches to PROGRESSING. PROGRESSING receives a 180Ringing message, then switches to CONNECTED. After receiving the 200OK (Type 4) message from CONNECTED, the message goes to CALLING. CALLING sends a BYE message or receives a BYE message and goes to IDLE; CONNECTED, PROGRESSING or WAITING sends a BYE message back to IDLE; CONNECTED, PROGRESSING or WAITING returns to IDLE after receiving 603Decline message.

3. According to claim 1, a point-to-point audio and video call method based on custom signaling is characterized in that: Only when both state machines of the calling party are in the IDLE state can a new call be initiated or a new call request be accepted.

4. A point-to-point audio and video call method based on custom signaling, including a calling end and a called end, characterized in that: The called end performs the following steps: When receiving the Invite message, if the called party's state machine is in the IDLE state, set the called party's state machine state switch, store the call object's user name, and create a PeerConnection; set the remote SDP, create a local SDP and put it in the 100Trying message, send the 100Trying message to the calling party, and set the called party's state machine state switch; After setting the local SDP, put the IceCandidate into the Candidate message and send it to the calling party to start the exchange of ice candidates; After ICE and SDP are successfully negotiated, a 180Ringing message is sent, and the called party's state machine is switched. At the same time, click Answer to make an audio or video call.

5. A point-to-point audio and video call method based on custom signaling according to claim 4, characterized in that: The called state machine includes five states: IDLE, INITIAL, PROGRESSING, CONNECTED, and CALLING. The state switching is as follows: IDLE receives the Invite message and goes to INITIAL; INITIAL sends 100Trying message to PROGRESSING; PROGRESSING switches to CONNECTED after both ICE and SDP are successfully negotiated and a 180Ringing message is sent; CONNECTED sends a 200OK (Type 4) message and then moves to CALLING; CALLING sends a BYE message or receives a BYE message and goes to IDLE; CONNECTED, PROGRESSING or INITIAL sends a 603Decline message back to IDLE; CONNECTED, PROGRESSING, or INITIAL receives BYE and returns to IDLE.

6. A point-to-point audio and video call method based on custom signaling according to claim 4, characterized in that: Only when both state machines of the called party are in the IDLE state can a new call be initiated or a new call request be accepted.

7. A point-to-point audio and video call system based on custom signaling, characterized in that: It includes a server and two clients, where the two clients serve as the calling end and the called end respectively, wherein: The client is used to perform signaling interaction with the server to realize audio and video calls; The server is used to forward signaling messages and IceCandidate between two clients.

8. A point-to-point audio and video call system based on custom signaling according to claim 7, characterized in that: The server comprises: MQTT server, used to forward signaling messages between two clients; ICE server, used to provide IceCandidates to two clients.

9. A point-to-point audio and video call system based on custom signaling according to claim 7, characterized in that: The client comprises: MQTT client module, used to interact with the WebRTC client module and call the custom signaling encapsulation and parsing module to encapsulate and parse custom signaling; Custom signaling encapsulation and parsing module, used to encode, send, receive and decode the MQTT payload field according to the custom signaling format; WebRTC client module, used to interact with the MQTT client module; The state machine module is used to perform client state transition according to the received and sent signaling.

Images