WebRtc-based audio and video call method and system

Abstract

The present invention relates to a method and system for audio and video calls based on WebRtc, which belongs to the technical field of video calls, and solves the problems that the existing cross-platform and cross-device audio and video calls have no quality guarantee and the server allocation is inflexible. It includes: a signaling server receives and parses the call joining signaling, adds the calling / selected user as a new member to the call group, obtains the terminal IP of the new member, and obtains the ICE server between the new member and each other member in the call group according to the terminal IP of the original member in the call group, the weight and load of the ICE server; when the end-to-end connection fails to be established, the public network IP and port are obtained through the STUN service of the ICE server, and the end-to-end connection is tried again. If it still fails to be established, a relay forwarding connection is established through the TURN service of the ICE server. If any of the above connections is successfully established, the two members start the audio and video call; until the connection between any two members in the call group is successfully established. High-quality audio and video calls and dynamic resource allocation of servers are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of video calls, and in particular to a WebRtc-based audio and video call method and system. Background Art

[0002] In some monitoring scenarios or between special groups of people, it is not convenient to use conventional social software for audio and video calls. Therefore, it is necessary to provide a cross-platform, cross-device audio and video call method and system that supports audio and video calls on PC and mobile phones.

[0003] Using the support provided by a third-party platform has a high cost overhead, and it is necessary to pay fees to the third-party platform regularly based on time or traffic, which is often not friendly to the system construction budget. Directly using the existing WebRtc, although it can realize the basic functions of audio and video calls, cannot guarantee the call quality, often with high latency and freezes, and is stretched in cross-platform support. When encountering NAT network forwarding, it is impossible to directly establish an end-to-end channel and cannot perform audio and video call functions. Summary of the invention

[0004] In view of the above analysis, the embodiments of the present invention aim to provide a WebRtc-based audio and video call method and system to solve the problems of unguaranteed quality of existing cross-platform and cross-device audio and video calls and inflexible resource allocation of audio and video servers.

[0005] On the one hand, an embodiment of the present invention provides an audio and video call method based on WebRtc, comprising the following steps:

[0006] The signaling server receives and parses the first join call signaling sent by the calling user terminal, creates a talk group, adds the calling user as a new member to the talk group, and notifies the selected user;

[0007] The signaling server receives and parses the second join call signaling sent by the selected user terminal, and adds the selected user as a new member to the talk group; when the new member is added to the talk group, the new member terminal IP is obtained, and the ICE server between the new member and each other member in the talk group is obtained according to the original member terminal IP, the weight and load of the ICE server in the talk group, and the talk group is bound and a join response signaling is sent to notify the new member;

[0008] When two members in the call group complete media negotiation, they try to establish a corresponding first end-to-end connection. If the first end-to-end connection fails to be established, the two members obtain the public IP and port through the STUN service of the corresponding ICE server, and try to establish a corresponding second end-to-end connection. If the second end-to-end connection fails to be established, a relay forwarding connection is established through the TURN service of the corresponding ICE server. If any of the above connections is successfully established, the two members start an audio or video call; until the connection between any two members in the call group is successfully established.

[0009] Based on the further improvement of the above method, the method also includes: the new member receives the joining response signaling sent by the signaling server, starts the heartbeat monitoring timing task, and sends the heartbeat signaling to the signaling server according to the preset monitoring interval; and receives and counts the heartbeat response signaling sent by the signaling server. If the heartbeat response signaling is not received for more than a preset number of consecutive times, the heartbeat signaling is stopped, prompting the call members to be disconnected or the network fails.

[0010] Based on the further improvement of the above method, the method further includes: when the new member sends a call joining signaling, setting the call quality reporting state of the new member terminal to be turned on; and,

[0011] After two members start an audio or video call, they obtain the data streams of the two members' terminals, and obtain the call quality from the data stream according to the preset reporting interval and report it to the signaling server; the call quality includes: audio sending bit rate, video sending bit rate, audio receiving bit rate, video receiving bit rate, video sending width, video sending height, video actual sending frame rate and video actual receiving frame rate.

[0012] Based on the further improvement of the above method, the method further includes: before the calling user terminal sends the first call joining signaling, and before the selected user terminal sends the second call joining signaling, checking whether the corresponding calling / selected user terminal has the corresponding hardware device according to the call type, if it is detected that the hardware device is missing, an error is prompted and the call ends; otherwise, further checking whether the hardware device is available, including:

[0013] Activate the camera and microphone of the user terminal;

[0014] Use H264 encoding format to obtain the audio and video streams of the user terminal according to the initialized video parameters. If the acquisition fails, adjust to X264 encoding format to obtain again. If the acquisition still fails, the corresponding abnormal information is prompted and the call ends; if the acquisition is successful, the video stream is rendered to the call interface.

[0015] Based on the further improvement of the above method, according to the terminal IP of the original member in the talk group, the weight and load of the ICE server, the ICE server between the new member and each other member in the talk group is obtained respectively, including:

[0016] If there are original members in the talk group, the original members in the talk group are traversed, and the currently taken original members and the new members are taken as the members to be processed in turn. Otherwise, the new member is directly taken as the member to be processed, and the ICE server list is obtained by screening according to the physical location corresponding to the terminal IP of the member to be processed and the ICE server status;

[0017] According to the weight of the ICE server, starting from the ICE server with the largest weight, according to the call type of the call group, it is determined in turn whether the remaining bandwidth of the ICE server can meet the call needs of the members to be processed, and the first ICE server that meets the call needs is taken as the ICE server of the members to be processed.

[0018] Based on the further improvement of the above method, according to the physical location and ICE server status corresponding to the IP address of the member terminal to be processed, the ICE server list is screened and obtained, including:

[0019] If the physical location of any terminal of the member to be processed is abroad, obtain an ICE server whose region is abroad and whose status is online as the ICE server list;

[0020] If the physical location of any terminal of the member to be processed is not outside the country or in mainland China, then obtain an ICE server whose region is Hong Kong, China and whose status is online as the ICE server list;

[0021] If the physical locations of the member terminals to be processed are all in mainland China, then randomly select an ICE server that is in the same city or adjacent to the physical location of any member terminal to be processed and is in an online state as the ICE server list;

[0022] If the ICE server list obtained above is empty, then ICE servers with an online status are obtained from all ICE servers as the ICE server list.

[0023] Based on the further improvement of the above method, according to the call type of the call group, it is judged in turn whether the remaining bandwidth of the ICE server can meet the call demand of the member to be processed, and the first ICE server that meets the call demand is taken as the ICE server of the member to be processed, including:

[0024] Take an ICE server from the ICE server list as the current ICE server;

[0025] Based on the preset one-way video stream consumption flow and one-way audio stream consumption flow, according to the call type of the talk group and the number of members to be processed, the bandwidth required by the terminal of the member to be processed is obtained;

[0026] If the remaining bandwidth of the current ICE server is greater than the required bandwidth, the remaining bandwidth of the current ICE server can meet the call needs of the members to be processed. The remaining bandwidth of the current ICE server minus the required bandwidth is used as the updated remaining bandwidth of the current ICE server, and the weight of the current ICE server is updated, and the traversal is exited; otherwise, continue to select the next ICE server.

[0027] Based on the further improvement of the above method, the weight of the current ICE server is updated dynamically according to the ratio of the remaining bandwidth after the current ICE server is updated to the threshold bandwidth, including:

[0028] According to the threshold ratio of the current ICE server, the threshold bandwidth of the current ICE server is obtained;

[0029] The ratio of the remaining bandwidth of the current ICE server to the threshold bandwidth is calculated, and 10% is regarded as 1 point. The score corresponding to the ratio is obtained as the updated weight of the current ICE server.

[0030] On the other hand, an embodiment of the present invention provides an audio and video call system based on WebRtc, including:

[0031] The ICE server is used to receive user terminal requests, obtain the public IP and port of the user terminal through the STUN service; receive and forward user terminal data streams through the TURN service; and regularly report the bandwidth status of the ICE server to the signaling server;

[0032] The signaling server is used to manage the ICE server, and receive and parse the first call joining signaling sent by the calling user terminal, create a call group, add the calling user as a new member to the call group, and notify the selected user; the signaling server receives and parses the second call joining signaling sent by the selected user terminal, and adds the selected user as a new member to the call group; wherein, when the new member is added to the call group, the new member terminal IP is obtained, and according to the original member terminal IP, the weight and load of the ICE server in the call group, the ICE server between the new member and each other member in the call group is obtained, the call group is bound, and a join response signaling is sent to notify the new member;

[0033] The terminal call module is used for users to initiate calls, join calls and audio and video calls, including: checking whether the calling / selected user terminal has the corresponding hardware device and whether the hardware device is available according to the call type, and obtaining the terminal data stream for the available device; sending a first join call signaling to the calling user terminal, and sending a second join call signaling to the selected user terminal, and receiving a join response signaling; when two members in the call group complete media negotiation, try to establish a corresponding first end-to-end connection. If the first end-to-end connection fails to be established, the two members obtain the public network IP and port through the STUN service of the corresponding ICE server, and try to establish a corresponding second end-to-end connection. If the second end-to-end connection fails to be established, a relay forwarding connection is established through the TURN service of the corresponding ICE server. If any of the above connections is successfully established, the two members start an audio and video call; until the connection between any two members in the call group is successfully established.

[0034] Further improvements based on the above system include:

[0035] The terminal heartbeat monitoring module is used to start the heartbeat monitoring timing task, send the heartbeat signaling to the signaling server according to the preset monitoring interval; and receive and count the heartbeat response signaling sent by the signaling server. If the heartbeat response signaling is not received for more than the preset number of consecutive times, the heartbeat signaling is stopped, prompting the call member to disconnect or the network is faulty;

[0036] The call quality reporting module is used to set the call quality reporting status of the new member's terminal to on when the new member sends a signal to join the call; and after two members start an audio or video call, the data streams of the two member terminals are obtained, and the call quality is obtained from the data stream according to the preset reporting interval and reported to the signaling server; the call quality includes: audio sending bit rate, video sending bit rate, audio receiving bit rate, video receiving bit rate, video sending width, video sending height, video actual sending frame rate and video actual receiving frame rate.

[0037] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0038] 1. Realize audio and video calls between PC web page, PC application and mobile terminals. Design a set of signaling rules for the three terminals with millisecond timestamps for easy monitoring and processing;

[0039] 2. The ICE server has both STUN and TURN functions, and performs relay forwarding when an end-to-end connection cannot be established;

[0040] 3. A signaling service layer is established before the ICE server to manage each group of audio and video calls and the ICE server. By locating the user terminal IP, the nearest server is preferentially selected for users making audio and video calls to improve the response speed; by dynamically updating the remaining traffic and weight of the ICE server, the server with the least load is selected for the user to avoid delays and freezes; at the same time, the bandwidth and traffic of each ICE server are dynamically monitored, and through log analysis, it supports dynamic setting of the ICE server's online and offline, so as to make full and reasonable use of resources and avoid waste.

[0041] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0043] Figure 1 This is a flow chart of the audio and video call method based on WebRtc in Example 1 of the present invention. DETAILED DESCRIPTION

[0044] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0045] Embodiment 1,

[0046] A specific embodiment of the present invention discloses an audio and video calling method based on WebRtc, which supports audio and video calling between three ends: a PC web page end, a PC application end, and a mobile end.

[0047] It should be noted that the PC web version runs directly based on the computer's browser, and the PC application version uses the Electron packaging technology framework to package programs suitable for the browser into independently run applications, which are downloaded and installed on the computer side with an embedded browser; the mobile side is an APP packaged based on the mobile phone operating system, which is downloaded and installed on the mobile phone side.

[0048] like Figure 1 As shown, the audio and video call method based on WebRtc includes the following steps:

[0049] S10: The user logs in to the system, checks the terminal hardware device according to the selected call type, selects a call user, initiates a call as a calling user, and sends a first join call signaling to the signaling server;

[0050] Specifically, taking user A as an example, the process of initiating a call includes the following steps:

[0051] ① After user A logs in to the system, he connects to the signaling server and sends a login signaling to the signaling server for permission verification. If the verification result is no permission, the user is prompted that he has no permission and the call ends;

[0052] It should be noted that the message exchange between the user terminal and the signaling server is carried out through the signaling server to assist the user terminals in the early media negotiation, network negotiation, etc. Among them, the message is a signaling message encapsulated based on the signaling rules, with a millisecond-level timestamp, which is easy to monitor and process.

[0053] Specifically, signaling rules include protocol attributes, command instructions, and parameters required in command instructions. Protocol attributes include: req, resp, and relay, where req means that the signaling is sent from the terminal to the server, resp and relay mean that it is sent from the server to the terminal, the difference is that resp means the response to req or relay, and relay means the forwarding of req.

[0054] The command instructions are used to perform operations according to the passed parameters and return the operation results. The command instructions include but are not limited to: login logon, join call join, leave leave, get offer credential offer, get answer credential answer, get candidate credential candidate, heartbeat response respKeepLive, report call quality reportInfo, channel connection peerConnected, join response respJoin, leave response respLeave, offer response respOffer, answer response respAnswer, candidate response respCandidate, switch call response respTurnTalkType, message response generalMsgResp, leave notification relayLeave, offer notification relayOffer, answer notification relayAnswer, candidate notification relayCandidate, switch call notification relayTurnTalkType, new connection newPeer, new connection notification notifyNewPeer and reconnection notification renewPeer.

[0055] Exemplarily, when a user logs into the audio and video call system and establishes a connection with the signaling server, the token generated by the login is used as the signaling parameter of the login signaling logon, and the login signaling is sent to the signaling server for identity and authority verification. The signaling server responds whether the user has the authority to use the audio and video call.

[0056] ② User A selects the call type and multiple call users, initiates the call, and checks whether the terminal has the corresponding hardware device according to the call type. If the corresponding hardware device is missing, the user is prompted with an error and the call ends;

[0057] It should be noted that call types include voice calls and video calls. Generally speaking, voice calls require a microphone and video calls require a camera.

[0058] This embodiment customizes different terminals to support the use of different terminals.

[0059] Exemplarily, for the encapsulated interface for detecting the status of the camera and microphone, for the PC web page and PC application that support the browser, the API provided by navigator.mediaDevices is encapsulated, and the enumerateDevices interface is used to check whether there is a camera and a microphone, and the getUserMedia interface is used to pass {video:true,audio:false} and {video:false, audio:true} to respectively check whether there is permission to call to open; on the Android mobile phone side, the camera's open is encapsulated to check whether it is available, and the microphone is available by determining the status of AudioRecord and encapsulating startRecording.

[0060] It should be noted that although the audio and video call establishment process can be extended to any number of terminals, the terminal needs to process one more audio and video stream for each additional party. Therefore, based on the performance of the call equipment, it is recommended not to exceed 9 people when selecting call users, depending on the performance of the signaling server and the terminal.

[0061] ③ If user A's terminal has the corresponding hardware equipment, the video parameters are initialized, the call quality reporting status of the user terminal is set to on, and further checks are made whether the camera and microphone are available. If the equipment is not available, the user is prompted with corresponding information and the call ends; the video parameters include resolution and frame rate. For example, the resolution is set to 640*480 and the frame rate is set to 10 frames per second.

[0062] It should be noted that further checking whether the hardware device is available includes:

[0063] Activate the terminal's camera and microphone;

[0064] Use H264 encoding format to obtain the audio and video streams of the terminal according to the initialized video parameters. If the acquisition fails, adjust to X264 encoding format to obtain again. If the acquisition still fails, the corresponding abnormal information is prompted and the call ends.

[0065] ④ If the terminal device of user A is available normally, the user terminal data stream, including audio stream and video stream, is obtained according to the video parameters, and the video stream is rendered to the interface. At the same time, a call joining signaling "join" is sent to the signaling server as the first call joining signaling.

[0066] S11: The signaling server receives and parses the first call joining signaling sent by the calling user terminal, creates a talk group, adds the calling user as a new member to the talk group, selects an ICE server according to the calling user terminal IP, the weight and load of the ICE server, and notifies the selected user;

[0067] It should be noted that the signaling server sends a new connection signaling to the selected user to notify the selected user. The new connection signaling includes: current time, talk group ID, call type and selected user ID.

[0068] Specifically, the subsequent processing step S11 of step S10 includes the following steps:

[0069] ① The signaling server receives the call joining signaling sent by user A, parses the call joining signaling, creates a call group, adds call member A as a new member to the call group, obtains user A's terminal IP, and selects an ICE server based on the terminal IP, the weight and load of the ICE server;

[0070] It should be noted that whenever a new member is added to the talk group, an appropriate ICE (Interactive Connectivity Establishment) server will be selected based on the new member's terminal IP. The ICE server also has STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) functions.

[0071] The information of the ICE server is pre-registered on the signaling server. The registration information includes: unique identification ID, server IP, ICE server type, negotiation strategy, real-time transport control protocol multi-path strategy, relay service port, hole punching service port, user name, password, loadable bandwidth and threshold ratio. According to the server IP, the country, province or state, and city are obtained and saved, the initial remaining bandwidth is set according to the loadable bandwidth, and the weight of the ICE server is initialized.

[0072] User A is the calling user who initiates the call. When joining the talk group, he is a new member of the talk group. There are no other members in the talk group. When selecting the ICE server, only the terminal IP of user A, the weight and load of the ICE server need to be considered, including:

[0073] The new member is regarded as a member to be processed, and the ICE server list is obtained by screening according to the physical location and ICE server status corresponding to the terminal IP of the member to be processed;

[0074] According to the weight of the ICE server, starting from the ICE server with the largest weight, according to the call type of the call group, it is determined in turn whether the remaining bandwidth of the ICE server can meet the call needs of the members to be processed, and the first ICE server that meets the call needs is taken as the ICE server of the members to be processed.

[0075] Specifically, according to the physical location and ICE server status corresponding to the IP address of the member terminal to be processed, the ICE server list is obtained by screening, including:

[0076] If the physical location of any terminal of the member to be processed is abroad, obtain an ICE server whose region is abroad and whose status is online as the ICE server list;

[0077] If the physical location of any terminal of the member to be processed is not outside the country or in mainland China, then obtain an ICE server whose region is Hong Kong, China and whose status is online as the ICE server list;

[0078] If the physical locations of the member terminals to be processed are all in mainland China, then randomly select an ICE server that is in the same city or adjacent to the physical location of any member terminal to be processed and is in an online state as the ICE server list;

[0079] If the ICE server list obtained above is empty, then ICE servers with an online status are obtained from all ICE servers as the ICE server list.

[0080] After obtaining the ICE server list, according to the call type of the call group, the first ICE server that meets the call requirements is selected as the ICE server of the member to be processed, including:

[0081] Take an ICE server from the ICE server list as the current ICE server;

[0082] Based on the preset one-way audio and video stream consumption flow and one-way audio stream consumption flow, according to the call type of the talk group and the number of members to be processed, the bandwidth required by the member terminal to be processed is obtained;

[0083] If the remaining bandwidth of the current ICE server is greater than the required bandwidth, the remaining bandwidth of the current ICE server can meet the call needs of the members to be processed. The remaining bandwidth of the current ICE server minus the required bandwidth is used as the updated remaining bandwidth of the current ICE server, and the weight of the current ICE server is updated, and the traversal is exited; otherwise, continue to select the next ICE server.

[0084] It should be noted that, by analyzing the log of the ICE server, it can be obtained that the bandwidth required for a one-way group of audio and video streams is in the range of 200K / S to 300K / S, and the bandwidth required for a one-way group of audio streams is in the range of 16K / S to 64K / S. Preferably, the traffic consumption of the one-way audio and video stream is set to 260K / S, and the traffic consumption of the one-way audio stream is set to 40K / S.

[0085] For example, if the remaining bandwidth of the ICE server with the largest weight is 2000K / S, and there is only user A at present, if a video call is initiated, the required bandwidth is 260K / S. The remaining bandwidth of this ICE server is greater than 260K / S and can meet the call demand. After selecting this ICE server, the remaining bandwidth is updated to 2000-260=1740K / S.

[0086] Once the current ICE server is selected, the weight of the ICE server is dynamically updated according to the ratio of the updated remaining bandwidth of the current ICE server to the threshold bandwidth, including:

[0087] According to the threshold ratio of the current ICE server, the threshold bandwidth of the current ICE server is obtained;

[0088] The ratio of the remaining bandwidth of the current ICE server to the threshold bandwidth is calculated, and 10% is regarded as 1 point. The score corresponding to the ratio is obtained as the updated weight of the current ICE server.

[0089] It should be noted that different threshold ratios are set according to the primary and backup responsibilities of the ICE server and whether other applications are deployed. Generally, it is 75% to 90% of the server's loadable bandwidth. When the used bandwidth reaches the threshold bandwidth, an email alarm is sent.

[0090] For example, the threshold ratio of the main ICE server in a city is 90%. If the loadable bandwidth is 10000K / S, the threshold bandwidth is 9000K / S. When the remaining bandwidth is 3600K / S, the corresponding weight is 4. When the remaining bandwidth is 1000K / S, an email alarm is issued.

[0091] The above method selects the ICE servers that are relatively close according to the physical location of the user terminal IP, and then obtains the most suitable ICE server from the servers with large weights according to the load, thereby improving the response speed and call quality, dynamically updating the remaining bandwidth and weight, balancing the pressure between servers, and optimizing the resource allocation between servers.

[0092] ② Bind the talk group to the ICE server, send a join response signaling respJoin to user A, inform the ICE server information, and send a new connection signaling newPeer to the selected user to notify him to join the talk group;

[0093] ③ User A receives the join response signaling returned by the signaling server, parses the join response signaling, obtains the ICE server information, starts the scheduled heartbeat service, and enters the waiting mode.

[0094] It should be noted that when a new member receives a join response signal from the signaling server, the heartbeat monitoring timing task is started to detect whether the user is truly and effectively online, so as to promptly inform the user of the status of group members. When there are no members in the call group, the ICE server is unbound in time to recycle resources and maximize resource utilization.

[0095] Specifically, according to the preset monitoring interval, the user terminal sends a heartbeat signaling to the signaling server; and receives and counts the heartbeat response signaling sent by the signaling server. If the heartbeat response signaling is not received for more than a preset number of consecutive times, the heartbeat signaling is stopped, prompting the call members to disconnect or the network fails.

[0096] Preferably, the monitoring interval is set to 5 seconds, and if no heartbeat response signaling is received for 5 consecutive times, the sending of heartbeat signaling is stopped.

[0097] S12: The selected user receives the new connection signaling, checks the terminal hardware device, and sends a second join call signaling to the signaling server;

[0098] Specifically, the selected user can be one or more users. Taking user B as the selected user, the process of joining the call is as follows:

[0099] ① The selected user B receives the new connection signaling newPeer sent by the signaling server, parses the signaling to obtain the corresponding call type, and starts the local terminal device check. If the corresponding hardware device is missing, the user is prompted with an error and the call ends;

[0100] ② If the selected user B terminal has the corresponding hardware equipment, the video parameters are initialized, the call quality reporting status of the user terminal is set to on, and the camera and microphone are further checked to see if they are available. If the equipment is not available, the user is prompted with the corresponding information and the call ends;

[0101] ③ If the device of the selected user B is available normally, the terminal data stream of the selected user B is obtained according to the video parameters, including the audio stream and the video stream, and the video stream is rendered to the interface. At the same time, a call joining signaling "join" is sent to the signaling server as the second call joining signaling.

[0102] S13: The signaling server receives and parses the second join call signaling sent by the selected user terminal, adds the selected user as a new member to the talk group, obtains the ICE server between the new member and each other member in the talk group according to the IP address of the original member terminal in the talk group, the weight and load of the ICE server, binds the talk group and sends a join response signaling to notify the new member;

[0103] The processing process of this step is similar to that of step S11, except that: there is no need to create a talk group again, and the signaling server directly adds the selected user as a new member to the talk group; when selecting an ICE server, the selected user as a new member needs to select an ICE server from each other member in the talk group, that is, traverse the original members in the talk group, and take the original members and new members currently taken out as members to be processed in turn, and filter out the ICE server list according to the physical location corresponding to the terminal IP of the member to be processed and the status of the ICE server; then, according to the weight of the ICE server, starting from the ICE server with the largest weight, and according to the call type of the talk group, judge in turn whether the remaining bandwidth of the ICE server can meet the call needs of the member to be processed, and take the first ICE server that meets the call needs as the ICE server of the member to be processed.

[0104] After determining the ICE servers between the new member and each other member in the talk group, the talk group is bound to the corresponding ICE server, and the ICE server information is sent to the talk group users. At the same time, the signaling server sends a new connection notification signaling notifyNewPeer to other members in the talk group to inform them that a new user has joined.

[0105] S14: After the two members in the call group complete the media negotiation, they try to establish a corresponding first end-to-end connection. If the first end-to-end connection fails to be established, the two members obtain the public network IP and port through the STUN service of the corresponding ICE server and try to establish a corresponding second end-to-end connection. If the second end-to-end connection fails to be established, a relay forwarding connection is established through the TURN service of the corresponding ICE server. If any of the above connections is successfully established, the two members start an audio or video call; until the connection between any two members in the call group is successfully established;

[0106] Specifically, the process of completing media negotiation and establishing a call connection is as follows:

[0107] ① User A receives the new user joining signaling, parses the signaling data, and adds the new user to the call user list. If user A has already created an offer, he will not create a new offer. If not, he will create an offer based on the local terminal device information and send it to the signaling server;

[0108] ② The signaling server forwards the offer to other members of the talk group. After receiving the offer, other members make corresponding settings and create answers, which are then forwarded to other users in the talk group via the signaling server.

[0109] ③Other users in the call group receive the answer and make corresponding settings;

[0110] ④ When the two members in the call group complete the corresponding settings, they start trying to establish an end-to-end connection. If the end-to-end connection fails to be established, the public IP and port are obtained through the STUN service of the corresponding ICE server, and an end-to-end connection is tried again. If the connection still fails to be established, a relay forwarding connection is established through the TURN service of the corresponding ICE server. Otherwise, if any of the above connections is successfully established, the two members start an audio and video call, and at the same time start the call quality reporting task and transmit it to the signaling server.

[0111] It should be noted that call quality reporting is when two members start an audio or video call, the data streams of the two members' terminals are obtained, and the call quality is obtained from the data stream according to the preset reporting interval and then reported to the signaling server; among them, the call quality includes: audio sending bit rate, video sending bit rate, audio receiving bit rate, video receiving bit rate, video sending width, video sending height, video actual sending frame rate and video actual receiving frame rate.

[0112] Preferably, the preset reporting interval is 5 seconds.

[0113] The signaling server records the quality of received calls and combines it with the ICE server log to analyze network conditions, server resource utilization, and call duration. It can also be used as basic data to facilitate subsequent reasonable call charges.

[0114] S15: The signaling server receives the leave signaling, removes the member from the talk group, and notifies other members. If the number of talk group members is 0, the talk group is destroyed and the ICE server is unbound.

[0115] Specifically, the process of hanging up a call is as follows:

[0116] ① Any member of the talk group hangs up the call and sends a leave signaling "leave" to the signaling server. The signaling server receives the leave signaling, removes the member from the talk group, and sends a leave notification signaling "relayLeave" to notify other members;

[0117] ②Other users in the call group receive the departure notification signal, close the connection channel with the departing user, remove the user from the call user list, and clear related configurations;

[0118] ③ Repeat the process of new members joining or leaving until the number of members in the talk group is 0, that is, all members leave and the call is closed. Then the talk group is destroyed and the ICE server is unbound.

[0119] It should be noted that when the connection channel with the leaving user is closed, the remaining bandwidth of the ICE server corresponding to the leaving member is increased according to the call type of the current call group, and the weight is updated. As the number of calling users decreases, the corresponding resources are recovered and the probability of the ICE server being selected is increased.

[0120] It is worth noting that, in this embodiment, the signaling server can manage each talk group and the members in each talk group during a call, detect the terminal heartbeat, collect the terminal call quality, and manage the online and offline of the registered ICE server.

[0121] Specifically, the ICE server uses a Linux server. Each server is set up with a scheduled task. It regularly reads the proc / net / dev file in the Linux server to obtain network information such as ICE server bandwidth usage and reports it to the signaling server. After receiving it, the signaling server records it in the log, including: current time, ICE server IP, real-time downlink network load and real-time uplink network load.

[0122] By analyzing the logs, other services of the server can be reduced during peak usage, corresponding servers can be added, or the ICE server can be set to a state of preparing to go offline to reduce server investment. At this time, the ICE server will no longer be allocated to new users until there are no users using it or the maximum recycling time is exceeded. The ICE server is marked offline and an email notification is sent.

[0123] For example, the analysis service is shut down during peak usage, and the start time of the scheduled service is adjusted to a period of less usage or no usage, so as to more reasonably allocate and utilize server resources.

[0124] Compared with the prior art, the present embodiment provides a WebRtc-based audio and video call method, which realizes audio and video calls between multiple terminals including a PC web page terminal, a PC application terminal, and a mobile terminal, and designs a set of signaling rules for the three terminals with millisecond-level timestamps for easy monitoring and processing; the ICE server has both STUN and TURN functions, and performs relay forwarding when an end-to-end connection cannot be established; a layer of signaling service is established before the ICE server to manage each group of audio and video calls and ICE servers, and by locating the user terminal IP, the nearest server is preferentially selected for users making audio and video calls, thereby improving the response speed; by dynamically updating the remaining traffic and weight of the ICE server, the server with the smallest load is selected for the user to avoid delays and freezes; at the same time, each ICE server is dynamically monitored for bandwidth and traffic, and through log analysis, it supports dynamic setting of the ICE server's online and offline, so as to make full and reasonable use of resources and avoid waste.

[0125] Embodiment 2,

[0126] Another embodiment of the present invention discloses an audio and video call system based on WebRtc, so as to implement the audio and video call method in Embodiment 1. The specific implementation of each module refers to the corresponding description in Embodiment 1, and the system includes:

[0127] The ICE server is used to receive user terminal requests, obtain the public IP and port of the user terminal through the STUN service; receive and forward user terminal data streams through the TURN service; and regularly report the bandwidth status of the ICE server to the signaling server;

[0128] Regular reporting of ICE server bandwidth conditions includes: each ICE server periodically reads the proc / net / dev file in the server to obtain network information such as ICE server bandwidth usage, and reports it to the signaling server. After receiving it, the signaling server records it in the log, including: current time, ICE server IP, real-time downlink network load, and real-time uplink network load.

[0129] The signaling server is used to manage the ICE server, and receive and parse the first call joining signaling sent by the calling user terminal, create a call group, add the calling user as a new member to the call group, and notify the selected user; the signaling server receives and parses the second call joining signaling sent by the selected user terminal, and adds the selected user as a new member to the call group; wherein, when a new member is added to the call group, the new member terminal IP is obtained, and according to the original member terminal IP, the weight and load of the ICE server in the call group, the ICE server between the new member and each other member in the call group is obtained, the call group is bound, and a join response signaling is sent to notify the new member;

[0130] The terminal call module is used for users to initiate calls, join calls and audio and video calls, including: checking whether the calling / selected user terminal has the corresponding hardware device and whether the hardware device is available according to the call type, and obtaining the terminal data stream for the available device; sending a first join call signaling to the calling user terminal, and sending a second join call signaling to the selected user terminal, and receiving a join response signaling; when two members in the call group complete media negotiation, try to establish a corresponding first end-to-end connection. If the first end-to-end connection fails to be established, the two members obtain the public network IP and port through the STUN service of the corresponding ICE server, and try to establish a corresponding second end-to-end connection. If the second end-to-end connection fails to be established, a relay forwarding connection is established through the TURN service of the corresponding ICE server. If any of the above connections is successfully established, the two members start an audio and video call; until the connection between any two members in the call group is successfully established.

[0131] It also includes: a terminal heartbeat monitoring module, which is used to start the heartbeat monitoring timing task, send a heartbeat signaling to the signaling server according to the preset monitoring interval; and receive and count the heartbeat response signaling sent by the signaling server. If the heartbeat response signaling is not received for more than a preset number of consecutive times, the heartbeat signaling is stopped, prompting the call member to disconnect or the network is faulty;

[0132] The call quality reporting module is used to set the call quality reporting status of the new member's terminal to on when the new member sends a signal to join the call; and after two members start an audio or video call, the data streams of the two member terminals are obtained, and the call quality is obtained from the data stream according to the preset reporting interval and reported to the signaling server; the call quality includes: audio sending bit rate, video sending bit rate, audio receiving bit rate, video receiving bit rate, video sending width, video sending height, video actual sending frame rate and video actual receiving frame rate.

[0133] Preferably, it also includes a log analysis module for analyzing logs, predicting server resource distribution, and dynamically setting the online and offline of ICE servers.

[0134] Specifically, by integrating and analyzing the monitoring logs and the information of the ICE server and its bound user groups, we can obtain the following: time, number of call users, call user addresses, ICE server addresses, ICE server types, remaining bandwidth of the ICE server, and ICE uplink and downlink traffic. Based on these data, we calculate the bandwidth required by each server at each time, the server address ratio, and the call user address ratio, and establish a resource usage distribution model based on time and the required bandwidth as the target.

[0135] Using historical data as samples, we train and learn the resource usage distribution model, predict the bandwidth required in the future, and predict the number of users and required ICE servers in each region through the proportion of user addresses and server addresses. According to the forecast, when the required resources decrease, servers in areas with fewer users will be taken offline or their allocation weights will be reduced in advance to provide resources for other services / programs; when the required service resources increase, servers with fewer users will be put online or their weights will be increased to balance resource allocation and avoid the situation where some servers are fully loaded and some are idle.

[0136] Compared with the prior art, the present embodiment provides a WebRtc-based audio and video call system, which manages each group of audio and video calls and ICE servers through a signaling server, and preferentially selects the nearest server for users making audio and video calls by locating the IP of the user terminal, thereby improving the response speed; by dynamically updating the remaining traffic and weight of the ICE server, the server with the smallest load is selected for the user to avoid delays and freezes; through a variety of scheduled tasks, the terminal and the ICE server are monitored, resources are recovered in time, and dynamic updates of the remaining bandwidth and weight of the ICE server are supported, and the online and offline of the ICE server are dynamically set, so as to reasonably use resources and avoid waste.

[0137] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0138] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A WebRtc-based audio and video call method, characterized in that: The steps include: The user logs into the system, checks the terminal hardware device according to the selected call type, selects the call user, initiates the call as the calling user, and sends the first join call signaling to the signaling server; The signaling server receives and parses the first call joining signaling sent by the calling user terminal, creates a talk group, adds the calling user as a new member to the talk group, selects an ICE server according to the calling user terminal IP, the weight and load of the ICE server, and notifies the selected user; The selected user receives the new connection signaling, checks the terminal hardware device, and sends a second join call signaling to the signaling server; The signaling server receives and parses the second join call signaling sent by the selected user terminal, and adds the selected user as a new member to the talk group; wherein, when the new member is added to the talk group, the new member terminal IP is obtained, and the ICE server between the new member and each other member in the talk group is obtained according to the original member terminal IP, the weight and load of the ICE server in the talk group, and the talk group is bound and a join response signaling is sent to notify the new member; When two members in the call group complete media negotiation, they try to establish a corresponding first end-to-end connection. If the first end-to-end connection fails to be established, the two members obtain the public network IP and port through the STUN service of the corresponding ICE server and try to establish a corresponding second end-to-end connection. If the second end-to-end connection fails to be established, a relay forwarding connection is established through the TURN service of the corresponding ICE server. If any of the above connections is successfully established, the two members start an audio or video call; until the connection between any two members in the call group is successfully established; By integrating and analyzing the monitoring logs and the information of the ICE server and its bound user groups, we can obtain the following: time, number of call users, call user addresses, ICE server addresses, ICE server types, remaining bandwidth of the ICE server, and ICE uplink and downlink traffic. Based on these data, we can calculate the bandwidth required by each server at each time, the server address ratio, and the call user address ratio, and establish a resource usage distribution model based on time and the required bandwidth as the target. Use historical data as samples to train and learn the resource usage distribution model, predict the bandwidth required in the future, and predict the number of users and the required ICE servers in each region through the proportion of user addresses and server addresses. According to the prediction, when the required resources decrease, the servers in the area with fewer users will be taken offline or their allocation weight will be reduced in advance to provide resources for other services / programs. When the required service resources increase, the servers with fewer users will be put online or their weight will be increased to balance resource allocation and avoid the situation where some servers are fully loaded and some are idle.

2. The WebRtc-based audio and video call method according to claim 1, characterized in that: The method also includes: the new member receives the joining response signaling sent by the signaling server, starts the heartbeat monitoring timing task, and sends the heartbeat signaling to the signaling server according to the preset monitoring interval; and receives and counts the heartbeat response signaling sent by the signaling server. If the heartbeat response signaling is not received for more than a preset number of consecutive times, the heartbeat signaling is stopped, prompting the call member to be disconnected or the network fails.

3. The WebRtc-based audio and video call method according to claim 1, characterized in that: The method further includes: when the new member sends a call joining signaling, setting the call quality reporting state of the new member terminal to be turned on; and After the two members start the audio and video call, the data streams of the two member terminals are obtained, and the call quality is obtained from the data stream according to the preset reporting interval and then reported to the signaling server.

4. The WebRtc-based audio and video call method according to claim 1, characterized in that: The method further includes: before the calling user terminal sends a first call joining signaling, and before the selected user terminal sends a second call joining signaling, checking whether the corresponding calling / selected user terminal has corresponding hardware devices according to the call type, if it is detected that the hardware device is missing, prompting an error and ending the call; otherwise, further checking whether the hardware device is available, including: Activate the camera and microphone of the user terminal; The H264 encoding format is used to obtain the audio stream and video stream of the user terminal according to the initialized video parameters. If the acquisition fails, the X264 encoding format is adjusted to obtain again. If the acquisition still fails, the corresponding abnormal information is prompted and the call ends; if the acquisition is successful, the video stream is rendered to the call interface.

5. A WebRtc-based audio and video call system, characterized in that: include: ICE server, used to receive user terminal requests and obtain the public IP and port of the user terminal through the STUN service; Receive and forward user terminal data streams through TURN service; And, regularly report the bandwidth status of the ICE server to the signaling server; A signaling server, configured to manage an ICE server, and to receive and parse a first join call signaling sent by a calling user terminal, create a talk group, add the calling user as a new member to the talk group, and notify the selected user; The signaling server receives and parses the second join call signaling sent by the selected user terminal, and adds the selected user as a new member to the talk group; wherein, when the new member is added to the talk group, the new member terminal IP is obtained, and the ICE server between the new member and each other member in the talk group is obtained according to the original member terminal IP, the weight and load of the ICE server in the talk group, and the talk group is bound and a join response signaling is sent to notify the new member; The terminal call module is used for users to initiate calls, join calls and audio and video calls, including: checking whether the calling / selected user terminal has the corresponding hardware device according to the call type, whether the hardware device is available, and obtaining the terminal data stream for the available device; sending a first join call signaling to the calling user terminal, and sending a second join call signaling to the selected user terminal, and receiving a join response signaling; when two members in the call group complete media negotiation, try to establish a corresponding first end-to-end connection. If the first end-to-end connection fails to be established, the two members obtain the public network IP and port through the STUN service of the corresponding ICE server, and try to establish a corresponding second end-to-end connection. If the second end-to-end connection fails to be established, a relay forwarding connection is established through the TURN service of the corresponding ICE server. If any of the above connections is successfully established, the two members start an audio and video call; until the connection between any two members in the call group is successfully established; The log analysis module is used to analyze logs, predict the distribution of server resources, and dynamically set the online and offline of ICE servers, including: integrating and analyzing the monitoring logs and the information of the ICE server and its bound user groups to obtain: time, number of call users, call user addresses, ICE server addresses, ICE server types, remaining bandwidth of ICE servers, and ICE uplink and downlink traffic; based on these data, the bandwidth required for each server at each time, the server address ratio and the call user address ratio are calculated, and a resource usage distribution model based on time and required bandwidth as the target is established; using historical data as samples, the resource usage distribution model is trained and learned, and the bandwidth required for future time is predicted, and the number of users and required ICE servers in each region are predicted by the user address and server address ratio; according to the predicted situation, when the required resources decrease, the servers in the area with fewer users are taken offline or the allocation weight is reduced in advance to provide resources for other services / programs; when the required service resources increase, the servers with fewer users are put online or the weight is increased to balance resource allocation and avoid the situation where some servers are fully loaded and some are idle.

6. The WebRtc-based audio and video call system according to claim 5, characterized in that: Also includes: The terminal heartbeat monitoring module is used to start the heartbeat monitoring timing task and send the heartbeat signaling to the signaling server according to the preset monitoring interval; And, receiving and counting the heartbeat response signaling sent by the signaling server, if the heartbeat response signaling is not received for more than a preset number of consecutive times, the heartbeat signaling is stopped, prompting the call member to disconnect or the network fails; The call quality reporting module is used to set the call quality reporting status of the new member terminal to on when the new member sends a signal to join the call; and after the two members start an audio and video call, obtain the data streams of the two member terminals, and obtain the call quality from the data stream according to the preset reporting interval and report it to the signaling server.

Images