SIP-based audio and video conferencing methods and systems

Abstract

This invention relates to the field of audio and video communication technology, and discloses a SIP-based audio and video conferencing method and system. The method includes: a conference initiating terminal creating a conference room through a host server and initiating a meeting invitation to a target audio and video conferencing terminal; the host server establishing a SIP channel with the target audio and video conferencing terminal via the communication topology based on the meeting invitation through static or dynamic routing; and optimizing the transmission of uplink and / or downlink audio and video data between the host server and the target audio and video conferencing terminal during the conference based on the audio and video MSG flow control protocol. Implementing this invention, through audio and video merging and flow control based on the MSG flow control protocol, bandwidth can be reduced by approximately 60% to 70%, reducing customer network bandwidth costs.

Description

Technical Field

[0001] This invention relates to the field of audio and video communication technology, and specifically to a SIP-based audio and video conferencing method and system. Background Technology

[0002] In existing SIP-based audio and video conferencing cascades, multi-level server cascading results in high audio and video transmission bandwidth, data redundancy, poor user experience, and an inability to meet the needs of a large number of simultaneous online users. Furthermore, it suffers from poor compatibility, requiring system redevelopment for some audio and video conferencing terminals, significantly increasing development costs. Summary of the Invention

[0003] To address the aforementioned shortcomings, this invention discloses a SIP-based audio and video conferencing method and system that can reduce customer network bandwidth costs and improve user experience.

[0004] The first aspect of this invention discloses a SIP-based audio and video conferencing method, which is implemented based on an IP network conferencing system. The IP network conferencing system includes: a master server, one or more slave servers, and audio and video conferencing terminals. The master server and one or more slave servers construct a communication topology, and the audio and video conferencing terminals are connected to the communication topology through corresponding slave servers.

[0005] The method includes:

[0006] The meeting initiating terminal creates a meeting room through the main server and sends a meeting invitation to the target audio and video conferencing terminal;

[0007] The main server establishes a SIP channel with the target audio and video conferencing terminal through the communication topology based on the membership invitation via static or dynamic routing.

[0008] During the meeting, the transmission of uplink and / or downlink audio and video data between the host server and the target audio and video conferencing terminal is optimized based on the audio and video MSG flow control protocol.

[0009] As an optional implementation, in a first aspect of the present invention, the meeting initiating terminal creates a meeting room through the main server, including:

[0010] The conference initiating terminal sends a first SIP invite request to the main server to create a conference room and establish a two-way audio and video transmission channel between the conference initiating terminal and the main server.

[0011] As an optional implementation, in a first aspect of the present invention, the main server establishes a SIP channel with the target audio / video conferencing terminal via the communication topology based on the membership invitation through static or dynamic routing, including:

[0012] The master server determines the transmission path through a static or dynamic routing table, and initiates a second SIP invite request to the target audio and video conferencing terminal according to the transmission path to establish a SIP channel between the master server and the target audio and video conferencing terminal.

[0013] Each slave server along the transmission path establishes a virtual conference room to distribute the second SIP invite request.

[0014] As an optional implementation, in the first aspect of the present invention, the master server determines the transmission path through a static routing table or a dynamic routing table, and initiates a second SIP invite request to the target audio / video conferencing terminal according to the transmission path to establish a SIP channel between the master server and the target audio / video conferencing terminal, including:

[0015] The master server queries the static or dynamic routing table to determine the transmission path between the master server and the target audio and video conferencing terminal.

[0016] The DataRoute information is determined based on the transmission path, and the DataRoute information is assembled into the second SIP invite request;

[0017] The preceding target server in the transmission path forwards the second SIP invite request to the following target server based on the DataRoute information, establishing a SIP channel between the two. When the preceding target server forwards the second SIP invite request, it deletes the DataRoute information corresponding to the following target server in the second SIP invite request.

[0018] If the second SIP invite request received by the target server in the transmission path does not carry DataRoute information, it is designated as an associated slave server. Then, the second SIP invite request is forwarded to the target audio and video conferencing terminal through the virtual conference room created by the associated slave server, thereby establishing a SIP channel between the associated slave server and the target audio and video conferencing terminal.

[0019] As an optional implementation, in the first aspect of the present invention, the main server queries a dynamic routing table to determine the transmission path between the main server and the target audio / video conferencing terminal, including:

[0020] In the communication topology, any two adjacent slave servers, as well as the master server and slave servers, probe each other's data, acquire probe data, and synchronize the probe data to all adjacent master servers or slave servers. Alternatively, all slave servers can send their own probe data to the master server through the transmission path between them, so that the master server can acquire the probe data from all slave servers. The probe data includes one or more of the following: latency, packet loss rate, validity period, and weight transmitted between adjacent slave servers, and between the master server and slave servers.

[0021] The master server determines all transmission paths to the target audio and video conferencing terminal, and determines the optimal path among all transmission paths based on the detection data, which is used as the transmission path between the master server and the target audio and video conferencing terminal and is denoted as the target path.

[0022] As an optional implementation, in the first aspect of the present invention, the transmission of uplink and / or downlink audio and video data between the host server and the target audio and video conferencing terminal is optimized based on the audio and video MSG flow control protocol, including:

[0023] Optimizations for uplink audio data transmission include: transmitting only the audio data of the target audio / video conferencing terminal speaking to the main server via the corresponding SIP channel;

[0024] Optimization of uplink video data transmission includes: transmitting video data from target audio and video conferencing terminals with display markers to the main server only through the corresponding SIP channel; or, the main server sends instructions to some target audio and video conferencing terminals not to upload video data, and the video data of target audio and video conferencing terminals that do not receive the instructions to not upload video data is transmitted to the main server through the corresponding SIP channel.

[0025] The optimization of downlink audio data transmission includes: the main server distributes the mixed audio stream to all muted target audio and video conferencing terminals at once; the main server sends the optimized mixed audio stream to the speaking target audio and video conferencing terminal through the corresponding SIP channel; the optimized mixed audio stream removes the audio data uploaded by the speaking target audio and video conferencing terminal.

[0026] Optimizations for downlink video data transmission include: the main server distributing the merged video feed to all target audio and video conferencing terminals at once.

[0027] The second aspect of this invention discloses a SIP-based audio and video conferencing system, which includes: a master server, one or more slave servers, a conference initiating terminal, and an audio and video conferencing terminal. The master server and one or more slave servers construct a communication topology. The audio and video conferencing terminal is connected to the communication topology through a corresponding slave server. The conference initiating terminal communicates directly with the master server.

[0028] The meeting initiating terminal creates a meeting room through the main server and sends a meeting invitation to the target audio and video conferencing terminal.

[0029] The main server establishes a SIP channel with the target audio and video conferencing terminal through the communication topology based on the membership invitation via static or dynamic routing.

[0030] During the meeting, the main server optimizes the transmission of uplink and / or downlink audio and video data between the main server and the target audio and video conferencing terminal based on the audio and video MSG flow control protocol.

[0031] As an optional implementation, in a second aspect of the present invention, the meeting initiating terminal sends a first SIP invite request to the main server to create a meeting room and establish a bidirectional audio and video transmission channel between the meeting initiating terminal and the main server;

[0032] The meeting initiating terminal sends a meeting invitation to the target audio and video conferencing terminal to the main server via API.

[0033] The master server determines the transmission path through a static or dynamic routing table, and initiates a second SIP invite request to the target audio and video conferencing terminal according to the transmission path to establish a SIP channel between the master server and the target audio and video conferencing terminal.

[0034] Each slave server along the transmission path establishes a virtual conference room to distribute the second SIP invite request.

[0035] As an optional implementation, in a second aspect of the present invention, the master server queries a static routing table or a dynamic routing table to determine the transmission path between the master server and the target audio / video conferencing terminal; the master server determines DataRoute information based on the transmission path and assembles the DataRoute information into the second SIP invite request;

[0036] The preceding target server in the transmission path forwards the second SIP invite request to the following target server based on the DataRoute information, establishing a SIP channel between the two. When the preceding target server forwards the second SIP invite request, it deletes the DataRoute information corresponding to the following target server in the second SIP invite request.

[0037] If the second SIP invite request received by the target server in the transmission path does not carry DataRoute information, it is designated as an associated slave server. Then, the second SIP invite request is forwarded to the target audio and video conferencing terminal through the virtual conference room created by the associated slave server, thereby establishing a SIP channel between the associated slave server and the target audio and video conferencing terminal.

[0038] As an optional implementation, in the second aspect of the present invention, the optimization of uplink audio data transmission includes: transmitting only the audio data of the target audio / video conferencing terminal speaking to the main server through the corresponding SIP channel;

[0039] Optimization of uplink video data transmission includes: transmitting video data from target audio and video conferencing terminals with display markers to the main server only through the corresponding SIP channel; or, the main server sends instructions to some target audio and video conferencing terminals not to upload video data, and the video data of target audio and video conferencing terminals that do not receive the instructions to not upload video data is transmitted to the main server through the corresponding SIP channel.

[0040] The optimization of downlink audio data transmission includes: the main server distributes the mixed audio stream to all muted target audio and video conferencing terminals at once; the main server sends the optimized mixed audio stream to the speaking target audio and video conferencing terminal through the corresponding SIP channel; the optimized mixed audio stream removes the audio data uploaded by the speaking target audio and video conferencing terminal.

[0041] Optimizations for downlink video data transmission include: the main server distributing the merged video feed to all target audio and video conferencing terminals at once.

[0042] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0043] 1. By using audio and video merging and flow control based on the MSG flow control protocol, bandwidth can be reduced by approximately 60% to 70%, thereby reducing customers' network bandwidth costs.

[0044] 2. Conference cascading: Audio and video conferencing terminals can be pulled into the conference through a relay server. The preceding slave server relays signaling and audio / video, which can solve problems such as network incompatibility between the master server and the subsequent slave server.

[0045] 3. Each slave server can form a separate network with its subordinate servers without interfering with each other. If the service of some servers is unavailable, it will not affect the use of devices on other servers.

[0046] 4. Optimize audio and video uplink and downlink bandwidth. Reduce the transmission of some audio and video data streams according to the flow control strategy. This can significantly increase the number of online participants in the meeting when network bandwidth is insufficient.

[0047] 5. Strong compatibility: It is compatible with various audio and video conferencing terminals. It can reduce bandwidth without modifying the audio and video conferencing terminals, thereby reducing the development costs of adapting the audio and video conferencing terminals and effectively reducing the customer's network bandwidth costs. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating a SIP-based audio and video conferencing method disclosed in an embodiment of the present invention;

[0050] Figure 2 This is the architecture of the IP network conferencing system disclosed in the embodiments of the present invention. Figure 1 ;

[0051] Figure 3 This is the architecture of the IP network conferencing system disclosed in the embodiments of the present invention. Figure 2 ;

[0052] Figure 4 This is a schematic diagram of the process for establishing a multi-level cascading conference as disclosed in an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of the DataRoute routing process disclosed in an embodiment of the present invention;

[0054] Figure 6 This is a flowchart illustrating the audio control instructions disclosed in an embodiment of the present invention;

[0055] Figure 7 This is a flowchart illustrating the video control instructions disclosed in an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] It should be noted that the terms "first," "second," "third," "fourth," etc., in the specification and claims of this invention are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "having," and any variations thereof, in the embodiments of this invention are intended to cover non-exclusive inclusion. Exemplarily, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0058] This invention discloses a SIP-based audio and video conferencing method and system, which effectively improves system performance and stability, offers high cost-effectiveness, and is easy to use. It provides high-quality audio and video conferencing communication functions for industries such as banking and business meetings; enhances server concurrency capabilities under the same network environment, improves audio and video transmission quality, and reduces network bandwidth costs and improves communication experience for customers. The solution is mainly based on the SIP protocol, extended from it, and uses the open-source FreeSWITCH softphone switching platform as a carrier. Flow control protocols are added between servers to achieve combined audio and video data forwarding, which can significantly increase the number of online participants required for the meeting. It has strong device compatibility and effectively reduces development costs, while also effectively reducing customer network bandwidth costs. A detailed description is provided below with reference to the accompanying drawings.

[0059] Example 1

[0060] Embodiment 1 of this invention discloses a SIP-based audio and video conferencing method. This method is implemented based on an IP network conferencing system architecture.

[0061] An IP network conferencing system mainly includes: a master server, one or more slave servers, a conference initiating terminal, and audio and video conferencing terminals. The master server and one or more slave servers construct a communication topology. The audio and video conferencing terminals are connected to the communication topology through corresponding slave servers. The conference initiating terminal communicates directly with the master server. It can be understood that any audio and video conferencing terminal can become a conference initiating terminal.

[0062] The communication topology can be arbitrarily constructed according to the needs of different scenarios. For example, it can adopt... Figure 2 The tree structure shown can also be used Figure 3 The mesh structure shown uses Figure 2 In a tree-like structure, due to the clear hierarchical relationship, the master server can be designated as a first-level server, and the slave servers that communicate directly with the first-level server as second-level servers, and so on. A mesh structure does not have a clear hierarchical relationship; however, when the transmission path is determined, it can still be classified according to this level. For example... Figure 3 In this context, when the master server establishes a SIP channel with the audio / video conferencing terminal 300101 through slave servers 2002, 2001, and 3001 in sequence, the master server, slave server 2002, slave server 2001, and slave server 3001 can be referred to as a level 1 server, a level 2 server, a level 3 server, and a level 4 server, respectively.

[0063] In some scenarios, any slave server can also become the master server, for example... Figure 2 When the audio-visual conferencing terminal 200201 creates a meeting room through the server 2002, the audio-visual conferencing terminal 200201 becomes the meeting initiating terminal, the server 2002 becomes the master server, and the master server 1001 becomes the slave server. The meeting initiating terminal that communicates with the master server 1001 becomes the audio-visual conferencing terminal.

[0064] Both the audio / video conferencing terminal and the conferencing initiation terminal are SIP terminals to adapt to the SIP protocol.

[0065] Please refer to Figure 1 As shown, a SIP-based audio and video conferencing method includes the following steps:

[0066] S110, the conference initiating terminal creates a conference room through the main server and sends a meeting invitation to the target audio and video conferencing terminal.

[0067] The meeting initiating terminal sends a meeting invitation to the host server via API, inviting the target audio / video conferencing terminals to join the meeting. The meeting initiating terminal sends a first SIP invite request to the host server, creates a meeting room, and establishes a bidirectional audio / video transmission channel between the meeting initiating terminal and the host server.

[0068] by Figure 2 Taking the structure as an example, when the meeting initiating terminal 100101 sends an invitation to the audio / video conferencing terminal 300101 to join the meeting, please refer to... Figure 4As shown in steps 1.1-1.4, the conference initiating terminal 100101 sends the first SIPinvite request to the main server 1001 to create a conference room and establish a bidirectional audio and video transmission channel (SIP channel) between the conference initiating terminal and the main server. The conference room can realize audio mixing, video fusion, and audio and video data distribution.

[0069] S120, the master server establishes a SIP channel with the target audio and video conferencing terminal via the communication topology through static or dynamic routing.

[0070] The master server determines the transmission path through a static or dynamic routing table, and initiates a second SIP invite request to the target audio and video conferencing terminal according to the transmission path to establish a SIP channel between the master server and the target audio and video conferencing terminal.

[0071] Specifically, it includes:

[0072] The master server queries the static or dynamic routing table to determine the transmission path between the master server and the target audio / video conferencing terminal.

[0073] The DataRoute information is determined based on the transmission path, and the DataRoute information is assembled into the second SIP invite request;

[0074] The preceding target server in the transmission path forwards the second SIP invite request to the following target server based on the DataRoute information, establishing a SIP channel between the two. When the preceding target server forwards the second SIP invite request, it deletes the DataRoute information corresponding to the following target server in the second SIP invite request.

[0075] If the second SIP invite request received by the target server in the transmission path does not carry DataRoute information, it is designated as an associated slave server. Then, the second SIP invite request is forwarded to the target audio and video conferencing terminal through the virtual conference room created by the associated slave server, thereby establishing a SIP channel between the associated slave server and the target audio and video conferencing terminal.

[0076] The conference initiating terminal sets up a conference room on the master server via SIP. Then, it invites audio and video conferencing terminals to join the conference via API commands. Invitations to audio and video conferencing terminals on subsequent slave servers must be forwarded through the preceding slave servers. When inviting audio and video conferencing terminals on subsequent slave servers, the next-hop slave server can be specified by adding a DataRoute header (where DataRoute is a list of server IDs) to the second SIP invite request. After each hop through a slave server, the current server ID is removed from the DataRoute, and the call proceeds to the next hop.

[0077] Please refer to Figure 5 As shown, when the conference initiating terminal 100101 sends an invitation to join the audio / video conferencing terminal 400101 under the fourth-level server 4001 through the main server 1001, assuming the transmission path is the main server 1001, the second-level server 2001, the third-level server 3001, and the fourth-level server 4001, then:

[0078] When the primary server 1001 sends the second SIP invite request to the secondary server 2001, it adds a DataRoute header carrying 3001 and 4001, as shown in the SIP message:

[0079] INVITE sip:100000001@10.1.8.201:5080SIP / 2.0

[0080] Via:SIP / 2.0 / UDP 10.1.8.200;rport;branch=z9hG4bKFrmU1Q85Q1arr

[0081] Route:<sip:100000001@10.1.8.201:5080>

[0082] Max-Forwards: 69

[0083] From:""<sip:100000001@10.1.8.200> ;tag=U54c6UQe790vm

[0084] To:<sip:external@10.1.8.201:5080>

[0085] Call-ID:ab119bb1-f3a2-123a-6193-29407c05220f

[0086] CSeq:46736243INVITE

[0087] Contact:<sip:mod_sofia@10.1.8.200:5080>

[0088] User-Agent:CoreServer

[0089] Allow:INVITE,ACK,BYE,CANCEL,OPTIONS,MESSAGE,INFO,UPDATE,REGISTER,REFER,NOTIFY,PUBLISH,SUBSCRIBE

[0090] Supported: timer, path, replaces

[0091] DataRoute:3001,4001

[0092] Secondary server 2001 forwards the second SIP invite request to tertiary server 3001 based on the first information (3001) in the DataRoute header. In the forwarded second SIP invite request, the 3001 information in the DataRoute header is removed. The reference SIP message is as follows:

[0093] INVITE sip:100000001@10.1.8.202:5080SIP / 2.0

[0094] Via:SIP / 2.0 / UDP 10.1.8.201;rport;branch=z9hG4bKFrmU1Q85Q1arr

[0095] Route:<sip:100000001@10.1.8.202:5080>

[0096] Max-Forwards: 69

[0097] From:""<sip:100000001@10.1.8.201> ;tag=U54c6UQe790vm

[0098] To:<sip:external@10.1.8.202:5080>

[0099] Call-ID:ab119bb1-f3a2-123a-6193-29407c05220f

[0100] CSeq:46736243INVITE

[0101] Contact:<sip:mod_sofia@10.1.8.201:5080>

[0102] User-Agent:CoreServer

[0103] Allow:INVITE,ACK,BYE,CANCEL,OPTIONS,MESSAGE,INFO,UPDATE,REGISTER,REFER,NOTIFY,PUBLISH,SUBSCRIBE

[0104] Supported: timer, path, replaces

[0105] DataRoute:4001

[0106] Therefore, the second SIP invite request received by Level 3 server 3001 only carries the 4001 information in the DataRoute header. Similarly, Level 3 server 3001 forwards the second SIP invite request to Level 4 server 4001 based on the first piece of information carried in the DataRoute header, namely 4001. In the forwarded second SIP invite request, the 4001 information carried in the DataRoute header is removed. The reference SIP message is as follows:

[0107] INVITE sip:100000001@10.1.8.203:5080SIP / 2.0

[0108] Via:SIP / 2.0 / UDP 10.1.8.202;rport;branch=z9hG4bKFrmU1Q85Q1arr

[0109] Route:<sip:100000001@10.1.8.203:5080>

[0110] Max-Forwards: 69

[0111] From:""<sip:100000001@10.1.8.202> ;tag=U54c6UQe790vm

[0112] To:<sip:external@10.1.8.203:5080>

[0113] Call-ID:ab119bb1-f3a2-123a-6193-29407c05220f

[0114] CSeq:46736243INVITE

[0115] Contact:<sip:mod_sofia@10.1.8.202:5080>

[0116] User-Agent:CoreServer

[0117] Allow:INVITE,ACK,BYE,CANCEL,OPTIONS,MESSAGE,INFO,UPDATE,REGISTER,REFER,NOTIFY,PUBLISH,SUBSCRIBE

[0118] Supported: timer, path, replaces

[0119] The second SIP invite request received by the level 4 server 4001 has no DataRoute header, indicating that the second SIP invite request should be sent to the audio / video conferencing terminal below it, namely audio / video conferencing terminal 400101.

[0120] As an example, still using Figure 2 Taking the structure as an example, when the conference initiating terminal 100101 sends an invitation to the audio / video conferencing terminal 300101, the transmission path is determined through the static routing table: primary server 1001, secondary server 2001, and tertiary server 3001. Please refer to... Figure 4 As shown, the specific process is as follows:

[0121] First, the primary server 1001 sends a second SIP invite request to the secondary server 2001, establishing a bidirectional audio and video transmission channel between the primary server 1001 and the secondary server 2001. Figure 4 Steps 2.1, 2.4, and 2.6.

[0122] Then, after receiving the second SIP invite request, the secondary server 2001 will create a virtual conference room on its local server (the virtual conference room mainly implements the copying and distribution of audio and video streams). It will then establish SIP / RTP communication with the tertiary server 3001. For example... Figure 4 Steps 3.1, 3.6, and 3.9.

[0123] When secondary server 2001 receives the second SIP invite request with DataRoute information, it will forward the second SIP invite request to tertiary server 3001 based on the server number carried in the DataRoute field of the SIPinvite header. For example... Figure 4Steps 3.2, 3.5, and 3.8.

[0124] Assuming a secondary server 2001 hosts an audio / video conferencing terminal 200101, when the conference initiating terminal 100101 sends an invitation to audio / video conferencing terminal 200101, if the second SIP invite request received by secondary server 2001 does not include DataRoute information, it indicates that audio / video conferencing terminal 200101 is an audio / video conferencing terminal under secondary server 2001. Secondary server 2001 will then initiate a SIP invite request to audio / video conferencing terminal 200101 through a virtual meeting room to establish a session, such as... Figure 4 Steps 2.2, 2.3, and 2.5.

[0125] Finally, after receiving the second SIP invite request forwarded by the second-level server 2001, the third-level server 3001 will establish a virtual conference room on its local server (the virtual conference room mainly implements the replication and distribution of audio and video streams). It will then establish SIP / RTP communication with the audio / video conferencing terminal 300101. For example... Figure 2 Steps 3.3, 3.4, and 3.7.

[0126] When inviting other servers and other target audio / video conferencing terminals to join the meeting, the above steps can be used as a reference. For Figure 3 The invitation process for the mesh topology is similar and will not be described in detail here.

[0127] Understandably, since the main server's conference room and all invited audio and video conferencing terminals are connected via separate SIP channels, bidirectional audio and video transmission and reception are possible.

[0128] For static routing tables, configuration is only required on the main server, making implementation simple. Network connectivity needs to be configured beforehand by maintenance personnel. For example, configuration can be performed using the following procedure:

[0129]

[0130] For dynamic routing tables, each server supports the function of probing other servers. The probing function can be in the form of UDP / TCP / SYN / HTTP, etc. The probing time can be once every 30 seconds. After the probing is completed, the data is sent to the adjacent server. Alternatively, all slave servers can send their own probing data to the master server through the transmission path between them and the master server. The master server calculates the shortest path based on the reported data and uses it as the route header DataRoute.

[0131] It is understandable that dynamic routing cannot have IP and port restrictions, otherwise there may be a phenomenon where the probe is successful but the actual communication IP and port will be restricted.

[0132] The dynamic route detection process is as follows:

[0133] 1) Each server (including master and slave servers) needs to be configured to detect neighboring servers. For example, to... Figure 3 Topology ( Figure 2 Taking a tree-like topology (where the relationship is hierarchical and dynamic routing is not possible) as an example, master server 1001 can detect slave servers 2001 and 2002; slave server 2001 can detect master server 1001, slave server 2002, slave server 3001, and slave server 3002; slave server 2002 can detect master server 1001, slave server 2001, slave server 3001, and slave server 3002; slave server 3001 can detect slave servers 2001, slave server 2002, and slave server 3002; and slave server 3002 can detect slave servers 2001, slave server 2002, and slave server 3001.

[0134] 2) Data probed by the server will be synchronized to neighboring servers. For example, data probed by server 2001 from master server 1001, slave server 2002, slave server 3001, and slave server 3002 will be synchronized to master server 1001, slave server 2002, slave server 3001, and slave server 3002. Servers receiving probe data from other servers will also forward it to neighboring nodes. For example, slave server 2002 will receive probe data between slave server 2001 and slave server 3001, or between slave server 3001 and slave server 3002, and will forward it to master server 1001. Synchronization only synchronizes incremental data, which can be done every 30 seconds. Synchronized data increments by 1 each time it is forwarded by a server. Once the increment exceeds 5 (configurable), forwarding will cease to prevent unlimited forwarding of probe data between servers. This ensures that each server has access to the original probe data from all servers. The synchronized probe data includes information such as inter-server latency, packet loss rate, validity period, and weight.

[0135] 3) After receiving probe data from all servers (including between the master server and slave servers, and between adjacent slave servers), the master server calculates edge weights using a specific algorithm based on information such as latency, packet loss rate, validity period, and weight. Then, it calculates the optimal path from the first-level server to all other servers using the shortest path algorithm. The optimal path calculation can be performed every 30 seconds, forming a dynamic routing table.

[0136] For example, latency can be considered alone as the optimal path factor, selecting the path with the shortest latency. Alternatively, latency, packet loss rate, and validity period can be weighted accordingly to determine the optimal path, for example:

[0137]

[0138] Where P is the calculated value, a i Let t be the weight between the i-th group of adjacent servers in a certain path; i d represents the latency between the i-th group of adjacent servers in a given path; i Let T be the packet loss rate between the i-th group of adjacent servers in a certain path; i Let be the validity period between the i-th group of adjacent servers in a certain path; T is the threshold for the validity period between adjacent servers; m is the number of servers in a certain path minus 1.

[0139] The path with the smallest P-value among all paths from the main server to the target audio / video conferencing terminal is selected as the optimal path.

[0140] 4) Before initiating a SIP invite, the target audio / video conferencing terminal will query the dynamic routing table and attach the routing information to the DataRoute header.

[0141] 5) Since each slave server has complete inter-server probe data, each slave server can calculate its own dynamic routing table, which is used as the basis for dynamic routing queries to invite target audio and video conferencing terminals to the meeting room on the slave server.

[0142] S130 optimizes the transmission of uplink and / or downlink audio and video data between the main server and the target audio and video conferencing terminal during the conference based on the audio and video MSG flow control protocol.

[0143] The main server conference room can control meeting member muting and video display switching via API, and optimize audio and video data transmission through the audio / video MSG flow control protocol. Specifically:

[0144] 1) Regarding the optimization of uplink audio data transmission, since the number of simultaneous speakers in a conference is relatively small, most participants only need to listen to the conference content. The target audio / video conferencing terminal that speaks will transmit its audio data to the main server's conference room via the established bidirectional audio / video channel between the main server and the target terminal. Silent target audio / video conferencing terminals, since they do not speak, do not send audio data to the main server's conference room. This method reduces the uplink bandwidth of the terminals.

[0145] 2) Regarding the optimization of uplink video transmission data, since it is not necessary to display the screens of all target audio and video conferencing terminals, only a few target audio and video conferencing terminals need to be displayed. When it is not necessary to display the screens of certain target audio and video conferencing terminals, the primary server conference room will send the audio and video MSG flow control protocol to the target audio and video conferencing terminal to prevent it from uploading video data. This method can reduce the bandwidth of uplink video.

[0146] For example, only the video data of target audio-visual conferencing terminals that have a display indicator (e.g., the target audio-visual conferencing terminal has a display indicator if the main server sends an instruction to the target audio-visual conferencing terminal to enable display) can be transmitted to the main server through the corresponding SIP channel; or, the main server can send an instruction not to upload video data to some target audio-visual conferencing terminals (e.g., the main server sends an instruction to disable display to the target audio-visual conferencing terminal), and the video data of target audio-visual conferencing terminals that do not receive the instruction not to upload video data can be transmitted to the main server through the corresponding SIP channel.

[0147] 3) Optimization of downlink audio transmission data: Audio transmission to muted target audio / video conferencing terminals is as follows: The master server's conference room will send one of the mixed audio streams to each of the next-level slave servers, based on the number of slave servers. The slave servers will then copy and distribute the audio data to the next lower-level slave servers or muted target audio / video conferencing terminals based on the currently participating target terminals. Audio transmission to speaking target audio / video conferencing terminals is as follows: Because speaking is required, the speaking target audio / video conferencing terminal needs to remove its own uploaded audio data from the mixed audio data received from the master server's conference room. Therefore, the speaking target audio / video conferencing terminal needs to send additional audio data.

[0148] 4) For downlink video transmission data optimization, the main server conference room will merge the video screens of the target audio and video conferencing terminals that need to be displayed. After merging, the video stream will be sent to each of the next-level slave servers according to the number of slave servers. The slave servers will then copy and distribute the video to the next lower-level slave servers or the target audio and video conferencing terminals according to the virtual conference room.

[0149] A simplified description of the audio stream control command flow (audio stream control commands using MSG messages) is provided below. Figure 6 As shown:

[0150] 1) Reference for Banning Process Figure 6After steps 1, 2, and 3 are completed, the audio / video conferencing terminal 300101 will not send audio data to the third-level server 3001. At the same time, the third-level server 3001 and the second-level server 2001 will not forward the audio stream data of the audio / video conferencing terminal 300101 to the main server 1001 conference room (reducing uplink audio bandwidth). The third-level server 3001 will use a shared mixed audio stream to copy and send it to the muted audio / video conferencing terminal 300101 (reducing downlink audio bandwidth). The audio data streams sent separately to the audio / video conferencing terminal 300101 from the main server 1001 conference room, the second-level server 2001, and the third-level server 3001 will stop being sent.

[0151] 2) Speech Flow Reference Figure 6 After steps 4, 5, and 6 are completed, the audio-visual conferencing terminal 300101 will send audio stream data to the third-level server 3001. At this time, the main server 1001 conference room will remove the audio data uploaded by the audio-visual conferencing terminal 300101 from the mixed audio data, and forward it to the audio-visual conferencing terminal 300101 separately through the second-level server 2001 and the third-level server 3001.

[0152] A simplified description of the video stream control command process (using MSG messages). For example, if the host wants to display the screens of conference initiating terminal 100101, audio / video conferencing terminal 200101, and audio / video conferencing terminal 300101, see below for details. Figure 7 As shown:

[0153] 1) The main server (meeting room 1001) will send a video upload command to the meeting initiating terminal (100101), see reference [link / reference]. Figure 7 Step 4: The meeting initiating terminal 100101 will upload the video to the main server 1001 meeting room.

[0154] 2) The main server 1001 will send video transmission start commands for audio / video conferencing terminals 200101 and 300101, and video upload stop commands for audio / video conferencing terminals 200102, 200103, 300102, and 300103, to the secondary server 2001. (Refer to...) Figure 7 Step 2.

[0155] 3) The secondary server 2001 virtual conference room will send video upload control commands to the audio / video conferencing terminal 200101, see reference. Figure 7In step 5, audio / video conferencing terminal 200101 will upload the video to secondary server 2001 and forward it to primary server 1001. Secondary server 2001's virtual meeting room sends a video upload stop command to audio / video conferencing terminals 200102 and 200103, refer to [reference needed]. Figure 7 In steps 6 and 7, audio and video conferencing terminals 200102 and 200103 will stop uploading video streams to the secondary server 2001 virtual conference room (reducing uplink video bandwidth).

[0156] 4) The virtual conference room on the secondary server 2001 will send a video upload start command to the audio / video conferencing terminal 300101 and a video upload stop command to the audio / video conferencing terminals 300102 and 300103 to the tertiary server 3001. (See reference...) Figure 7 Step 3.

[0157] 5) The third-level server 3001 virtual conference room will send video upload control commands to the audio / video conferencing terminal 300101, see reference. Figure 7 In step 8, audio / video conferencing terminal 300101 will upload the video to the third-level server 3001, and then forward it to the second-level server 2001, which in turn forwards it to the main server 1001. The virtual conference room on the third-level server 3001 sends a video upload stop command to audio / video conferencing terminals 300102 and 300103, as per [reference needed]. Figure 7 In steps 9 and 10, audio and video conferencing terminals 300102 and 300103 will stop uploading video streams to the third-level server 3001 virtual conference room. At this time, the second-level server 2001 will also stop forwarding these video streams to the first-level server 1001 (reducing uplink video bandwidth).

[0158] 6) The main server 1001 will receive video streams from audio and video conferencing terminals 100101, 200101, and 300101, merge these video streams into one screen, and forward it to other audio and video conferencing terminals.

[0159] 7) Since the audio / video conferencing terminals 200101, 200102, 200103, 300101, 300102, and 300103 display the same video feed, the main server only needs to send one video stream to the secondary server 2001 (reducing downlink video bandwidth). The secondary server 2001 then copies and distributes four video streams to the audio / video conferencing terminals 200101, 200102, and 200103 and the tertiary server 3001 (reducing downlink video bandwidth).

[0160] 8) The virtual conference room of the third-level server 3001 then copies and distributes three video streams to the audio and video conferencing terminals 300101, 300102, and 300103 respectively.

[0161] 9) Some target audio and video conferencing terminals can also display different video screens. A separate video stream can be sent to this target audio and video conferencing terminal. The method can refer to step 2) of the audio stream control instruction flow.

[0162] Example 2

[0163] Please see Figure 2 As shown in Figure 3, a SIP-based audio and video conferencing system may include: a master server, one or more slave servers, a conference initiating terminal, and an audio and video conferencing terminal. The master server and one or more slave servers construct a communication topology. The audio and video conferencing terminal is connected to the communication topology through a corresponding slave server. The conference initiating terminal communicates directly with the master server.

[0164] The meeting initiating terminal creates a meeting room through the main server and sends a meeting invitation to the target audio and video conferencing terminal.

[0165] The master server establishes a SIP channel with the target audio and video conferencing terminal via the communication topology through static or dynamic routing.

[0166] During the meeting, the main server optimizes the transmission of uplink and / or downlink audio and video data between the main server and the target audio and video conferencing terminal based on the audio and video MSG flow control protocol.

[0167] Preferably, the meeting initiating terminal sends a first SIP invite request to the main server to create a meeting room and establish a two-way audio and video transmission channel between the meeting initiating terminal and the main server;

[0168] The meeting initiating terminal sends a meeting invitation to the target audio and video conferencing terminal to the main server via API.

[0169] The master server determines the transmission path through a static or dynamic routing table, and initiates a second SIP invite request to the target audio and video conferencing terminal according to the transmission path to establish a SIP channel between the master server and the target audio and video conferencing terminal.

[0170] Each slave server along the transmission path establishes a virtual conference room to distribute the second SIP invite request.

[0171] Preferably, the master server queries a static or dynamic routing table to determine the transmission path between the master server and the target audio / video conferencing terminal; the master server determines DataRoute information based on the transmission path and assembles the DataRoute information into the second SIP invite request;

[0172] The preceding target server in the transmission path forwards the second SIP invite request to the following target server based on the DataRoute information, establishing a SIP channel between the two. When the preceding target server forwards the second SIP invite request, it deletes the DataRoute information corresponding to the following target server in the second SIP invite request.

[0173] If the second SIP invite request received by the target server in the transmission path does not carry DataRoute information, it is designated as an associated slave server. Then, the second SIP invite request is forwarded to the target audio and video conferencing terminal through the virtual conference room created by the associated slave server, thereby establishing a SIP channel between the associated slave server and the target audio and video conferencing terminal.

[0174] Preferably, the optimization of uplink audio data transmission includes: transmitting only the audio data of the target audio / video conferencing terminal speaking to the main server through the corresponding SIP channel;

[0175] Optimization of uplink video data transmission includes: transmitting video data from target audio and video conferencing terminals with display markers to the main server only through the corresponding SIP channel; or, the main server sends instructions to some target audio and video conferencing terminals not to upload video data, and the video data of target audio and video conferencing terminals that do not receive the instructions to not upload video data is transmitted to the main server through the corresponding SIP channel.

[0176] The optimization of downlink audio data transmission includes: the main server distributes the mixed audio stream to all muted target audio and video conferencing terminals at once; the main server sends the optimized mixed audio stream to the speaking target audio and video conferencing terminal through the corresponding SIP channel; the optimized mixed audio stream removes the audio data uploaded by the speaking target audio and video conferencing terminal.

[0177] Optimizations for downlink video data transmission include: the main server distributing the merged video feed to all target audio and video conferencing terminals at once.

[0178] In various embodiments of the present invention, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0179] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they can be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0180] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0181] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several requests to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the methods described in the various embodiments of the present invention.

[0182] In the embodiments provided by this invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0183] Those skilled in the art will understand that some or all of the steps in the various methods of the embodiments described can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0184] The foregoing has provided a detailed description of a SIP-based audio and video conferencing method and system disclosed in the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A SIP-based audio and video conferencing method, characterized in that, It is based on an IP network conferencing system, which includes: a master server, one or more slave servers, and audio and video conferencing terminals. The master server and one or more slave servers construct a communication topology, and the audio and video conferencing terminals are connected to the communication topology through corresponding slave servers. The method includes: The meeting initiating terminal creates a meeting room through the main server and sends a meeting invitation to the target audio and video conferencing terminal; The main server establishes a SIP channel with the target audio / video conferencing terminal via the communication topology based on the membership invitation, using either static or dynamic routing. This specifically includes: The master server determines the transmission path through a static or dynamic routing table, determines DataRoute information based on the transmission path, and assembles the DataRoute information into a second SIP invite request. The preceding target server in the transmission path forwards the second SIP invite request to the following target server based on the DataRoute information, establishing a SIP channel between the two. When the preceding target server forwards the second SIP invite request, it deletes the DataRoute information corresponding to the following target server in the second SIP invite request. If the second SIP invite request received by the target server in the transmission path does not carry DataRoute information, it is referred to as an associated slave server. Then, the second SIP invite request is forwarded to the target audio and video conferencing terminal through the virtual conference room created by the associated slave server, and a SIP channel between the associated slave server and the target audio and video conferencing terminal is established. Each slave server along the transmission path establishes a virtual conference room to distribute the second SIP invite request; During the meeting, the transmission of uplink and / or downlink audio and video data between the host server and the target audio and video conferencing terminal is optimized based on the audio and video MSG flow control protocol.

2. The SIP-based audio and video conferencing method according to claim 1, characterized in that, The meeting initiating terminal creates a meeting room through the main server, including: The conference initiating terminal sends a first SIP invite request to the main server to create a conference room and establish a two-way audio and video transmission channel between the conference initiating terminal and the main server.

3. The SIP-based audio and video conferencing method according to claim 1, characterized in that, When the master server determines the transmission path through the static routing table, the master server queries the static routing table to determine the transmission path between the master server and the target audio and video conferencing terminal.

4. The SIP-based audio and video conferencing method according to claim 1, characterized in that, When the master server determines the transmission path through the dynamic routing table, the master server queries the dynamic routing table to determine the transmission path between the master server and the target audio and video conferencing terminal.

5. The SIP-based audio and video conferencing method according to claim 4, characterized in that, The master server queries the dynamic routing table to determine the transmission path between the master server and the target audio / video conferencing terminal, including: In the communication topology, any two adjacent slave servers, as well as the master server and slave servers, probe each other's data, acquire probe data, and synchronize the probe data to all adjacent master servers or slave servers. Alternatively, all slave servers can send their own probe data to the master server through the transmission path between them, so that the master server can acquire the probe data from all slave servers. The probe data includes one or more of the following: latency, packet loss rate, validity period, and weight transmitted between adjacent slave servers, and between the master server and slave servers. The master server determines all transmission paths to the target audio and video conferencing terminal, and determines the optimal path among all transmission paths based on the detection data, which is used as the transmission path between the master server and the target audio and video conferencing terminal and is denoted as the target path.

6. The SIP-based audio and video conferencing method according to claim 1, characterized in that, The transmission of uplink and / or downlink audio and video data between the master server and the target audio and video conferencing terminal is optimized based on the audio and video MSG flow control protocol, including: Optimizations for uplink audio data transmission include: transmitting only the audio data of the target audio / video conferencing terminal speaking to the main server via the corresponding SIP channel; Optimization of uplink video data transmission includes: transmitting video data from target audio and video conferencing terminals with display markers to the main server only through the corresponding SIP channel; or, the main server sends instructions to some target audio and video conferencing terminals not to upload video data, and the video data of target audio and video conferencing terminals that do not receive the instructions to not upload video data is transmitted to the main server through the corresponding SIP channel. The optimization of downlink audio data transmission includes: the main server distributes the mixed audio stream to all muted target audio and video conferencing terminals at once; the main server sends the optimized mixed audio stream to the speaking target audio and video conferencing terminal through the corresponding SIP channel; the optimized mixed audio stream removes the audio data uploaded by the speaking target audio and video conferencing terminal. Optimizations for downlink video data transmission include: the main server distributing the merged video feed to all target audio and video conferencing terminals at once.

7. A SIP-based audio and video conferencing system, characterized in that, It includes: The system comprises a master server, one or more slave servers, a conference initiating terminal, and an audio / video conferencing terminal. The master server and one or more slave servers construct a communication topology. The audio / video conferencing terminal connects to the communication topology through a corresponding slave server. The conference initiating terminal communicates directly with the master server. The meeting initiating terminal creates a meeting room through the main server and sends a meeting invitation to the target audio and video conferencing terminal. The main server establishes a SIP channel with the target audio / video conferencing terminal via the communication topology based on the membership invitation, using either static or dynamic routing. This specifically includes: The master server determines the transmission path through a static or dynamic routing table, determines DataRoute information based on the transmission path, and assembles the DataRoute information into a second SIP invite request. The preceding target server in the transmission path forwards the second SIP invite request to the following target server based on the DataRoute information, establishing a SIP channel between the two. When the preceding target server forwards the second SIP invite request, it deletes the DataRoute information corresponding to the following target server in the second SIP invite request. If the second SIP invite request received by the target server in the transmission path does not carry DataRoute information, it is referred to as an associated slave server. Then, the second SIP invite request is forwarded to the target audio and video conferencing terminal through the virtual conference room created by the associated slave server, and a SIP channel between the associated slave server and the target audio and video conferencing terminal is established. Each slave server along the transmission path establishes a virtual conference room to distribute the second SIP invite request; During the meeting, the main server optimizes the transmission of uplink and / or downlink audio and video data between the main server and the target audio and video conferencing terminal based on the audio and video MSG flow control protocol.

8. The SIP-based audio and video conferencing system according to claim 7, characterized in that, The meeting initiating terminal sends a first SIP invite request to the main server to create a meeting room and establish a two-way audio and video transmission channel between the meeting initiating terminal and the main server. The meeting initiating terminal sends a meeting invitation to the main server via API to invite the target audio and video conferencing terminal to join the meeting.

9. The SIP-based audio and video conferencing system according to claim 7, characterized in that, Optimizations for uplink audio data transmission include: transmitting only the audio data of the target audio / video conferencing terminal speaking to the main server via the corresponding SIP channel; Optimization of uplink video data transmission includes: transmitting video data from target audio and video conferencing terminals with display markers to the main server only through the corresponding SIP channel; or, the main server sends instructions to some target audio and video conferencing terminals not to upload video data, and the video data of target audio and video conferencing terminals that do not receive the instructions to not upload video data is transmitted to the main server through the corresponding SIP channel. The optimization of downlink audio data transmission includes: the main server distributes the mixed audio stream to all muted target audio and video conferencing terminals at once; the main server sends the optimized mixed audio stream to the speaking target audio and video conferencing terminal through the corresponding SIP channel; the optimized mixed audio stream removes the audio data uploaded by the speaking target audio and video conferencing terminal. Optimizations for downlink video data transmission include: the main server distributing the merged video feed to all target audio and video conferencing terminals at once.

Images