RTC-based multi-mode high-availability method and system for instant audio and video communication

Abstract

The invention discloses a multi-mode high availability method and system for instant audio and video communication based on RTC, and the method comprises the following steps: carrying out the real-time network quality monitoring of a client in an online communication system, and reporting the monitoring data to a server through a heartbeat packet; the server analyzes and judges the monitoring data according to a set network quality judgment rule, and marks the available state of the first RTC channel; in response to the fact that the state of the first RTC channel is marked as unavailable, the server marks a corresponding virtual space, modifies a second RTC channel corresponding to the virtual space, and issues an RTC channel switching instruction to the client; the client obtains information of a second RTC channel according to the channel switching instruction, records a virtual space state, switches to the second RTC channel, and releases the first RTC channel; and the client recovers the recorded virtual space state. According to the method provided by the invention, high availability of online communication is realized, and the switching process is non-inductive to a user, so that the user experience is improved.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and in particular relates to a high-availability method and system for multi-modal real-time audio and video communication based on RTC. Background Art

[0002] In traditional online audio and video communication (such as online live broadcast), content is generally distributed through CDN (Content Delivery Network) based on the RTMP (Real-Time Messaging Protocol) standard protocol. Due to the cache mechanism, in order to provide users with a better viewing experience, there is often a delay of several seconds or even more than ten seconds. The interaction between the anchor and the audience can only be weakly interactive through text, gift sending, etc. This is why when we watch live broadcasts selling goods, users inquire about a product through text, and it takes more than ten seconds or even longer to receive the anchor's reply.

[0003] In recent years, real-time communication (RTC, or real-time audio and video communication) technology has become an important bridge connecting people and businesses. From simple online chats to complex remote collaborations, to modern virtual reality and augmented reality experiences, the application scope of real-time audio and video technology continues to expand. RTC technology can be simply understood as the transmission and interaction of audio, video and data in a real-time network environment. It is a technology that supports real-time voice, real-time video and real-time chat. Compared with traditional communication technologies, RTC technology has lower latency and better audio and video effects, and can meet the strict requirements of real-time communication.

[0004] The Chinese invention patent application with publication number CN113630439A involves the application of RTC technology, which discloses a real-time communication RTC connection method, server and storage medium. The server is configured to bind the business process to the UDP port one-to-one. After receiving the SDP request information for resource negotiation sent by one or more clients based on the TCP port of the server, the business process sends SDP response information to each client respectively. The SDP response information carries the UDP port information corresponding to the business process that processes the SDP request; the business process receives the UDP connection request sent by the client from the corresponding UDP port, and uses the UDP connect method to establish a UDP connection, thereby realizing the RTC connection.

[0005] When online audio and video real-time communication is based on RTC technology, the quality of audio and video may be degraded or disconnected due to network instability such as network fluctuations, packet loss, etc., causing communication failures and affecting the high availability of online communication. Therefore, the Chinese invention patent application with publication number CN112804105A discloses a method and system for quickly repairing terminal communication failures in an RTC network, including the following steps: S1: generating a fault scheduling configuration for the terminal according to the fault, the fault scheduling configuration including information of the fault debugging node; S2: scheduling the terminal to the fault debugging node according to the fault scheduling configuration; S3: the fault debugging node detects and repairs the fault and generates a fault repair upgrade version; S4: generating a new scheduling configuration for the terminal according to the result of the fault repair.

[0006] It can be seen that the mainstream multi-person online instant audio and video chat scenarios on the market at this stage mostly rely on a single RTC mode. When a service fails, since there is only one RTC channel in a single mode and it is only provided by one service provider, when the service provider fails to rescue the line in time, and there are no other modes to switch to, it will lead to problems such as timely communication service interruption, causing serious impact and loss to users. The above solution only troubleshoots and repairs the faults that occur. Troubleshooting and repair may take time, especially in a complex network environment, resulting in a long communication interruption. During the fault handling period, users may encounter audio and video interruptions, affecting the communication experience. Although multiple RTC modes have been explored, they have failed to achieve user-unawareness and real-time switching. Summary of the invention

[0007] The present invention provides a high-availability method and system for multi-modal real-time audio and video communication based on RTC, aiming to solve the problem of low system availability and impaired user experience caused by long response time during RTC fault processing in the prior art.

[0008] In order to solve the above technical problems, the online communication high availability method proposed in the present invention comprises the following steps:

[0009] In the online communication system, the client monitors the network quality in real time and reports the monitoring data to the server through the heartbeat packet;

[0010] The server analyzes and determines the monitoring data according to the set network quality judgment rules, and marks the first RTC channel as available;

[0011] In response to the first RTC channel status being marked as unavailable, the server marks the corresponding virtual space, modifies the virtual space to correspond to the second RTC channel, and sends an RTC channel switching instruction to the client;

[0012] The client obtains information of the second RTC channel according to the channel switching instruction, saves a snapshot of the virtual space, switches to the second RTC channel, and releases the first RTC channel;

[0013] The client uses the virtual space snapshot to restore the virtual space to the state before switching the RTC channel.

[0014] Preferably, the dimensions of network quality monitoring include login quality, stream pulling quality, stream pushing quality, network indicators, audio quality and video quality.

[0015] Preferably, the login quality includes login success rate and login time consumption;

[0016] The method for calculating the login success rate is as follows: a user attempts to log in to a specified virtual space, and the parameters of the login method include a virtual space ID, user information, and virtual space configuration; after the login request is processed, the system performs a callback process, and records the number of login successes or failures according to the callback parameters; the login success rate is calculated according to the number of login successes and login failures;

[0017] The method for calculating the login time is as follows: before calling the login method, record the first timestamp as the start time of the login; in the callback of the login result, record the second timestamp as the end time of the login; the login time is the difference between the first timestamp and the second timestamp.

[0018] Preferably, the evaluation index of the flow pulling quality is the flow pulling success rate, and the calculation method of the flow pulling success rate is as follows:

[0019] Each time a user requests to pull a stream, it is recorded as a stream pulling attempt;

[0020] Monitor the stream pulling status in the stream pulling status callback and determine whether the stream pulling is successful based on the callback parameters;

[0021] If the stream is not successfully pulled within the custom stream pulling timer, this attempt will be considered a failure;

[0022] Maintain two counters to record the number of successful and failed stream pulling.

[0023] The success rate of pulling the stream is calculated based on the number of successful pulling and the number of failed pulling.

[0024] Preferably, the evaluation index of the streaming quality is the streaming success rate, and the streaming success rate is calculated as follows:

[0025] Each time a user requests to push a stream, it is recorded as a push attempt.

[0026] Monitor the streaming status in the streaming status callback and determine whether the streaming is successful based on the callback parameters;

[0027] If the push attempt fails within the custom push timer, the attempt will be considered a failure.

[0028] Maintain two counters to record the number of successful and failed pushes.

[0029] The streaming success rate is calculated based on the number of successful streaming and the number of failed streaming.

[0030] Preferably, the network indicators include network delay and packet loss rate;

[0031] The network delay calculation method is as follows: defining a round trip delay, where the round trip delay represents the round trip time from the streaming end to the server; recording the current round trip delay each time the network quality function is called;

[0032] The packet loss rate is calculated by directly obtaining the current packet loss rate each time the network quality function is called.

[0033] Preferably, the evaluation indicators of the audio quality and video quality include first frame time, cumulative freeze rate and cumulative freeze duration;

[0034] The calculation method of the first frame time consumption is as follows: when calling the method for starting to pull the stream, the third timestamp is recorded as the time when the stream is started; when the first frame of audio or video data arrives, the callback trigger of receiving the first frame is received, and the fourth timestamp is recorded as the time when the first frame is received; the difference between the third timestamp and the fourth timestamp is used as the first frame time consumption;

[0035] The cumulative freeze duration is calculated as follows: in the media event callback, the audio or video freeze event is monitored, the freeze start event and the freeze end event are recorded, and the single freeze duration is calculated; at the end of each freeze event, the freeze duration is added to the cumulative freeze duration;

[0036] The cumulative jamming rate is calculated as follows: when starting to pull the stream, the start time is recorded, and the total stream pulling time is calculated, and the total stream pulling time is updated in each network quality function callback; at the end of each stream pulling, the current cumulative jamming time is obtained, and the ratio of the current cumulative jamming time to the current total stream pulling time is used as the cumulative jamming rate.

[0037] Preferably, the method by which the server analyzes and determines the monitoring data according to the set network quality determination rule is specifically:

[0038] Calculate an average freeze rate of users in the virtual space within a set first time interval, and if the average freeze rate exceeds a set first threshold, set the freeze rate status to not meeting the standard;

[0039] Calculate the average packet loss rate of users in the set first time interval, and if the average packet loss rate exceeds the set second threshold, set the packet loss rate status to not meeting the standard;

[0040] Calculate the average user delay in the set first time interval, and if the average delay exceeds a set third threshold, set the average delay status to not meeting the standard;

[0041] If the number of times that the jamming rate does not meet the standard and / or the packet loss rate does not meet the standard and / or the average delay does not meet the standard exceeds the set number of times within the set second time interval, it is determined that the corresponding RTC channel is unavailable.

[0042] After determining that the corresponding RTC channel is unavailable, the server also performs new RTC detection and volume expansion operations;

[0043] The new RTC detection method is as follows: select the virtual space with the lowest number of users in the room as the detection space; issue an instruction to notify the room owner to actively detect the freeze rate, packet loss rate and average delay when joining the virtual space based on the new RTC creation engine; if the new RTC detection passes, it will be gradually increased;

[0044] The method of increasing the volume is as follows: gradually increasing the volume at a preset first ratio, continuously performing quality detection of the jamming rate, packet loss rate and average delay during the increasing volume, and if the quality detection passes, gradually continuing to increase the volume until the total proportion of the increased volume reaches a preset second ratio.

[0045] Preferably, when the server sends the RTC channel switching instruction to the client, it also records the audio input device allocation status in the current virtual space; after the client switches to the second RTC channel, the recorded audio input device allocation status is distributed to the client.

[0046] Another aspect of the present invention further provides a high-availability system for multi-modal real-time audio and video communication based on RTC, the system is used to implement the above-mentioned high-availability method, including a client and a server, the client uses multiple RTC channels to connect to the server; the client further includes a status monitoring module and a first channel switching module, the server further includes a service quality monitoring module and a second channel switching module;

[0047] The status monitoring module monitors the network quality in real time, obtains login quality, stream pulling quality, stream pushing quality, network indicators, audio quality and video quality, and reports the monitoring data to the server through the heartbeat packet;

[0048] The first channel switching module performs an RTC channel switching operation according to the RTC channel switching instruction sent by the server, saves a virtual space snapshot, starts a second RTC channel, adds the user to the second RTC channel, and uses the virtual space snapshot to restore the virtual space to a state before switching the RTC channel;

[0049] The service quality monitoring module server analyzes and determines the monitoring data according to the set network quality determination rules, and marks the first RTC channel as available;

[0050] When the first RTC channel is marked as unavailable, the second channel switching module marks the corresponding virtual space, modifies the virtual space to correspond to the second RTC channel, and sends an RTC channel switching instruction to the client.

[0051] Compared with the prior art, the present invention has the following technical effects:

[0052] 1. The online communication high-availability method proposed in the present invention uses multiple RTC channels. When the RTC line at one end is detected to be unstable, the multi-RTC mode can be switched online through intelligent comparison and analysis to ensure the user's instant communication is uninterrupted and provide stable service quality. The availability of the system can be improved. Even if one channel fails, it can still switch to other channels seamlessly to ensure the continuity of communication.

[0053] 2. The online communication high availability method proposed by the present invention usually performs fault switching automatically, and the user hardly feels the interruption, thereby improving the user experience in online audio and video communication.

[0054] 3. The online communication high-availability system proposed by the present invention can dynamically adjust channel usage according to different network conditions and usage scenarios, thereby improving flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flowchart of the online communication high availability method of the present invention;

[0056] Figure 2 It is a schematic diagram of the process of the client performing network quality monitoring and reporting according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the objectives, technical solutions and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in combination with specific embodiments of the present application and with reference to the accompanying drawings.

[0058] Embodiment 1

[0059] This embodiment is a high-availability method for multi-modal real-time audio and video communication based on RTC, such as Figure 1 As shown, the steps include steps one to five:

[0060] Step 1: In the online communication system, the client monitors the network quality in real time and reports the monitoring data to the server through the heartbeat packet. The process of client RTC quality monitoring reporting is as follows: Figure 2 shown.

[0061] In this embodiment, the dimensions of network quality monitoring include login quality, stream pulling quality, stream pushing quality, network indicators, audio quality and video quality.

[0062] Specifically, the login quality includes login success rate and login time.

[0063] The method for calculating the login success rate is as follows: the user attempts to log in to the specified virtual space, and the parameters of the login method include the virtual space ID, user information, and virtual space configuration; after the login request is processed, the system performs a callback process and records the number of successful or failed logins according to the callback parameters; the login success rate is calculated according to the number of successful logins and the number of failed logins. It can be understood by those skilled in the art that the virtual space described in this embodiment can be a conference room for a virtual space online meeting, a live broadcast room for a webcast, an interactive room or classroom for online teaching, or a conversation room or voice chat room for other online exchanges.

[0064] In one embodiment of the present invention, the user logs in to the specified virtual space through the mExpressEngine.loginRoom() method. When the login request is processed, the system calls the onRoomLoginResult method in the IZegoRoomLoginCallback interface, and the callback contains the errorCode parameter. If the errorCode is 0, it means that the login is successful; if the errorCode is a non-zero value, it means that the login failed (for example, token exception, etc.). The result of each login attempt is recorded. The number of successful logins (successCount) plus the number of failed logins (failureCount) can get the total number of attempts. The ratio of the number of successful logins to the total number of login attempts is the login success rate.

[0065] In some embodiments of the present invention, calculating the login success rate also includes timeout processing: if there is no callback within 20 minutes, that is, the result cannot be received due to network abnormality or other reasons, a timer (for example, 5s or 10s) can be set, and the timeout is also regarded as a login failure.

[0066] The method for calculating the login time is as follows: before calling the login method, record the first timestamp as the start time of the login; in the callback of the login result, record the second timestamp as the end time of the login; the login time is the difference between the first timestamp and the second timestamp.

[0067] In one embodiment of the present invention, before calling the mExpressEngine.loginRoom() method, the current timestamp is recorded as the login start time. Then a login request is sent, and the system starts processing the login operation. When the login request is completed, the result is returned through the onRoomLoginResult method of the RoomLoginCallback interface. In the onRoomLoginResult callback, the current timestamp is recorded as the login end time. The login time can be calculated by subtracting the start time from the end time, and the unit is generally milliseconds.

[0068] The evaluation index of the pulling quality is the pulling success rate, and the calculation method of the pulling success rate is as follows:

[0069] Each time a user requests to pull a stream, such as calling startPlayingStream(), is recorded as a stream pulling attempt; in this step, first define the number of stream pulling attempts. Monitor the stream pulling status in the stream pulling status callback, such as onPlayerStateUpdate(), and determine whether the stream pulling is successful based on the callback parameters; the callback parameters include the passed-in state and errorCode. When the state is PLAYING, it indicates that the stream pulling is successful. When the state is NO_PLAY and the errorCode is not 0, it indicates that the stream pulling failed. If the stream pulling fails within the custom stream pulling timer (such as 5 seconds or 10 seconds), this attempt is considered a failure. Maintain two counters successCount and failureCount, where successCount records the number of successful stream pulling and failureCount records the number of failed stream pulling. Calculate the stream pulling success rate based on the number of successful stream pulling and the number of failed stream pulling.

[0070] The evaluation index of the streaming quality is the streaming success rate. The calculation method of the streaming success rate is similar to that of the pulling success rate. The calculation method of the streaming success rate is as follows:

[0071] Each time a user requests to push a stream, such as calling startPublishingStream(), is recorded as a push attempt. Monitor the push status in the push status callback, such as onPublisherStateUpdate(), and determine whether the push is successful based on the callback parameters; the callback parameters include the passed-in state and errorCode. When the state is PUBLISHING, it means that the push is successful; when the state is NO_PUBLISH and the errorCode is not 0, it means that the push failed; when the state is NO_PUBLISH and the errorCode is 0, it means that the user actively stopped the push. If the push fails within the custom push timer (such as 5 seconds or 10 seconds), this attempt is considered a failure. Maintain two counters, successCount and failureCount. SuccessCount records the number of successful pushes, and failureCount records the number of failed pushes. Calculate the push success rate based on the number of successful pushes and the number of failed pushes.

[0072] The network indicators include network delay and packet loss rate.

[0073] The calculation method of the network delay is as follows: define the round trip time (RTT), which represents the round trip time from the streaming end to the server; record the current round trip delay each time the network quality function is called. Specifically, in the onPlayerQualityUpdate() callback, quality.rtt represents the round trip time (RTT) from the streaming end to the server, and this value can be regarded as an important indicator of network delay. Each time onPlayerQualityUpdate() is called, the current quality.rtt value is recorded. This will reflect the current network delay. The client can periodically send heartbeat packets to the server and record the timestamp each time it is sent for comparison with the RTT. In order to more comprehensively evaluate the network delay, multiple RTT values ​​can be collected over a period of time and their average value can be calculated. This can reduce the impact of fluctuations caused by a single measurement.

[0074] The packet loss rate is calculated as follows: each time the network quality function is called, the current packet loss rate is directly obtained. Specifically, in the onPlayerQualityUpdate() callback, quality.packetLostRate represents the packet loss rate from the streaming end to the server. This value is usually expressed as a percentage, reflecting the last statistical packet loss at the callback time point. Each time onPlayerQualityUpdate() is called, the current quality.packetLostRate value is directly obtained to reflect the current network packet loss situation. In order to more comprehensively evaluate the network packet loss situation, multiple packet loss rate values ​​can be collected over a period of time, and their average value can be calculated to reduce the impact of short-term fluctuations.

[0075] The evaluation indicators of the audio quality and video quality include first frame time, cumulative freeze rate and cumulative freeze duration;

[0076] The calculation method of the first frame time consumption is: when calling the method to start streaming, record the third timestamp as the start time of streaming; when the first frame of audio or video data arrives, receive the callback trigger of the first frame, and record the fourth timestamp as the time of receiving the first frame; the difference between the third timestamp and the fourth timestamp is used as the first frame time consumption.

[0077] Specifically, the first frame time refers to the time from the start of the stream pull request to the receipt of the first frame of audio or video data. When calling the method to start the stream pull, such as startPlayingStream(), record the current timestamp as the start time of the stream pull. When the first frame of audio or video data arrives, the onPlayerRecvFirstFrame() callback will be triggered. In the onPlayerRecvFirstFrame() callback, record the current timestamp as the time when the first frame is received. The first frame time can be calculated by subtracting the start time from the end time.

[0078] The cumulative freeze duration is calculated as follows: in a media event callback, audio or video freeze events are monitored, freeze start events and freeze end events are recorded, and a single freeze duration is calculated; at the end of each freeze event, the freeze duration is added to the cumulative freeze duration.

[0079] Specifically, the cumulative freeze duration refers to the sum of the durations of all audio freeze events during a stream pulling process. In the onPlayerMediaEvent() callback, monitor the audio freeze events. When event is PlayerMediaEvent.AUDIO_BREAK_OCCUR, it indicates that the audio freeze has started; when event is PlayerMediaEvent.AUDIO_BREAK_RESUME, it indicates that the audio freeze has ended. When the freeze starts, record the current timestamp as the freeze start time; when the freeze ends, calculate the time difference from the start to the end and add it to the total freeze duration. At the end of each freeze event, add the freeze duration of that segment to the cumulative freeze duration.

[0080] The cumulative jamming rate is calculated as follows: when starting to pull the stream, the start time is recorded, and the total stream pulling time is calculated, and the total stream pulling time is updated in each network quality function callback; at the end of each stream pulling, the current cumulative jamming time is obtained, and the ratio of the current cumulative jamming time to the current total stream pulling time is used as the cumulative jamming rate.

[0081] Specifically, when starting to play the stream, call startPlayingStream() to record the start time and calculate the total streaming duration. Update the duration in each onPlayerQualityUpdate() callback. In the onPlayerQualityUpdate() callback, calculate the total duration from the start of the stream to the current time and update the total streaming duration totalDuration.

[0082] Step 2: The server analyzes and determines the monitoring data according to the set network quality determination rules, and marks the first RTC channel as available.

[0083] In this step, the method in which the server analyzes and determines the monitoring data according to the set network quality determination rules is specifically as follows:

[0084] Calculate an average freeze rate of users in the virtual space within a set first time interval, and if the average freeze rate exceeds a set first threshold, set the freeze rate status to not meeting the standard;

[0085] Calculate the average packet loss rate of users in the set first time interval, and if the average packet loss rate exceeds the set second threshold, set the packet loss rate status to not meeting the standard;

[0086] Calculate the average user delay in the set first time interval, and if the average delay exceeds a set third threshold, set the average delay status to not meeting the standard;

[0087] If the number of times that the jam rate does not meet the standard and / or the packet loss rate does not meet the standard and / or the average delay does not meet the standard exceeds the set number of times within the set second time interval, the corresponding RTC channel is determined to be unavailable. Those skilled in the art can understand that the three states of jam rate not meeting the standard, packet loss rate not meeting the standard, and average delay not meeting the standard can be combined and operated or operated in conjunction as required, such as jam rate not meeting the standard and packet loss rate not meeting the standard and average delay not meeting the standard, (jamming rate not meeting the standard and packet loss rate not meeting the standard) or average delay not meeting the standard, jam rate not meeting the standard or packet loss rate not meeting the standard or average delay not meeting the standard, jam rate not meeting the standard and (packet loss rate not meeting the standard or average delay not meeting the standard), etc., and the configuration of the and or relationship can be performed according to specific implementation requirements.

[0088] In some other embodiments of the present invention, the server further performs new RTC detection and volume expansion operations after determining that the corresponding RTC channel is unavailable;

[0089] The new RTC detection method is: select the virtual space with the lowest number of users in the room as the detection space; issue instructions to notify the room owner to actively detect the jamming rate, packet loss rate and average delay when joining the virtual space based on the new RTC creation engine; if the new RTC detection passes, that is, the above jamming rate, packet loss rate and average delay indicators are all up to standard, or the combination of these three indicators is the preset standard state, it will be gradually increased. During detection, only the available state can be detected, and other quality indicators cannot be detected. It can be understood by those skilled in the art that, under the premise of quality indicator requirements, the available state can be detected during detection, as well as the detection of other quality indicators.

[0090] The method of increasing the volume is as follows: gradually increasing the volume at a preset first ratio, continuously performing quality detection of the jamming rate, packet loss rate and average delay during the increasing volume, and gradually continuing to increase the volume if the quality detection passes, until the total volume ratio reaches a preset second ratio. Both the first ratio and the second ratio can be specifically configured according to implementation requirements. In a specific implementation of the present invention, the first ratio is configured to 10% and the second ratio is configured to 100%. If an RTC server exception occurs and the service is unavailable, the system can also forcibly configure the second ratio to 100%.

[0091] In this embodiment, the first time interval is set to 30 seconds, the second time interval is set to 10 minutes, the first threshold is set to 30%, the second threshold is set to 10%, the third threshold is set to 1500 milliseconds, and the number of settings can be 10. Those skilled in the art can understand that in other embodiments of the present invention, the above values ​​can be specifically set according to needs.

[0092] Step three, in response to the first RTC channel status being marked as unavailable, the server marks the corresponding virtual space, modifies the virtual space to correspond to the second RTC channel, and sends an RTC channel switching instruction to the client.

[0093] Specifically, if it is determined that the RTC channel needs to be switched due to substandard RTC service quality or other operational policy requirements, the virtual space, such as the chat room status, is marked as "RTC switching in progress" to avoid triggering the old RTC chat room closing callback after the client releases the old RTC channel, causing the room to be disbanded and unable to switch. Then modify the chat room to correspond to the new RTC channel, and new users who enter the chat room will go directly to the newly added RTC channel through the room entry process to ensure that the providers used by users' audio and video streams are consistent and can interact normally. Finally, send the RTC switching command to the client, and at the same time, add the RTC channel attributes currently being used by the room to the technology that maintains real-time final consistency of key data in the chat room for 3 seconds to ensure that all users in the chat room use the same RTC channel.

[0094] When the server sends the RTC channel switching instruction to the client, it also records the audio input device allocation status in the current virtual space; after the client switches to the second RTC channel, the recorded audio input device allocation status is distributed to the client. For example, in an online conference room, when the server sends the RTC channel switching instruction to the client, it records the microphone status of the current conference room, so that after the RTC channel switching is completed, the original microphone user can still use the microphone, which minimizes the impact on the user.

[0095] Step 4: The client obtains information of the second RTC channel according to the channel switching instruction, saves a snapshot of the virtual space, switches to the second RTC channel, and releases the first RTC channel.

[0096] The internal processing logic of the client receiving the switching instruction is as follows:

[0097] After receiving the switch command, the authentication logic is re-executed to obtain the second RTC channel attributes. When the second RTC channel is consistent with the current user's local channel, the RTC token is obtained. If they are inconsistent, the switch operation is performed. The current background music is recorded and the snapshot information of the music playback progress is recorded. In the recovery phase, the music can be located at which position to start playing, so that users can have a better senseless experience. Start the second RTC channel engine, so that users can join the new RTC channel for interactive communication. Finally, release (destroy) the old RTC channel engine.

[0098] Step 5: The client uses the virtual space snapshot to restore the virtual space to the state before switching the RTC channel. The microphone user sets the anchor identity to push the stream to ensure that the original microphone user can still use the microphone. If there is music, resume the music playback progress and play the music normally.

[0099] The present embodiment shows two RTC channels. Those skilled in the art will appreciate that the number of RTC channels may be set to a number greater than two to better ensure high availability of online communication. The specific switching method is as described in the present embodiment.

[0100] Embodiment 2

[0101] This embodiment is a high-availability system for multi-modal real-time audio and video communication based on RTC. The system is used to implement the high-availability method as described in Example 1, including a client and a server. The client uses multiple RTC channels to connect to the server; the client further includes a status monitoring module and a first channel switching module, and the server further includes a service quality monitoring module and a second channel switching module;

[0102] The status monitoring module monitors the network quality in real time, obtains login quality, stream pulling quality, stream pushing quality, network indicators, audio quality and video quality, and reports the monitoring data to the server through the heartbeat packet;

[0103] The first channel switching module performs an RTC channel switching operation according to the RTC channel switching instruction sent by the server, saves a virtual space snapshot, starts a second RTC channel, adds the user to the second RTC channel, and uses the virtual space snapshot to restore the virtual space to a state before switching the RTC channel;

[0104] The service quality monitoring module server analyzes and determines the monitoring data according to the set network quality determination rules, and marks the first RTC channel as available;

[0105] When the first RTC channel is marked as unavailable, the second channel switching module marks the corresponding virtual space, modifies the virtual space to correspond to the second RTC channel, and sends an RTC channel switching instruction to the client.

[0106] The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A high availability method for multi-modal real-time audio and video communication based on RTC, characterized in that: The following steps are involved: In the online communication system, the client monitors the network quality in real time and reports the monitoring data to the server through the heartbeat packet; the dimensions of the network quality monitoring include login quality, stream pulling quality, stream pushing quality, network indicators, audio quality and video quality; The server analyzes and determines the monitoring data according to the set network quality judgment rules, and marks the first RTC channel as available; In response to the first RTC channel status being marked as unavailable, the server marks the corresponding virtual space, modifies the virtual space to correspond to the second RTC channel, and sends an RTC channel switching instruction to the client; The client obtains information of the second RTC channel according to the channel switching instruction, saves a snapshot of the virtual space, switches to the second RTC channel, and releases the first RTC channel; The client uses the virtual space snapshot to restore the virtual space to the state before switching the RTC channel.

2. The high availability method for multi-modal real-time audio and video communication based on RTC according to claim 1 is characterized in that: The login quality includes login success rate and login time; The method for calculating the login success rate is as follows: a user attempts to log in to a specified virtual space, and the parameters of the login method include a virtual space ID, user information, and virtual space configuration; after the login request is processed, the system performs a callback process, and records the number of login successes or failures according to the callback parameters; the login success rate is calculated according to the number of login successes and login failures; The method for calculating the login time consumption is as follows: before calling the login method, a first timestamp is recorded as the login start time; In the callback of the login result, the second timestamp is recorded as the time when the login ends; the login time is the difference between the first timestamp and the second timestamp.

3. The high availability method for multi-modal real-time audio and video communication based on RTC according to claim 1, characterized in that: The evaluation index of the pulling quality is the pulling success rate, and the calculation method of the pulling success rate is as follows: Each time a user requests to pull a stream, it is recorded as a stream pulling attempt; Monitor the stream pulling status in the stream pulling status callback and determine whether the stream pulling is successful based on the callback parameters; If the stream is not successfully pulled within the custom stream pulling timer, this attempt will be considered a failure; Maintain two counters to record the number of successful and failed stream pulling. The success rate of pulling the stream is calculated based on the number of successful pulling and the number of failed pulling.

4. The high availability method for multi-modal real-time audio and video communication based on RTC according to claim 1, characterized in that: The evaluation index of the streaming quality is the streaming success rate, and the calculation method of the streaming success rate is as follows: Each time a user requests to push a stream, it is recorded as a push attempt. Monitor the streaming status in the streaming status callback and determine whether the streaming is successful based on the callback parameters; If the push attempt fails within the custom push timer, the attempt will be considered a failure. Maintain two counters to record the number of successful and failed pushes. The streaming success rate is calculated based on the number of successful streaming and the number of failed streaming.

5. The high availability method for multi-modal real-time audio and video communication based on RTC according to claim 1, characterized in that: The network indicators include network delay and packet loss rate; The network delay calculation method is as follows: defining a round trip delay, where the round trip delay represents the round trip time from the streaming end to the server; recording the current round trip delay each time the network quality function is called; The packet loss rate is calculated by directly obtaining the current packet loss rate each time the network quality function is called.

6. The high availability method for multi-modal real-time audio and video communication based on RTC according to claim 1, characterized in that: The evaluation indicators of the audio quality and video quality include first frame time, cumulative freeze rate and cumulative freeze duration; The calculation method of the first frame time consumption is: when calling the method for starting to pull the stream, record the third timestamp as the time when the stream is pulled; When the first frame of audio or video data arrives, the callback for receiving the first frame is triggered, and the fourth timestamp is recorded as the time when the first frame was received; The difference between the third timestamp and the fourth timestamp is taken as the first frame time consumption; The method for calculating the accumulated freeze duration is as follows: in the media event callback, monitoring the audio or video freeze event, recording the freeze start event and the freeze end event, and calculating the single freeze duration; At the end of each freeze event, the freeze duration is added to the cumulative freeze duration; The cumulative jam rate is calculated as follows: when starting to pull the stream, the start time is recorded, and the total stream pulling time is calculated, and the total stream pulling time is updated in each network quality function callback; At the end of each streaming, the current cumulative freeze duration is obtained, and the ratio of the current cumulative freeze duration to the current total streaming duration is taken as the cumulative freeze rate.

7. The high availability method for multi-modal real-time audio and video communication based on RTC according to claim 1, characterized in that: The method for the server to analyze and determine the monitoring data according to the set network quality determination rules is specifically as follows: Calculate an average freeze rate of users in the virtual space within a set first time interval, and if the average freeze rate exceeds a set first threshold, set the freeze rate status to not meeting the standard; Calculate the average packet loss rate of users in the set first time interval, and if the average packet loss rate exceeds the set second threshold, set the packet loss rate status to not meeting the standard; Calculate the average user delay in the set first time interval, and if the average delay exceeds a set third threshold, set the average delay status to not meeting the standard; If the number of times that the jamming rate does not meet the standard and / or the packet loss rate does not meet the standard and / or the average delay does not meet the standard exceeds the set number of times within the set second time interval, it is determined that the corresponding RTC channel is unavailable.

8. The high availability method for multi-modal real-time audio and video communication based on RTC according to claim 7, characterized in that: After determining that the corresponding RTC channel is unavailable, the server also performs new RTC detection and volume expansion operations; The new RTC detection method is as follows: select the virtual space with the lowest number of users in the room as the detection space; issue an instruction to notify the room owner to actively detect the freeze rate, packet loss rate and average delay when joining the virtual space based on the new RTC creation engine; if the new RTC detection passes, it will be gradually increased; The method of increasing the volume is as follows: gradually increasing the volume at a preset first ratio, continuously performing quality detection of the jamming rate, packet loss rate and average delay during the increasing volume, and if the quality detection passes, gradually continuing to increase the volume until the total proportion of the increased volume reaches a preset second ratio.

9. The high availability method for multi-modal real-time audio and video communication based on RTC according to claim 1, characterized in that: When the server sends the RTC channel switching instruction to the client, the server also records the audio input device allocation state in the current virtual space; after the client switches to the second RTC channel, the recorded audio input device allocation state is distributed to the client.

10. A high-availability system for multi-modal real-time audio and video communication based on RTC, characterized in that: The system is used to implement the high availability method according to any one of claims 1 to 9, comprising a client and a server, wherein the client is connected to the server using multiple RTC channels; the client further comprises a status monitoring module and a first channel switching module, and the server further comprises a service quality monitoring module and a second channel switching module; The status monitoring module monitors the network quality in real time, obtains login quality, stream pulling quality, stream pushing quality, network indicators, audio quality and video quality, and reports the monitoring data to the server through the heartbeat packet; The first channel switching module performs an RTC channel switching operation according to the RTC channel switching instruction sent by the server, saves a virtual space snapshot, starts a second RTC channel, adds the user to the second RTC channel, and uses the virtual space snapshot to restore the virtual space to a state before switching the RTC channel; The service quality monitoring module server analyzes and determines the monitoring data according to the set network quality determination rules, and marks the first RTC channel as available; When the first RTC channel is marked as unavailable, the second channel switching module marks the corresponding virtual space, modifies the virtual space to correspond to the second RTC channel, and sends an RTC channel switching instruction to the client.

Images