Audio and video communication method and device and electronic equipment

Abstract

The invention provides an audio and video communication method and device and electronic equipment. The audio and video communication method comprises the steps that a sending side sends RTP data of audio and video to a receiving side; the RTP extension is marked with a unified number, and the sending side records the sending time of the RTP data; the receiving side receives the RTP data, analyzes the RTP extension of the RTP data to obtain a unified number, records the receiving time of the RTP data and sends the receiving time to the sending side through the RTCP; the sending side predicts the network bandwidth based on the sending time, the receiving time and the size of the RTP data; and if the network bandwidth is smaller than the minimum sum of the occupation of the audio bandwidth and the video bandwidth, the sending side enters an audio guarantee state, sends the audio data and stops sending the video data. The network condition can be detected, and when the network state is not good, the video is automatically closed to enter an audio guarantee state; the network state is automatically detected in the audio guarantee state, and the video is automatically opened at a proper time, so that the use comfort of a user is improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to an audio and video communication method, device and electronic equipment. Background Art

[0002] Currently, real-time audio and video communication is widely used across various industries, providing significant convenience for people's work and daily lives. In audio and video communication, users primarily communicate via voice, with video used to enhance user comfort. Therefore, audio is generally more important than video. When network conditions deteriorate, audio and video often compete for limited bandwidth, resulting in a lack of smooth transmission for both. In these situations, users expect audio communication to be prioritized to meet basic communication needs; when network conditions improve, audio and video communication will automatically resume, enhancing user comfort.

[0003] However, if users manually disable video while retaining audio, while this ensures basic audio communication, it's not quick or convenient. With only audio retained, users won't know when network conditions have improved. If they enable video before the network has improved, both audio and video quality will be significantly degraded. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide an audio and video communication method, device and electronic device that can detect network conditions, automatically turn off the video and enter the audio protection state when the network status is not good; automatically detect the network status in the audio protection state, and automatically turn on the video at an appropriate time to improve user comfort.

[0005] In a first aspect, an embodiment of the present invention provides an audio and video communication method, the method comprising: a sending side sends RTP data of audio and video to a receiving side; wherein, the RTP extension of the RTP data is marked with a unified number, and the sending side records the sending time of the RTP data; the receiving side receives the RTP data, parses the RTP extension of the RTP data to obtain a unified number, records the receiving time of the RTP data and sends it to the sending side via RTCP; the sending side predicts the network bandwidth between the sending side and the receiving side based on the sending time, the receiving time and the size of the RTP data; if the network bandwidth is less than the sum of the minimum audio bandwidth and the video bandwidth occupancy, the sending side enters an audio guarantee state, sends audio data and stops sending video data.

[0006] In an optional embodiment of the present application, after the step that the sending side enters the audio guarantee state, sends audio data, and stops sending video data, the method further includes: the sending side periodically sends RTP probe data to the receiving side; the sending side predicts the network bandwidth based on the RTP probe data; if the network bandwidth is greater than or equal to the sum of the minimum audio bandwidth and video bandwidth occupancy, the sending side enters the audio-video communication state, continues to send audio data, and resumes sending video data.

[0007] In an optional embodiment of the present application, the step that the sending side sends RTP data of audio and video to the receiving side includes: the sending side sends variable bitrate audio data and video data based on the RTP protocol to the receiving side; wherein, the bitrate range of the audio data is [amin, amax], and the bitrate range of the video data is [vmin, vmax].

[0008] In an optional embodiment of the present application, the step of recording the reception time of the RTP data and sending it to the sending side via RTCP includes: the receiving side packs the reception times of multiple RTP data into an RTCP data packet and sends the RTCP data packet to the sending side. [

[0009] In an optional embodiment of the present application, the step that the sending side predicts the network bandwidth between the sending side and the receiving side based on the sending time, reception time, and size of the RTP data includes: the sending side analyzes the RTCP data packet to obtain the unified number and reception time of the RTP data; the sending side determines the size of the RTP data based on the unified number, and predicts the network bandwidth between the sending side and the receiving side based on the sending time, reception time, and size of the RTP data.

[0010] In an optional embodiment of the present application, where bhat is the network bandwidth predicted by the sending side, the method further includes: if bhat < amax + vmax, gradually lower the bitrates of the audio data and video data; wherein, amax is the highest audio bandwidth, and vmax is the highest video bandwidth; if the network bandwidth is less than the sum of the minimum audio bandwidth and video bandwidth occupancy, the step that the sending side enters the audio guarantee state, sends audio data, and stops sending video data includes: if bhat < amin + vmin, the sending side enters the audio guarantee state, initializes the actual network sending bandwidth and the predicted network bandwidth to amin, sends audio data, and stops sending video data; wherein, amin is the lowest audio bandwidth, and vmin is the lowest video bandwidth.

[0011] In an optional embodiment of the present application, the above-mentioned step of periodically sending RTP detection data from the sending side to the receiving side includes: the sending side sends p RTP detection packets and q audio packets to the receiving side within one period based on the sending rate x0; wherein, p≥1, q≥1, x0=bhat×r, bhat is the network bandwidth, and r is the preset multiplier.

[0012] In an optional embodiment of the present application, the above-mentioned step of predicting the network bandwidth based on the RTP detection data on the sending side includes: the receiving side packages the reception time of multiple RTP detection data into an RTCP data packet, and sends the RTCP data packet to the sending side; the sending side determines the reception time interval of p RTP detection packets and q audio packets on the receiving side based on the RTCP data packet; the sending side divides the total size of the p RTP detection packets and q audio packets by the reception time interval as the detection rate of the receiving side, takes the smaller value of the detection rate and the sending rate as the target rate, and changes the network bandwidth based on the target rate.

[0013] In a second aspect, an embodiment of the present invention further provides an audio and video communication device, which includes: an RTP data sending module, which is used for the sending side to send RTP data of audio and video to the receiving side; wherein the RTP extension of the RTP data is marked with a unified number, and the sending side records the sending time of the RTP data; an RTP data receiving module, which is used for the receiving side to receive RTP data, parse the RTP extension of the RTP data to obtain a unified number, record the receiving time of the RTP data and send it to the sending side via RTCP; a network bandwidth prediction module, which is used for the sending side to predict the network bandwidth between the sending side and the receiving side based on the sending time, receiving time and the size of the RTP data; an audio guarantee status module, which is used for the sending side to enter the audio guarantee state, send audio data and stop sending video data if the network bandwidth is less than the sum of the minimum audio bandwidth and video bandwidth occupancy.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the above-mentioned audio and video communication method.

[0015] The embodiments of the present invention bring the following beneficial effects:

[0016] Embodiments of the present invention provide an audio and video communication method, device, and electronic device, wherein a sending side sends audio and video RTP data to a receiving side; wherein the RTP extension of the RTP data is marked with a unified number, and the sending side records the sending time of the RTP data; the receiving side receives the RTP data, parses the RTP extension of the RTP data to obtain a unified number, records the receiving time of the RTP data, and sends it to the sending side via RTCP; the sending side predicts the network bandwidth between the sending side and the receiving side based on the sending time, receiving time, and the size of the RTP data; if the network bandwidth is less than the sum of the minimum audio bandwidth and video bandwidth occupancy, the sending side enters an audio guarantee state, sends audio data, and stops sending video data. In this method, the network status can be detected. When the network status is poor, the video is automatically turned off and the audio guarantee state is entered; in the audio guarantee state, the network status is automatically detected and the video is automatically turned on at the appropriate time, thereby improving user comfort.

[0017] Other features and advantages of the present disclosure will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by practicing the above-mentioned technology of the present disclosure.

[0018] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of an audio and video communication method provided by an embodiment of the present invention;

[0021] Figure 2 A flowchart of another audio and video communication method provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a specific example of real-time audio and video communication with audio assurance provided by an embodiment of the present invention;

[0023] Figure 4 A schematic structural diagram of an audio and video communication device provided in an embodiment of the present invention;

[0024] Figure 5 A schematic structural diagram of another audio and video communication device provided by an embodiment of the present invention;

[0025] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0027] Currently, real-time audio and video communication technology uses RTP (Real-Time Transport Protocol) / RTCP (RTP Control Protocol) as the commonly used transmission protocol, which has the ability to transmit audio and video simultaneously. Users can communicate with other remote users in full-duplex mode.

[0028] Although the RTP / RTCP protocols have functions such as packet loss detection and bit rate estimation, they are unable to detect and calculate the network bandwidth available for video transmission after video transmission is turned off.

[0029] When network conditions deteriorate, most communication software will experience audio and video freezes, which will not resolve until the network condition recovers. A small number of software will reduce audio and video quality, and even suggest that users manually turn off the video to ensure communication quality, but will not automatically restore the video when the network condition recovers. Based on this, the embodiments of the present invention provide an audio and video communication method, device, and electronic device, specifically providing a real-time audio and video communication method with audio protection, which can detect network conditions and automatically turn off the video when the network condition is poor and enter the audio protection state; automatically detect the network condition in the audio protection state and automatically turn on the video at the appropriate time to improve user comfort.

[0030] Existing real-time audio and video communication technologies have the following main defects:

[0031] 1. When network communication deteriorates, the quality of real-time audio and video will deteriorate, affecting users' real-time communication. Some software users can manually turn off the video, but this is not convenient.

[0032] 2. When the network condition improves, audio communication cannot automatically resume video. Users cannot sense whether the network condition has improved. Opening video during a poor network condition will degrade both audio and video network quality.

[0033] To facilitate understanding of this embodiment, an audio and video communication method disclosed in an embodiment of the present invention is first introduced in detail.

[0034] Example 1:

[0035] The embodiment of the present invention provides an audio and video communication method, see Figure 1 The flowchart of an audio and video communication method shown in FIG. 1 includes the following steps:

[0036] Step S102: The sending side sends RTP data of audio and video to the receiving side. The RTP extension of the RTP data is marked with a unified number, and the sending side records the sending time of the RTP data.

[0037] In this embodiment, the sending side can uniformly send the RTP data of audio and video, mark a unified number on the RTP extension, and record the sending time locally on the sending side.

[0038] Step S104: The receiving side receives the RTP data, parses the RTP extension of the RTP data to obtain a unified number, records the receiving time of the RTP data, and sends it to the sending side through RTCP.

[0039] In this embodiment, after receiving the RTP data, the receiving side can parse the unified number in the RTP number extension and feed it back to the sending side through RTCP.

[0040] Step S106: The sending side predicts the network bandwidth between the sending side and the receiving side based on the sending time, the receiving time and the size of the RTP data.

[0041] After receiving the RTCP feedback from the receiving side, the sending side in this embodiment can predict the current network bandwidth and evaluate the network status by comparing the sending time and receiving time of the RTP data and the size of the RTP time.

[0042] Step S108: If the network bandwidth is less than the sum of the minimum audio bandwidth and video bandwidth, the sending side enters the audio guarantee state, sends audio data and stops sending video data.

[0043] When the network deteriorates and the network bandwidth is less than the sum of the minimum audio bandwidth and video bandwidth, the sending side can enter the audio guarantee state, send audio data and stop sending video data, and reduce bandwidth consumption by actively shutting down video transmission.

[0044] In some embodiments, the sending side periodically sends RTP detection data to the receiving side; the sending side predicts the network bandwidth based on the RTP detection data; if the network bandwidth is greater than or equal to the sum of the minimum audio bandwidth and video bandwidth occupancy, the sending side enters the audio and video communication state, continues to send audio data and resends video data.

[0045] After entering the audio guarantee state, the sending side periodically sends a small amount of RTP probe data to predict whether the network bandwidth has changed. The predicted bandwidth is not limited by the actual bandwidth used for sending audio. When the predicted network bandwidth is greater than or equal to the sum of the minimum audio bandwidth and video bandwidth, the sending side resumes sending audio data and restarts sending video data, restoring normal audio and video communication.

[0046] An embodiment of the present invention provides an audio and video communication method, in which a sending side sends RTP data of audio and video to a receiving side; wherein the RTP extension of the RTP data is marked with a unified number, and the sending side records the sending time of the RTP data; the receiving side receives the RTP data, parses the RTP extension of the RTP data to obtain a unified number, records the receiving time of the RTP data, and sends it to the sending side via RTCP; the sending side predicts the network bandwidth between the sending side and the receiving side based on the sending time, receiving time, and the size of the RTP data; if the network bandwidth is less than the sum of the minimum audio bandwidth and video bandwidth occupancy, the sending side enters an audio guarantee state, sends audio data, and stops sending video data. In this method, the network status can be detected. When the network status is poor, the video is automatically turned off and the audio guarantee state is entered; in the audio guarantee state, the network status is automatically detected and the video is automatically turned on at the appropriate time, thereby improving user comfort.

[0047] Example 2:

[0048] This embodiment provides another audio and video communication method, which builds upon the previous embodiment and focuses on the specific implementation of real-time audio and video communication with audio support. To facilitate understanding for professionals, this embodiment uses specific user terminal operating systems and parameters. Changing these parameters does not affect the method provided in this embodiment.

[0049] In practice, audio and video communications are generally full-duplex, meaning both parties can receive the remote audio and video, and the audio and video streams sent in both directions are independent. In this example, for simplicity, it is assumed that terminal A is the transmitter and only sends audio and video, while terminal B is the receiver and only receives audio and video.

[0050] Based on the above description, you can refer to Figure 2 The flowchart of another audio and video communication method shown in FIG. 1 includes the following steps:

[0051] Step S202: The sending side sends RTP data of audio and video to the receiving side. The RTP extension of the RTP data is marked with a unified number, and the sending side records the sending time of the RTP data.

[0052] In some embodiments, the sending side sends audio data and video data with variable bit rates based on the RTP protocol to the receiving side; wherein the bit rate range of the audio data is [amin, amax], and the bit rate range of the video data is [vmin, vmax].

[0053] In this embodiment, terminal A can use the RTP protocol to send audio and video externally, and use RTP extensions to uniformly number the data (including retransmission data, redundant data, and detection data required for audio and video), and record the time when each RTP number is sent externally. The method provided by this embodiment does not conflict with existing audio and video transmission technologies. Assume that the audio and video data sent by terminal A can both use variable bit rates, where the audio bit rate range is [amin, amax] and the video bit rate range is [vmin, vmax]. Fixed bit rate can be regarded as a special case of variable bit rate.

[0054] In step S204, the receiving side receives the RTP data, parses the RTP extension of the RTP data to obtain a unified number, records the receiving time of the RTP data, and sends it to the sending side through RTCP.

[0055] In some embodiments, the receiving side packages the receiving time of multiple RTP data into an RTCP data packet and sends the RTCP data packet to the sending side.

[0056] In this embodiment, after receiving the RTP audio and video data, terminal B records the RTP reception time and parses the unified number in the RTP extension, and feeds the number and the corresponding reception time back to terminal A via RTCP. To reduce the number of interactions, terminal B packages the reception times of several RTPs into one RTCP packet (i.e., RTCP data packet) and feeds it back to terminal A.

[0057] Step S206: The sending side predicts the network bandwidth between the sending side and the receiving side based on the sending time, the receiving time and the size of the RTP data.

[0058] In some embodiments, the sending side parses the RTCP data packet to obtain a unified number and receiving time of the RTP data; the sending side determines the size of the RTP data based on the unified number, and predicts the network bandwidth between the sending side and the receiving side based on the sending time, receiving time and the size of the RTP data.

[0059] In some embodiments, after receiving the RTCP packet, terminal A can parse out the unified number of RTP packets and the transmission time recorded locally at terminal A. Based on the size of the RTP packet and the difference between the reception time and the transmission time, it can be used to estimate the congestion state of the network and predict the network bandwidth. Among them, in this embodiment, a corresponding relationship can be preset, and by substituting the transmission time, reception time, and size of the RTP data into the above corresponding relationship, the network bandwidth can be predicted.

[0060] Step S208, if the network bandwidth is less than the sum of the minimum audio bandwidth and video bandwidth occupancy, the sending side enters the audio guarantee state, sends audio data, and stops sending video data.

[0061] In some embodiments, if bhat < amax + vmax, gradually lower the bitrates of the audio data and video data; where bhat is the network bandwidth predicted by the sending side, amax is the highest audio bandwidth, and vmax is the highest video bandwidth; if bhat < amin + vmin, the sending side enters the audio guarantee state, initializes the actual network sending bandwidth and the predicted network bandwidth to amin, sends audio data, and stops sending video data; where amin is the lowest audio bandwidth and vmin is the lowest video bandwidth.

[0062] When the network bandwidth bhat calculated by terminal A is less than amax + vmax, gradually lower the bitrates of the audio and video. When the network bandwidth bhat is less than amin + vmin, close the video transmission. At this time, terminal A enters the audio guarantee sending state, and both the actual network sending bandwidth b and the predicted value bhat are initialized to amin. In the subsequent execution process of this embodiment, it is allowed that bhat is higher than the network sending bandwidth b, and the network bandwidth prediction is not limited by the actual sending bitrate.

[0063] Step S210, the sending side periodically sends RTP probe data to the receiving side; the sending side predicts the network bandwidth based on the RTP probe data; if the network bandwidth is greater than or equal to the sum of the minimum audio bandwidth and video bandwidth occupancy, the sending side enters the audio-video communication state, continues to send audio data, and resumes sending video data.

[0064] In some embodiments, within one period, the sending side sends p RTP probe packets and q audio packets to the receiving side based on the sending rate x0; where p ≥ 1, q ≥ 1, x0 = bhat × r, bhat is the network bandwidth, and r is a preset magnification factor.

[0065] In this embodiment, in the audio state guarantee state, terminal A periodically sends detection data, continuously sends p RTP detection packets. If q audio packets are sent in one period, they are also recorded. Calculate the transmission time interval t0 between the first and the last of this group of RTP packets. Divide the total size of (p + q) RTP detection packets by the time t0 to obtain the transmission rate x0, which takes a value of r times the current network bandwidth bhat. The detection time t0 is generally set to about 10 ms to minimize the occupancy of the network bandwidth by the detection behavior. The p RTP detection packets do not need to carry audio and video information, but use the RTP padding mechanism to carry empty information. The retransmission packets or redundant packets of audio and video can also be used as detection packets to reduce the transmission of additional audio and video data.

[0066] In some embodiments, the receiving side packs the reception times of multiple RTP detection data into RTCP data packets and sends the RTCP data packets to the sending side; the sending side determines the reception time interval of p RTP detection packets and q audio packets on the receiving side based on the RTCP data packets; the sending side takes the value obtained by dividing the total size of p RTP detection packets and q audio packets by the reception time interval as the detection rate on the receiving side, takes the smaller value of the detection rate and the transmission rate as the target rate, and changes the network bandwidth based on the target rate.

[0067] Terminal B in this embodiment does not need to distinguish RTP detection packets from other RTP packets, only needs to record the RTP reception time, identify the unified number in the RTP extension, and then feedback it to terminal A through RTCP.

[0068] After receiving the RTCP feedback, terminal A in this embodiment calculates the reception time interval t1 of p RTP detection packets and q audio packets on terminal B as the denominator, and takes the total size of (p + q) RTPs as the numerator to calculate the detection rate x1 on the receiving side. Take the smaller value x of x0 and x1 as the detection rate for this time and update bhat.

[0069] In the audio guarantee stage of this embodiment, if bhat < amin + vmin, maintain the state of only sending audio. At this time, the audio encoder is consistent with bhat within the range of [amin, amax]. If bhat > amax, the code rate of the audio encoding sent out is fixed at amax. If terminal A predicts that the bandwidth is greater than or equal to amin + vmin, automatically resume audio and video transmission.

[0070] Embodiment 3:

[0071] This embodiment provides another audio and video communication method, which is implemented based on the above embodiment, and focuses on describing a specific example of real-time audio and video communication with audio guarantee. It can be seen Figure 3Schematic diagram of a specific example of real-time audio and video communication with audio guarantee.

[0072] As Figure 3 shown, assume that the audio coding bitrate range is [amin, amax], and the video coding bitrate range is [vmin, vmax], where amax < vmin. For the sake of simplicity in explanation, this article only introduces the control process of the terminal A sending media stream to the terminal B. This method also applies to the terminal B sending media stream to the terminal A, and mainly includes the following steps:

[0073] 1. The terminal A sends audio and video media stream to the terminal B, where the RTP extension contains a unified number, and records the sending time of the RTP.

[0074] 2. The terminal B records the receiving time of the RTP, forms a group of the receiving times of the RTP within a period of time, and feeds it back to the terminal A through RTCP.

[0075] 3. The terminal A calculates and updates the network bandwidth bhat according to the size of the RTP, as well as the sending time difference and the receiving time difference. The terminal A adjusts the audio and video encoder according to bhat and sends out audio and video data.

[0076] 4. If amin + vmin <= bhat < amax + vmax, lower the audio and video bitrate; if bhat < amin + vmin, the terminal A closes the video sending, enters the audio guarantee state, and only sends audio.

[0077] 5. Every T seconds (for example, T is taken as 2 seconds) for the terminal A, take x0 = bhat × r, where x0 is the detection rate and r is the detection bitrate increase ratio (for example, taken as 1.1). Within the time t0 (t0 can be set to 5 - 20 ms, preferably 10 ms), if there are q audio packets within the time t0, take p RTP detection packets, and make the total size of the p RTP detection packets and the q audio packets divided by the time t0 equal to x0. The detection packets of the RTP can directly use the padding mechanism, do not need to rely on the encoder to generate data, and can also use the retransmitted data or redundant data of the audio.

[0078] 6. The terminal B does not need to distinguish between detection packets and audio packets, and still feeds back the time when the RTP arrives at the terminal B (i.e., the receiving time) in the RTCP manner.

[0079] 7. The terminal A calculates the receiving rate x1 of the p RTP detection packets and the q audio packets on the terminal B side according to the RTCP feedback; if there is packet loss in the network, then use the RTP statistical receiving rate that arrives at the terminal B. Take x = min(xo, x1), and update bhat with x.

[0080] 8. During the audio guarantee phase, if bhat < amin + vmin, the audio encoder is consistent with bhat within the range of [amin, amax]; if bhat > amax, the fixed transmission bitrate of the audio encoding is amax.

[0081] 9. If bhat >= amin + vmin, terminal A resumes audio and video transmission.

[0082] In summary, the real-time audio and video communication method for audio guarantee provided by the embodiments of the present invention mainly includes the following steps:

[0083] (1) During the audio guarantee phase, the terminal periodically sends RTP probe packets to detect the network condition. Each RTP probe packet has a short time and less data, minimizing the impact on audio guarantee communication.

[0084] (2) The terminal calculates the predicted transmission bitrate based on the information of the sending side and the feedback of the receiving side. The predicted bitrate is updated by detection and does not rely on the actual transmission bitrate of the audio. When the predicted bitrate is greater than the maximum value of the audio encoding, the audio encoding bitrate is sent according to the maximum value of the encoder; when the predicted bitrate is lower than the maximum value of the audio encoding, the audio encoding is sent according to the predicted bitrate.

[0085] (3) The terminal automatically controls the switching between sending audio only or sending audio and video simultaneously according to the predicted bitrate, without manual intervention by the user. This not only improves the user's comfort but also avoids a decline in communication quality caused by rashly opening the video during the audio guarantee phase.

[0086] In summary, the real-time audio and video communication method for audio guarantee provided by the embodiments of the present invention mainly has the following advantages:

[0087] (1) When the terminal conducts audio and video communication, if the network condition deteriorates, the program can automatically close the video to ensure the most basic audio communication. At the same time, it also reduces the trouble of manual operation by the user and improves the comfort of use.

[0088] (2) During the audio guarantee communication phase, the method of detecting network state changes will not cause a further decline in audio quality and does not rely on the maximum bitrate output by the audio encoder, having wide applicability.

[0089] (3) The terminal can switch from audio guarantee to audio and video communication state according to bitrate prediction, avoiding a decline in audio communication quality caused by switching in a poor network condition.

[0090] Embodiment 4:

[0091] Corresponding to the above method embodiment, the embodiments of the present invention provide an audio and video communication device. See Figure 4The structure diagram of an audio and video communication device shown in FIG. 1 includes:

[0092] The RTP data sending module 41 is used for sending the RTP data of audio and video from the sending side to the receiving side. The RTP extension of the RTP data is marked with a unified number, and the sending side records the sending time of the RTP data.

[0093] The RTP data receiving module 42 is used to receive RTP data at the receiving side, parse the RTP extension of the RTP data to obtain a unified number, record the reception time of the RTP data and send it to the sending side via RTCP;

[0094] The network bandwidth prediction module 43 is used for the sending side to predict the network bandwidth between the sending side and the receiving side based on the sending time, the receiving time and the size of the RTP data and send it to the sending side through RTCP;

[0095] The audio guarantee state module 44 is configured to, if the network bandwidth is less than the sum of the minimum audio bandwidth and the video bandwidth, cause the sending side to enter the audio guarantee state, send audio data, and stop sending video data.

[0096] An embodiment of the present invention provides an audio and video communication device, in which a sending side sends RTP data of audio and video to a receiving side; wherein the RTP extension of the RTP data is marked with a unified number, and the sending side records the sending time of the RTP data; the receiving side receives the RTP data, parses the RTP extension of the RTP data to obtain a unified number, records the receiving time of the RTP data, and sends it to the sending side via RTCP; the sending side predicts the network bandwidth between the sending side and the receiving side based on the sending time, receiving time, and the size of the RTP data; if the network bandwidth is less than the sum of the minimum audio bandwidth and video bandwidth occupancy, the sending side enters an audio guarantee state, sends audio data, and stops sending video data. In this method, the network status can be detected. When the network status is poor, the video is automatically turned off and the audio guarantee state is entered; in the audio guarantee state, the network status is automatically detected and the video is automatically turned on at the appropriate time, thereby improving user comfort.

[0097] See also Figure 5 The structural diagram of another audio and video communication device shown in the figure further includes: an audio and video communication status module 45, which is used for the sending side to periodically send RTP detection data to the receiving side; the sending side predicts the network bandwidth based on the RTP detection data; if the network bandwidth is greater than or equal to the sum of the minimum audio bandwidth and video bandwidth occupancy, the sending side enters the audio and video communication state, continues to send audio data and resends video data.

[0098] The above RTP data sending module is used to send variable bitrate audio data and video data based on the RTP protocol from the sending side to the receiving side; among them, the bitrate range of the audio data is [amin, amax], and the bitrate range of the video data is [vmin, vmax].

[0099] The above RTP data receiving module is used for the receiving side to pack the reception times of multiple RTP data into RTCP data packets and send the RTCP data packets to the sending side.

[0100] The above network bandwidth prediction module is used for the sending side to parse the RTCP data packet to obtain the unified number and reception time of the RTP data; the sending side determines the size of the RTP data based on the unified number, and predicts the network bandwidth between the sending side and the receiving side based on the sending time, reception time and the size of the RTP data.

[0101] The above bhat is the network bandwidth, and the device further includes: a bitrate reduction module, which is used to gradually reduce the bitrates of the audio data and video data if bhat < amax + vmax; where amax is the highest audio bandwidth and vmax is the highest video bandwidth; if the network bandwidth is less than the sum of the occupancies of the lowest audio bandwidth and video bandwidth, the sending side enters the audio guarantee state, and the steps of sending audio data and stopping sending video data include: if bhat < amin + vmin, the sending side enters the audio guarantee state, initializes the actual network sending bandwidth and the predicted network bandwidth to amin, sends audio data and stops sending video data; where amin is the lowest audio bandwidth and vmin is the lowest video bandwidth.

[0102] The above audio-video communication status module is used for the sending side to send p RTP probe packets and q audio packets to the receiving side within one cycle based on the sending rate x0; where p ≥ 1, q ≥ 1, x0 = bhat × r, bhat is the network bandwidth, and r is a preset magnification factor.

[0103] The above audio-video communication status module is used for the receiving side to pack the reception times of multiple RTP probe data into RTCP data packets and send the RTCP data packets to the sending side; the sending side determines the reception time interval of the p RTP probe packets and q audio packets on the receiving side based on the RTCP data packet; the sending side takes the value obtained by dividing the total size of the p RTP probe packets and q audio packets by the reception time interval as the probe rate of the receiving side, takes the smaller value of the probe rate and the sending rate as the target rate, and replaces the network bandwidth based on the target rate.

[0104] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the audio and video communication device described above can refer to the corresponding process in the embodiment of the aforementioned audio and video communication method, and will not be repeated here.

[0105] Embodiment 5:

[0106] The embodiment of the present invention further provides an electronic device for executing the above-mentioned audio and video communication method; Figure 6 The structure diagram of an electronic device shown in the figure includes a memory 100 and a processor 101, wherein the memory 100 is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor 101 to implement the above-mentioned audio and video communication method.

[0107] Furthermore, Figure 6 The electronic device shown further includes a bus 102 and a communication interface 103 , and the processor 101 , the communication interface 103 and the memory 100 are connected via the bus 102 .

[0108] The memory 100 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is achieved through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0109] The processor 101 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 101 or software instructions. The above processor 101 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as a random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or register. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100 and, in conjunction with its hardware, completes the steps of the method of the aforementioned embodiment.

[0110] An embodiment of the present invention also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned audio and video communication method. The specific implementation can be found in the method embodiment and will not be repeated here.

[0111] The computer program products of the audio and video communication methods, devices, and electronic devices provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the previous method embodiments. For specific implementation, please refer to the method embodiments and will not be repeated here.

[0112] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process of the system and / or device described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0113] In addition, in the description of the embodiments of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to mechanical connections or electrical connections; they may refer to direct connections or indirect connections through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0114] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0115] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0116] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An audio and video communication method, characterized in that: The method includes: Sending RTP data of audio and video from the sending side to the receiving side; wherein, there is a marked unified number on the RTP extension of the RTP data, and the sending side records the sending time of the RTP data; The receiving side receives the RTP data, parses the RTP extension of the RTP data to obtain the unified number, records the receiving time of the RTP data and sends it to the sending side through RTCP; The sending side predicts the network bandwidth between the sending side and the receiving side based on the sending time, the receiving time and the size of the RTP data; If the network bandwidth is less than the sum of the minimum occupied audio bandwidth and video bandwidth, the sending side enters the audio guarantee state, sends audio data and stops sending video data.

2. The method according to claim 1, characterized in that After the step that the sending side enters the audio guarantee state, sends audio data and stops sending video data, the method further includes: The sending side periodically sends RTP probe data to the receiving side; The sending side predicts the network bandwidth based on the RTP probe data; If the network bandwidth is greater than or equal to the sum of the minimum occupied audio bandwidth and video bandwidth, the sending side enters the audio-video communication state, continues to send the audio data and resends the video data.

3. The method according to claim 1, characterized in that The step of the sending side sending RTP data of audio and video to the receiving side includes: The sending side sends variable bit rate audio data and video data based on the RTP protocol to the receiving side; wherein, the bit rate range of the audio data is [amin, amax], and the bit rate range of the video data is [vmin, vmax].

4. The method according to claim 1, wherein The step of recording the receiving time of the RTP data and sending it to the sending side through RTCP includes: The receiving side packs the receiving times of multiple RTP data into an RTCP data packet and sends the RTCP data packet to the sending side.

5. The method according to claim 4, characterized in that The step of the sending side predicting the network bandwidth between the sending side and the receiving side based on the sending time, the receiving time and the size of the RTP data includes: The sending side parses the RTCP data packet to obtain the unified number and the receiving time of the RTP data; The sending side determines the size of the RTP data based on the unified number, and predicts the network bandwidth between the sending side and the receiving side based on the sending time, the receiving time and the size of the RTP data.

6. The method according to claim 3, characterized in that Let bhat be the network bandwidth predicted by the sending side through calculation, the method further includes: if bhat < amax + vmax, gradually lower the bit rates of the audio data and the video data; wherein, amax is the highest audio bandwidth and vmax is the highest video bandwidth; If the network bandwidth is less than the sum of the minimum audio bandwidth and video bandwidth occupancy, the sending side enters the audio guarantee state. The steps of sending audio data and stopping sending video data include: If bhat < amin + vmin, the sending side enters the audio guarantee state, initializes the actual network sending bandwidth and the predicted network bandwidth to amin, sends audio data and stops sending video data; where amin is the minimum audio bandwidth and vmin is the minimum video bandwidth.

7. The method according to claim 2, characterized in that The step of the sending side periodically sending RTP probe data to the receiving side includes: The sending side sends p RTP probe packets and q audio packets to the receiving side within one period based on the sending rate x0; where p ≥ 1, q ≥ 1, x0 = bhat × r, bhat is the network bandwidth, and r is a preset magnification factor.

8. The method according to claim 7, characterized in that The step of the sending side predicting the network bandwidth based on the RTP probe data includes: The receiving side packs the reception times of multiple RTP probe data into an RTCP data packet and sends the RTCP data packet to the sending side; The sending side determines the reception time intervals of p RTP probe packets and q audio packets at the receiving side based on the RTCP data packet; The sending side takes the value obtained by dividing the total size of p RTP probe packets and q audio packets by the reception time interval as the probe rate at the receiving side, takes the smaller value of the probe rate and the sending rate as the target rate, and replaces the network bandwidth based on the target rate.

9. An audio and video communication device, characterized in that: The device includes: An RTP data sending module, configured to send RTP data of audio and video from the sending side to the receiving side; where there is a marked unified number on the RTP extension of the RTP data, and the sending side records the sending time of the RTP data; An RTP data receiving module, configured to receive the RTP data by the receiving side, parse the RTP extension of the RTP data to obtain the unified number, record the reception time of the RTP data and send it to the sending side through RTCP; A network bandwidth prediction module, configured to predict the network bandwidth between the sending side and the receiving side by the sending side based on the sending time, the reception time and the size of the RTP data; An audio guarantee state module, configured to if the network bandwidth is less than the sum of the minimum audio bandwidth and video bandwidth occupancy, the sending side enters the audio guarantee state, sends audio data and stops sending video data.

10. An electronic device, characterized in that: It includes a processor and a memory, the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the audio-video communication method according to any one of claims 1 to 8.

Images