Audio and video transmission method and system for online meeting, electronic device and storage medium

Abstract

The application provides an online conference audio and video transmission method and system, electronic equipment and a storage medium, and relates to the technical field of audio and video data transmission. The audio and video transmission method comprises the following steps: configuring the transmission priority of the audio and video data of the host terminal according to a preset priority rule; predicting the network prediction state of the host terminal in a preset future time period based on a network state prediction model; in the case that the network prediction state is a weak network state, determining the transmission channel corresponding to the transmission priority and the dynamic coding strategy corresponding to the transmission priority; and synchronously transmitting the audio and video data corresponding to the transmission priority to the participant terminal through the corresponding transmission channel based on the dynamic coding strategy. The application can guarantee the accurate transmission of key information in the audio and video data and solve the problem of key information loss in the weak network state. In addition, the application can guarantee the reliability and real-time performance of the audio and video data transmission through multi-path transmission.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio and video data transmission, in particular to an online conference audio and video transmission method and system, an electronic device and a storage medium. BACKGROUND

[0002] With the rapid development of Internet technology and communication technology, real-time audio and video communication, online education, remote conference and other application scenarios are increasingly popular.

[0003] However, the present inventors have found that in the case of unstable network environment (such as weak network environment), the conventional audio and video transmission strategy often cannot effectively guarantee the transmission quality of key audio and video information, resulting in loss of key information.

[0004] For example, the present inventors have found that in the case of unstable network environment, the conventional audio and video transmission strategy (such as weak network degradation strategy) has problems such as indiscriminate degradation, lack of priority division and insufficient redundancy mechanism.

[0005] The content of the background section is only the knowledge of the present inventors and does not necessarily represent the prior art in the field. SUMMARY

[0006] The present application provides an online conference audio and video transmission method and system, an electronic device and a storage medium, aiming to solve the problems of indiscriminate degradation, lack of priority division and insufficient redundancy mechanism of the conventional audio and video transmission strategy in the case of unstable network environment.

[0007] According to an aspect of the present application, the present application provides an online conference audio and video transmission method, comprising: configuring the transmission priority of the audio and video data of the host end according to a preset priority rule; predicting the network prediction state of the host end in a preset future time period based on a network state prediction model; in the case of weak network state of the network prediction state, determining the transmission channel corresponding to the transmission priority and the dynamic encoding strategy corresponding to the transmission priority; and synchronously transmitting the audio and video data corresponding to the transmission priority to the participant end through the corresponding transmission channel based on the dynamic encoding strategy.

[0008] According to some embodiments of the present application, predicting the network prediction state of the host end in a preset future time period based on a network state prediction model comprises: determining historical network data information of a preset historical time period; predicting the prediction bandwidth information of the preset future time period based on the network state prediction model and the historical network data information; and in the case that the prediction bandwidth information is less than a preset threshold, determining that the network prediction state is a weak network state.

[0009] According to some embodiments of the present application, the transmitting the audio and video data corresponding to the transmission priority to the participant terminal through the corresponding transmission channel based on the dynamic coding strategy comprises: dynamically adjusting the forward error correction redundancy of the audio and video data based on the predicted bandwidth information.

[0010] According to some embodiments of the present application, the audio and video transmission method further comprises: dynamically determining a visual attention area of the user of the participant terminal based on visual tracking; and in the case that the network prediction state is a weak network state, improving the display resolution of the visual attention area to a preset resolution.

[0011] According to some embodiments of the present application, the audio and video transmission method further comprises: in the case that the audio and video data received by the participant terminal is detected to be lost frames, determining the lost frame data based on the front frame data and the rear frame data of the audio and video data received by the participant terminal.

[0012] According to some embodiments of the present application, the audio and video transmission method further comprises: in the case that the delay of the audio data and the video data in the audio and video data is greater than a preset delay threshold, dynamically stretching the audio data or compressing or speeding up the video data based on the audio waveform, so that the audio data and the video data are synchronously transmitted.

[0013] According to another aspect of the present application, the present application further provides an audio and video transmission method for online conference, comprising: configuring the transmission priority of the audio and video data of the participant terminal according to a preset priority rule; predicting the network prediction state of the participant terminal in a preset future time period based on a network state prediction model; in the case that the network prediction state is a weak network state, determining the transmission channel corresponding to the transmission priority and the dynamic coding strategy corresponding to the transmission priority; and transmitting the audio and video data corresponding to the transmission priority to the participant terminal through the corresponding transmission channel based on the dynamic coding strategy.

[0014] According to another aspect of the present application, the present application further provides an audio and video transmission system for online conference, comprising a content classification determination module, a network state prediction module, a channel and strategy control module, and an audio and video data transmission module. The content classification determination module configures the transmission priority of the audio and video data of the participant terminal according to a preset priority rule; the network state prediction module predicts the network prediction state of the participant terminal in a preset future time period based on a network state prediction model; the channel and strategy control module determines the transmission channel corresponding to the transmission priority and the dynamic coding strategy corresponding to the transmission priority in the case that the network prediction state is a weak network state; and the audio and video data transmission module transmits the audio and video data corresponding to the transmission priority to the participant terminal through the corresponding transmission channel based on the dynamic coding strategy.

[0015] According to a further aspect of the present application, the present application further provides an electronic device. The electronic device comprises: one or more processors; a storage device configured to store one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement the audio / video transmission method as described above.

[0016] According to a further aspect of the present application, the present application further provides a non-volatile computer readable storage medium. The storage medium has stored thereon a computer program, which, when executed by a processor, enables the processor to implement the audio / video transmission method as described above.

[0017] According to a further aspect of the present application, the present application further provides a computer program product. The computer program product comprises: a computer program stored on a computer readable storage medium; the computer program comprises program instructions, which, when executed by a computer, cause the computer to perform the audio / video transmission method as described above.

[0018] Advantages

[0019] The present application determines the transmission channel and the dynamic encoding strategy corresponding to the transmission priority by configuring the transmission priority of the audio / video data of the host speaker, so that the synchronous transmission of the audio / video data of different transmission priorities can be realized based on the corresponding transmission channel and the corresponding dynamic encoding strategy.

[0020] The present application realizes the content classification of the audio / video data according to the content importance of the audio / video data, so that the multi-path transmission of the audio / video data of different transmission priorities can be realized based on the dynamic encoding strategy in the weak network state. Therefore, the accurate transmission of the key information in the audio / video data can be ensured, and the problem of key information loss in the weak network state is solved. In addition, the reliability and real-time performance of the audio / video data transmission can be ensured simultaneously by the multi-path transmission. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 A flowchart of an audio / video transmission method according to an embodiment of the present application is shown;

[0023] Figure 2 Another flowchart of an audio / video transmission method according to an embodiment of the present application is shown;

[0024] Figure 3 Another flow diagram showing the method of audio / video transmission according to an embodiment of the present application;

[0025] Figure 4 Another flow diagram showing the method of audio / video transmission according to an embodiment of the present application;

[0026] Figure 5 Another flow diagram showing the method of audio / video transmission according to an embodiment of the present application;

[0027] Figure 6 Another flow diagram showing the method of audio / video transmission according to an embodiment of the present application;

[0028] Figure 7 A structure diagram showing the system of audio / video transmission according to an embodiment of the present application.

[0029] BRIEF DESCRIPTION OF DRAWINGS

[0030] Audio / video transmission system 1; content rating determination module 10; network state prediction module 20; channel and policy control module 30; audio / video data transmission module 40. DETAILED DESCRIPTION

[0031] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments, however, can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the figures, and descriptions of the same elements will not be repeated.

[0032] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the

[0033] Moreover, the terms "including" and "having," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a list of steps or elements is not necessarily limited to those listed steps or elements, but can include other not-listed steps or elements, and such process, method, system, product, or apparatus can further include steps or elements inherent in such process, method, system, product, or apparatus.

[0034] The terms "first", "second", etc. in the specification and claims of the present application and in the above drawings are used to distinguish different objects, rather than to describe a specific order.

[0035] The technical solutions of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0036] The inventors found that in the case of unstable network environment, the traditional audio and video transmission strategy at least has the following problems:

[0037] 1. No differential degradation: The traditional weak network degradation strategy usually adopts a non-discriminatory way, that is, when the bandwidth is insufficient, the code rate or resolution of all contents is uniformly reduced. However, this strategy fails to distinguish the importance of the content, resulting in key information (such as text, charts, etc.) and secondary content (such as background video, decorative elements, etc.) being degraded equally.

[0038] For example, in the online meeting or distance education scenario, the key text information in the PPT (PowerPoint) may become blurred due to degradation, seriously affecting the information transmission effect.

[0039] 2. Lack of priority division: The traditional audio and video transmission strategy fails to effectively prioritize the transmission content, resulting in high importance content (such as voice, text, and person picture) failing to obtain sufficient transmission guarantee.

[0040] For example, in the online meeting or distance education scenario, the importance of voice and PPT key frame is much higher than that of background video, but the traditional scheme fails to dynamically adjust the transmission strategy according to the importance of the content, resulting in critical information being lost or severely degraded in a weak network environment.

[0041] 3. Insufficient redundancy mechanism: Usually relying on a single transmission path, such as using only TCP (Transmission Control Protocol) or UDP (User Datagram Protocol) protocol in the data transmission process, lacking multi-path redundancy transmission and intelligent retransmission mechanism. When the network fluctuates or data packets are lost, critical data cannot be quickly recovered, resulting in interruption or quality degradation of information transmission.

[0042] For example, in a weak network environment, although TCP protocol can guarantee the reliability of data, its retransmission mechanism may cause the increase of delay; and although UDP protocol has lower delay, it lacks reliability guarantee and is easy to cause the loss of key data packets.

[0043] Therefore, the inventors believe that the prior art has limitations in the audio and video transmission strategy in a weak network environment, and cannot effectively guarantee the transmission quality of key information.

[0044] According to an aspect of the present application, the present application provides an audio and video transmission method for online conference. Figure 1 A flowchart of the audio and video transmission method according to an embodiment of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the audio and video transmission method can include steps S100-S400.

[0045] Exemplarily, the audio and video transmission method can be executed by an audio and video transmission system (such as a server or a host computer) with computing capability.

[0046] Online conference is a virtual conference form that connects participants distributed in different geographical locations together through internet technology, so as to realize real-time audio and video communication, screen sharing, file transmission and other functions.

[0047] According to an example embodiment, the online conference can include at least one host terminal and multiple participant terminals. For example, the host terminal can be a terminal device where a host of the conference is located; and the participant terminal can be a terminal device where other participants of the conference are located. In the present application, the terminal device includes but is not limited to a computer, a mobile phone and a tablet computer, and the present application does not limit this.

[0048] Exemplarily, the audio and video transmission method can be used for the host terminal, and the present application will be described below taking the host terminal as an example.

[0049] According to an example embodiment, in step S100, the audio and video transmission system configures the transmission priority of the audio and video data of the host terminal according to a preset priority rule.

[0050] For example, the audio and video data of the host terminal can include audio data and video data. The audio data at least includes sound data generated by the host terminal during the conference, such as the sound signal of the host and other background sound signals; and the video data at least includes video data generated by the host terminal during the conference, such as the real-time video picture of the host, the shared screen page (such as the display page of PPT, Word document, software operation, etc.) and other background pictures.

[0051] The preset priority rule can be customized by the user according to actual needs. For example, the audio and video transmission system can respond to the configuration instruction of the user, and divide the audio and video data according to the importance of the data to realize content classification. It can be understood that the preset priority rule can be adjusted in real time according to actual needs, rather than relying on fixed rules. In this way, the configuration flexibility of the audio and video data transmission can be ensured.

[0052] As an embodiment, the audio and video transmission system can configure the sound signal of the main speaker and the shared screen page (such as PPT, Word document, software operation display page) as the first transmission priority according to the preset priority rule, configure the real-time video picture of the main speaker as the second transmission priority, and configure other background pictures as the third transmission priority.

[0053] For example, the audio and video transmission system can extract the preset conference content keywords (such as "budget", "deadline", etc.) through natural language processing technology, and mark the keywords as the first transmission priority, etc. For another example, the audio and video transmission system can automatically identify the complex content such as charts and formulas in the PPT based on the convolutional neural network, so as to ensure the accuracy of content classification.

[0054] In step S200, the audio and video transmission system predicts the network prediction state of the main speaker in the preset future time period based on the network state prediction model.

[0055] For example, the network prediction state can include strong network state, medium network state and weak network state. The network state prediction model can be an LSTM (Long Short-Term Memory, Long Short-Term Memory Network) prediction model. In the prediction of network state, LSTM can predict the future bandwidth fluctuation by learning the change rule of historical data, so as to predict the network state in a specific future time period.

[0056] Exemplarily, the audio and video transmission system can also combine the real-time routing node state (such as congestion situation) to optimize the input features of the LSTM prediction model, so as to improve the prediction accuracy of the LSTM prediction model.

[0057] Figure 2 Another flowchart of the audio and video transmission method of the embodiment of the application is shown.

[0058] Optionally, as shown in Figure 2 S200 can also include S210-S30.

[0059] In step S210, the audio and video transmission system determines the historical network data information of the preset historical time period.

[0060] In step S220, the audio-video transmission system predicts the predicted bandwidth information of the preset future time period according to the historical network data information based on the network state prediction model.

[0061] In step S230, the audio-video transmission system determines that the network prediction state is a weak network state when the predicted bandwidth information is less than the preset threshold.

[0062] For example, the historical network data information at least includes bandwidth information, delay information and packet loss rate information. Exemplarily, the preset historical time period can be 30s before the current time, and the preset future time period can be 10s in the future of the current time.

[0063] The audio-video transmission system trains the LSTM prediction model based on the historical network data information, so that the LSTM prediction model can predict the predicted bandwidth information of 10s in the future. The audio-video transmission system determines that the network prediction state is a weak network state when the predicted bandwidth information is less than the preset threshold (such as 1Mbps).

[0064] Through the above embodiment, the present application can accurately predict the network state based on the LSTM prediction model and the historical network data information.

[0065] In step S300, the audio-video transmission system determines the transmission channel corresponding to the transmission priority and the dynamic encoding strategy corresponding to the transmission priority when the network prediction state is a weak network state.

[0066] For example, the transmission channel can include a high-reliability transmission channel (such as TCP) and a low-delay transmission channel (such as UDP). The high-reliability transmission channel can provide reliable data transmission, ensuring that data packets are not lost, not repeated and arrive in sequence; the low-delay transmission channel can provide low-delay data transmission.

[0067] The audio-video transmission system can configure different transmission channels according to the transmission priority of the audio-video data to transmit the audio-video data based on different transmission channels, so as to realize multi-path transmission in a weak network state.

[0068] As an embodiment, in a weak network state, the first transmission priority can correspond to a high-reliability transmission channel (such as TCP), the second transmission priority can correspond to a high-reliability transmission channel and a low-delay transmission channel (such as TCP+UDP), and the third transmission priority can correspond to a low-delay transmission channel (such as UDP).

[0069] The dynamic coding strategy can include a dynamic code rate adjustment strategy. For example, in a weak network state, the audio and video transmission system can determine a dynamic code rate adjustment strategy corresponding to the transmission priority. Each transmission priority of audio and video data can correspond to a transmission code rate, so that the audio and video data can be transmitted based on the transmission code rate.

[0070] As an embodiment, in a weak network state, the first transmission priority can correspond to a high compression code rate (such as 1080p high compression), the second transmission priority can correspond to a medium compression code rate (such as 720p medium compression), and the third transmission priority can correspond to a low compression code rate (such as 480p low compression).

[0071] In step S400, the audio and video transmission system synchronously transmits the audio and video data corresponding to the transmission priority to the participant terminal through the corresponding transmission channel based on the dynamic coding strategy.

[0072] For example, in a weak network state, the audio and video transmission system can synchronously transmit the audio and video data corresponding to the first transmission priority to the participant terminal through a high-reliability transmission channel based on a high compression code rate; the audio and video transmission system can synchronously transmit the audio and video data corresponding to the second transmission priority to the participant terminal through a high-reliability transmission channel and a low-delay transmission channel based on a medium compression code rate; and the audio and video transmission system can synchronously transmit the audio and video data corresponding to the third transmission priority to the participant terminal through a low-delay transmission channel based on a low compression code rate.

[0073] As an embodiment, in a weak network state, the audio and video transmission system can also combine other network transmission paths (including but not limited to Wi-Fi6, 5G, etc. Network transmission path) for multi-path transmission.

[0074] Through the above embodiments, the audio and video data transmission priority of the host terminal can be configured. In the case of predicting that the current network environment is a weak network state, the transmission channel and the dynamic coding strategy corresponding to the transmission priority can be determined, so that the audio and video data of different transmission priorities can be synchronously transmitted based on the corresponding transmission channel and the corresponding dynamic coding strategy.

[0075] The present application realizes content classification of audio and video data according to the importance of the content of the audio and video data. In a weak network state, the audio and video data of different transmission priorities can be transmitted based on a dynamic coding strategy. Thus, the accurate transmission of key information in the audio and video data can be ensured, and the problem of key information loss in a weak network state can be solved. In addition, through multi-path transmission, the present application can ensure the reliability and real-time performance of audio and video data transmission.

[0076] Optionally, in step S400, the audio-video transmission system can also dynamically adjust the forward error correction redundancy of the audio-video data based on the predicted bandwidth information.

[0077] For example, the dynamic coding strategy can also include a dynamic forward error correction redundancy adjustment strategy. The dynamic forward error correction redundancy adjustment strategy can be to dynamically adjust the forward error correction redundancy of the audio-video data according to the predicted bandwidth information.

[0078] For example, the audio-video transmission system can determine whether the bandwidth of the current network is predicted to increase or decrease according to the predicted bandwidth information and the current bandwidth information.

[0079] In the case where the bandwidth of the current network is predicted to increase, the audio-video transmission system can reduce the forward error correction redundancy, so as to reduce the occupied bandwidth resources and improve the transmission quality of the audio-video. In the case where the bandwidth of the current network is predicted to decrease, the audio-video transmission system can increase the forward error correction redundancy, so as to increase the error correction capability of data transmission, and avoid problems such as video freezing or picture quality degradation caused by packet loss.

[0080] As an example, in the case where the LSTM prediction model predicts that the bandwidth of the current network is predicted to decrease by 30%, the audio-video transmission system increases the forward error correction redundancy of the audio-video data corresponding to the first transmission priority by 20%.

[0081] Through the above examples, the present application can optimize and adjust the forward error correction redundancy according to the change of network conditions through the dynamic forward error correction redundancy adjustment strategy, so as to balance the error correction capability and bandwidth occupation, thereby improving the quality and stability of audio-video data transmission.

[0082] Figure 3 Another flowchart of an audio-video transmission method according to an embodiment of the present application is shown.

[0083] Optionally, as shown in the figure, Figure 3 The audio-video transmission method can further include steps S510-S520.

[0084] In step S510, the audio-video transmission system dynamically determines the visual attention area of the user of the participant terminal based on visual tracking.

[0085] In step S520, the audio-video transmission system improves the display resolution of the visual attention area to a preset resolution in the case where the network prediction state is a weak network state.

[0086] For example, the audio-video transmission system can collect the image of the user based on the front camera of the participant terminal, and track the eye movement of the user in real time through visual tracking technology (such as a deep learning model), so as to determine the gaze area (i.e., the viewing area) of the user. Then, the audio-video transmission system can improve the display resolution of the gaze area (such as from 1080p to 4k).

[0087] Through the above embodiments, the visual gaze area of the user of the participant terminal can be determined through visual tracking, so that the transmission of data packets in the key gaze area can be prioritized, and the retransmission and redundancy of non-critical areas can be reduced, thereby improving the amount of effective information per Mbps bandwidth.

[0088] Figure 4 Another flowchart of the audio-video transmission method of the embodiment of the present application is shown.

[0089] Optionally, as shown in Figure 4 The audio-video transmission method can further include step S600.

[0090] In step S600, the audio-video transmission system determines the lost frame data based on the front frame data and the rear frame data of the audio-video data received by the participant terminal when detecting that the audio-video data received by the participant terminal is lost.

[0091] For example, when the audio-video transmission system detects that part of the frames (such as from the 10th frame directly to the 12th frame) of the audio-video data of the participant terminal are lost, the audio-video transmission system automatically determines the missing data (such as the 11th frame) based on the received front and rear frame data (10th and 12th frames) through an algorithm. In this way, the smoothness of the audio-video data playback can be ensured.

[0092] Optionally, the audio-video transmission system can only trigger data retransmission for the lost frame data corresponding to the first transmission priority, without retransmitting all audio-video data corresponding to the first transmission priority.

[0093] In this way, the flow resources required for redundant data retransmission can be reduced. The audio-video transmission system can combine NACK (Negative Acknowledgment, negative acknowledgment) message fast feedback and high-reliability channel transmission, so as to shorten the recovery time of the key data.

[0094] According to an example embodiment, the audio-video transmission system triggering data retransmission for the lost frame data corresponding to the first transmission priority can include S1-S10.

[0095] In S1, the audio / video transmission system embeds a priority identification field in each audio / video data to mark the corresponding transmission priority (e.g., the first transmission priority is marked as 0x01).

[0096] In S2, the audio / video transmission system assigns a globally increasing sequence number (e.g., 32 bits) to each audio / video data to track the order and integrity of the audio / video data.

[0097] In S3, the audio / video transmission system divides the audio / video data of the first transmission priority (e.g., the key frame of the PPT) into fixed-size data blocks (e.g., 512 bytes per block), each of which can be independently numbered. In this way, the lost unit can be accurately identified.

[0098] In S4, the audio / video transmission system can detect whether there is a lost data block at the participant end through the continuity of the sequence number.

[0099] For example, the participant end can maintain a dynamic receiving buffer to record the sequence numbers of the received data blocks. The audio / video transmission system determines the lost data block by comparing the continuity of adjacent sequence numbers (e.g., if the participant end receives sequence numbers 100 and 102, it can determine that 101 is a lost data block). In this way, the audio / video transmission system can identify the loss of the audio / video data of the first transmission priority at the participant end in real time. Illustratively, the audio / video transmission system can also identify the range of continuous packet loss through a sliding window mechanism, and the window size of the sliding window can be dynamically adjusted according to the network delay.

[0100] In S5, the audio / video transmission system generates a NACK message when it detects a lost data block. The NACK message can include the sequence number range of the lost data block and the corresponding priority identification field, etc.

[0101] In S6, the audio / video transmission system sends the NACK message generated by the participant end to the presenter end through a low-delay channel (e.g., UDP).

[0102] In S7, the presenter end can maintain a ring-shaped sending buffer for the audio / video data of the first transmission priority to save the recently sent data blocks (e.g., the audio / video data within the last 5s is retained).

[0103] Illustratively, the capacity of the ring-shaped sending buffer can be dynamically adjusted according to the network delay to ensure that the possible packet loss time window is covered.

[0104] In S8, the audio / video transmission system can extract the corresponding lost data block from the ring-shaped sending buffer after receiving and analyzing the NACK packet at the main speaker end. The audio / video transmission system adds the lost data block to the high-priority retransmission queue and then sends the lost data block immediately through a high-reliability channel (such as TCP).

[0105] For example, if the lost data block has expired (i.e., exceeds the time window of the ring-shaped sending buffer), the audio / video transmission system can ignore the request and feed back an error code.

[0106] For example, the audio / video transmission system can combine the network state prediction result to perform data retransmission. For example, in the case of sufficient current bandwidth, the audio / video transmission system directly triggers the retransmission of the lost data block; in the case of tight current bandwidth, the transmission of non-critical data is suspended to prioritize the retransmission of the lost data block. In this way, the audio / video transmission system can combine dynamic bandwidth allocation and auxiliary redundancy through forward error correction to balance the network load, thereby improving the transmission efficiency of data.

[0107] In S9, after the participant end obtains the retransmission data, the audio / video transmission system inserts the retransmission data into the target position of the original data stream according to the sequence number and updates the playback buffer to ensure continuous playback of the audio / video stream.

[0108] For example, the audio / video transmission system can also dynamically add redundant error correction packets (such as one redundant block for every five data blocks) for the audio / video data of the first transmission priority. In the case of a small amount of lost data blocks, the audio / video transmission system can directly recover through the redundant error correction packets to reduce the retransmission demand.

[0109] In S10, the audio / video transmission system can also ensure the synchronization of the retransmission data through a synchronization correction mechanism.

[0110] For example, in the case of audio / video data desynchronization due to the delay caused by the retransmission of the lost data block, the audio / video transmission system can ensure the synchronization of the audio / video data through audio waveform dynamic stretching (such as fine-tuning the length of the audio data through a dynamic time warping algorithm) or video frame rate fine-tuning (such as discarding redundant video frames or inserting interpolation frames to keep the synchronization error of the audio / video data within an acceptable threshold range).

[0111] Optionally, the audio / video transmission system can also set a timeout timer (such as 200 ms) for each NACK packet. In the case of timeout without receiving the retransmission data at the participant end, a secondary NACK packet request is triggered, and the transmission priority of the lost data block is lowered to avoid blocking subsequent data transmission.

[0112] Figure 5 Another flowchart of the audio / video transmission method according to an embodiment of the present application is shown.

[0113] Optionally, as shown in Figure 5 the audio and video transmission method can further include step S700.

[0114] In step S700, the audio and video transmission system dynamically stretches the audio data or compresses the video data based on the audio waveform, in the case that the delay of the audio data and the video data in the audio and video data is greater than a preset delay threshold, so as to make the audio data and the video data synchronously transmitted.

[0115] For example, in the audio and video transmission system, when the delay difference of the audio data and the video data exceeds a preset threshold (such as ±80ms), the audio and video transmission system can realize accurate synchronization of the audio data and the video data through audio waveform dynamic stretching / compression or frame rate adjustment.

[0116] According to another aspect of the present application, the present application also provides an audio and video transmission method for online conference. Figure 6 Another flowchart of the audio and video transmission method of the embodiment of the present application is shown. As shown in Figure 6 the audio and video transmission method can include steps S100a-S400a.

[0117] Exemplarily, the audio and video transmission method can be used for a participant terminal, and the present application will be described below taking the participant terminal as an example.

[0118] According to an example embodiment, in step S100a, the audio and video transmission system configures the transmission priority of the audio and video data of the participant terminal according to a preset priority rule.

[0119] In step S200a, the audio and video transmission system predicts the network prediction state of the participant terminal in a preset future time period based on a network state prediction model.

[0120] In step S300a, the audio and video transmission system determines the transmission channel corresponding to the transmission priority and the dynamic encoding strategy corresponding to the transmission priority in the case that the network prediction state is a weak network state.

[0121] In step S400a, the audio and video transmission system synchronously transmits the audio and video data corresponding to the transmission priority to the host terminal through the corresponding transmission channel based on the dynamic encoding strategy.

[0122] It can be understood here that the audio and video transmission method for the participant terminal is completely the same as the technical solution adopted by the audio and video transmission method for the host terminal described above, only the execution end is different. The technical solution adopted by the audio and video transmission method for the host terminal has been described in detail above, and therefore will not be described here again.

[0123] According to another aspect of the present application, the present application further provides an audio and video transmission system for online conference. Figure 7 A structure diagram of an audio and video transmission system according to an embodiment of the present application is shown. As shown, the audio and video transmission system 1 can include a content classification determining module 10, a network state predicting module 20, a channel and strategy control module 30 and an audio and video data transmission module 40. Figure 7

[0124] Exemplarily, the audio and video transmission system can be used for a host terminal, and the present application will be described below by taking the host terminal as an example.

[0125] According to an example embodiment, the content classification determining module 10 configures transmission priorities of audio and video data of the host terminal according to a preset priority rule.

[0126] For example, the audio and video data of the host terminal can include audio data and video data. The audio data at least includes sound data generated by the host terminal during the conference, such as sound signals of the host and other background sound signals, etc.; and the video data at least includes video data generated by the host terminal during the conference, such as real-time video pictures of the host, shared screen pages (such as PPT, Word document, software operation display pages, etc.) and other background pictures, etc.

[0127] The preset priority rule can be self-defined by a user according to actual needs. For example, the content classification determining module 10 can respond to a configuration instruction of the user, and divide the audio and video data according to importance of the data to realize content classification. It can be understood here that the preset priority rule can be adjusted in real time according to actual needs, rather than relying on a fixed rule. In this way, the configuration flexibility of the audio and video data transmission can be ensured.

[0128] As an example, the content classification determining module 10 can configure sound signals of the host and shared screen pages (such as PPT, Word document, software operation display pages, etc.) as a first transmission priority, configure real-time video pictures of the host as a second transmission priority, and configure other background pictures as a third transmission priority according to the preset priority rule.

[0129] For example, the content classification determining module 10 can extract preset conference content keywords (such as “budget”, “deadline”, etc.) by natural language processing technology, and mark the keywords as the first transmission priority, etc. As another example, the content classification determining module 10 can automatically identify complex content such as charts and formulas in the PPT based on a convolutional neural network, so that the accuracy of content classification can be ensured.

[0130] ​According to an example embodiment, the network state prediction module 20 predicts a network prediction state of the presenter in a preset future time period based on a network state prediction model.

[0131] For example, the network prediction state can include a strong network state, a medium network state and a weak network state. The network state prediction model can be a LSTM (Long Short-Term Memory) prediction model. In the prediction of the network state, the LSTM can predict the bandwidth fluctuation in the future by learning the change rule of the historical data, so as to predict the network state in a specific time period in the future.

[0132] For example, the network state prediction module 20 can also optimize the input features of the LSTM prediction model in combination with the real-time routing node state (such as congestion situation), so as to improve the prediction accuracy of the LSTM prediction model.

[0133] Optionally, the network state prediction module 20 determines historical network data information in a preset historical time period.

[0134] The network state prediction module 20 predicts prediction bandwidth information in a preset future time period based on the network state prediction model according to the historical network data information.

[0135] The network state prediction module 20 determines that the network prediction state is a weak network state when the prediction bandwidth information is less than a preset threshold.

[0136] For example, the historical network data information at least includes bandwidth information, delay information and packet loss rate information. For example, the preset historical time period can be 30s before the current time, and the preset future time period can be 10s in the future of the current time.

[0137] The network state prediction module 20 trains the LSTM prediction model based on the historical network data information, so that the LSTM prediction model can predict the prediction bandwidth information in the future 10s. The network state prediction module 20 determines that the network prediction state is a weak network state when the prediction bandwidth information is less than a preset threshold (such as 1Mbps).

[0138] Through the above embodiment, the application can accurately predict the network state based on the LSTM prediction model based on the historical network data information.

[0139] According to an example embodiment, the channel and policy control module 30 determines a transmission channel corresponding to the transmission priority and a dynamic encoding policy corresponding to the transmission priority when the network prediction state is a weak network state.

[0140] For example, the transmission channels can include a high-reliability transmission channel (such as TCP) and a low-latency transmission channel (such as UDP). The high-reliability transmission channel can provide reliable data transmission, ensuring that data packets are not lost, not repeated, and arrive in order; the low-latency transmission channel can provide low-latency data transmission.

[0141] The channel and policy control module 30 can configure different transmission channels according to the transmission priorities of the audio and video data, so that the transmission of the audio and video data based on different transmission channels can be realized, thereby realizing multi-path transmission in a weak network state.

[0142] As an example, in a weak network state, the first transmission priority can correspond to a high-reliability transmission channel (such as TCP), the second transmission priority can correspond to a high-reliability transmission channel and a low-latency transmission channel (such as TCP+UDP), and the third transmission priority can correspond to a low-latency transmission channel (such as UDP).

[0143] The dynamic encoding strategy can include a dynamic code rate adjustment strategy. For example, in a weak network state, the channel and policy control module 30 can determine a dynamic code rate adjustment strategy corresponding to the transmission priority, and the audio and video data of each transmission priority can correspond to a transmission code rate, so that the audio and video data can be transmitted based on the transmission code rate.

[0144] As an example, in a weak network state, the first transmission priority can correspond to a high compression code rate (such as 1080p high compression), the second transmission priority can correspond to a medium compression code rate (720p medium compression), and the third transmission priority can correspond to a low compression code rate (such as 480p low compression).

[0145] According to the example embodiment, the audio and video data transmission module 40 synchronously transmits the audio and video data corresponding to the transmission priority to the participant terminal based on the dynamic encoding strategy through the corresponding transmission channel.

[0146] For example, in a weak network state, the audio and video data transmission module 40 can synchronously transmit the audio and video data corresponding to the first transmission priority to the participant terminal based on a high compression code rate through a high-reliability transmission channel; the audio and video data transmission module 40 synchronously transmits the audio and video data corresponding to the second transmission priority to the participant terminal based on a medium compression code rate through a high-reliability transmission channel and a low-latency transmission channel; and the audio and video data transmission module 40 synchronously transmits the audio and video data corresponding to the third transmission priority to the participant terminal based on a low compression code rate through a low-latency transmission channel.

[0147] As an example, in a weak network state, the audio and video data transmission module 40 can also combine other network transmission paths (including but not limited to Wi-Fi 6, 5G, etc. Network transmission path) for multi-path transmission.

[0148] Through the above embodiment, the application can configure the transmission priority of the audio and video data of the host, and in the case of predicting that the current network environment is a weak network state, the transmission channel and the dynamic encoding strategy corresponding to the transmission priority can be determined, so that the application can realize the synchronous transmission of audio and video data with different transmission priorities based on the corresponding transmission channel and the corresponding dynamic encoding strategy.

[0149] The application realizes content classification of audio and video data according to the content importance of the audio and video data, and can realize multi-path transmission of audio and video data with different transmission priorities based on a dynamic encoding strategy in a weak network state. Thus, the accurate transmission of key information in the audio and video data can be ensured, and the problem of key information loss in a weak network state is solved. In addition, through multi-path transmission, the application can ensure the reliability and real-time performance of audio and video data transmission.

[0150] Optionally, the channel and strategy control module 30 can also dynamically adjust the forward error correction redundancy of the audio and video data based on the predicted bandwidth information.

[0151] For example, the dynamic encoding strategy can also include a dynamic forward error correction redundancy adjustment strategy. The dynamic forward error correction redundancy adjustment strategy can dynamically adjust the forward error correction redundancy of the audio and video data according to the predicted bandwidth information.

[0152] Exemplarily, the channel and strategy control module 30 can determine whether the bandwidth of the current network is predicted to increase or decrease according to the predicted bandwidth information and the current bandwidth information.

[0153] In the case of the bandwidth of the current network being predicted to increase, the channel and strategy control module 30 can reduce the forward error correction redundancy, so as to reduce the occupied bandwidth resources and improve the transmission quality of the audio and video. In the case of the bandwidth of the current network being predicted to decrease, the channel and strategy control module 30 can increase the forward error correction redundancy, so as to increase the error correction capability of data transmission, and can avoid problems such as video freezing or picture quality degradation caused by packet loss.

[0154] As an example, in the case of the LSTM prediction model predicting that the bandwidth of the current network is predicted to decrease by 30%, the channel and strategy control module 30 increases the forward error correction redundancy of the audio and video data corresponding to the first transmission priority by 20%.

[0155] Through the above embodiment, the application can optimize and adjust the forward error correction redundancy accordingly in the case of network condition changes through the dynamic forward error correction redundancy adjustment strategy, so as to balance the error correction capability and the bandwidth occupation, thereby improving the quality and stability of the audio and video data transmission.

[0156] Optionally, the channel and strategy control module 30 dynamically determines the visual attention area of the user of the participant terminal based on visual tracking.

[0157] The channel and strategy control module 30 increases the display resolution of the visual attention area to a preset resolution in the case that the network prediction state is a weak network state.

[0158] For example, the channel and strategy control module 30 can collect the image of the user based on the front camera of the participant terminal, and track the eye movement of the user in real time through a visual tracking technology (such as a deep learning model), so as to determine the attention area (i.e., the viewing area) of the user. Then, the channel and strategy control module 30 can increase the display resolution of the attention area (such as increasing the resolution from 1080p to 4k).

[0159] Through the above embodiment, the visual attention area of the user of the participant terminal can be determined through visual tracking, so that the transmission of data packets of the key attention area can be preferentially ensured, the retransmission and redundancy of non-key areas can be reduced, and the amount of effective information per Mbps bandwidth can be increased.

[0160] Optionally, the channel and strategy control module 30 determines the lost frame data based on the front frame data and the rear frame data of the audio and video data received by the participant terminal in the case that the audio and video data received by the participant terminal is detected to be lost frames.

[0161] For example, when the channel and strategy control module 30 detects that part of the frames (such as directly jumping from the 10th frame to the 12th frame) of the audio and video data of the participant terminal are lost, the channel and strategy control module 30 automatically determines the missing data (such as the 11th frame) based on the received front and rear frame data (the 10th frame and the 12th frame) through an algorithm. In this way, the smoothness of the audio and video data playback can be ensured.

[0162] Optionally, the audio and video data transmission module 40 can also trigger data retransmission only for the lost data of the audio and video data corresponding to the first transmission priority, without retransmitting all the audio and video data corresponding to the first transmission priority.

[0163] In this way, the flow resources required for redundant data retransmission can be reduced. The audio and video data transmission module 40 can combine the NACK message fast feedback and the high-reliability channel transmission, so as to shorten the recovery time of the key data.

[0164] According to an example embodiment, the audio and video transmission system triggers data retransmission for the lost frame data corresponding to the first transmission priority.

[0165] For example, the audio / video data transmission module 40 embeds a corresponding priority identification field in each audio / video data to mark the corresponding transmission priority (e.g., the first transmission priority is marked as 0x01).

[0166] The audio / video data transmission module 40 assigns a globally increasing sequence number (e.g., 32 bits) to each audio / video data for tracking the order and integrity of the audio / video data.

[0167] The audio / video data transmission module 40 can divide the audio / video data of the first transmission priority (e.g., the key frame of the PPT) into fixed-size data blocks (e.g., 512 bytes per block), and each data block can be independently numbered. In this way, the lost unit can be accurately identified.

[0168] The audio / video data transmission module 40 can detect whether there is a lost data block by the continuity of the sequence number.

[0169] For example, the participant can maintain a dynamic receiving buffer to record the sequence numbers of the received data blocks. The audio / video data transmission module 40 determines the lost data block by comparing the continuity of adjacent sequence numbers (e.g., if the participant receives sequence numbers 100 and 102, it can be determined that 101 is a lost data block). In this way, the audio / video data transmission module 40 can identify the loss of the audio / video data of the first transmission priority in real time. For example, the audio / video data transmission module 40 can also identify the range of continuous packet loss by a sliding window mechanism, and the window size of the sliding window can be dynamically adjusted according to the network delay.

[0170] When the audio / video data transmission module 40 detects a lost data block, the participant generates a NACK (Negative Acknowledgment) message. The NACK message can include the sequence number range of the lost data block and the corresponding priority identification information.

[0171] The audio / video data transmission module 40 sends the NACK message generated by the participant to the presenter through a low-delay channel (e.g., UDP).

[0172] The presenter can maintain a ring-shaped sending buffer for the audio / video data of the first transmission priority to save the recently sent data blocks (e.g., the audio / video data within the last 5s is retained).

[0173] For example, the capacity of the ring-shaped sending buffer can be dynamically adjusted according to the network delay to ensure that the possible packet loss time window is covered.

[0174] The audio / video data transmission module 40 can extract the corresponding lost data block from the ring-shaped sending buffer after receiving and analyzing the NACK message at the host terminal. The audio / video data transmission module 40 adds the lost data block to the high-priority retransmission queue and then sends the lost data block immediately through a high-reliability channel (such as TCP).

[0175] For example, if the lost data block has expired (i.e., exceeds the time window of the ring-shaped sending buffer), the audio / video data transmission module 40 can ignore the request and feedback an error code.

[0176] For example, the audio / video data transmission module 40 can combine network state prediction results for data retransmission. For example, in the case of sufficient bandwidth, the audio / video data transmission module 40 directly triggers retransmission of the lost data block; in the case of tight bandwidth, the transmission of non-critical data is suspended to prioritize the retransmission of the lost data block. In this way, the audio / video data transmission module 40 can balance network load by combining dynamic bandwidth allocation and forward error correction assisted redundancy, thereby improving data transmission efficiency.

[0177] After the participant terminal obtains the retransmitted data, the audio / video data transmission module 40 inserts it into the target position of the original data stream according to the sequence number and updates the playback buffer to ensure continuous playback of the audio / video stream.

[0178] For example, the audio / video data transmission module 40 can also dynamically add redundant error correction packets (such as one redundant block for every five data blocks) to the audio / video data of the first transmission priority. In the case of a small number of lost data blocks, the audio / video data transmission module 40 can directly recover through redundant error correction packets, reducing the need for retransmission.

[0179] The audio / video data transmission module 40 can also ensure the synchronization of retransmitted data through a synchronization correction mechanism.

[0180] For example, in the case of audio / video data desynchronization due to delay caused by retransmission of lost data blocks, the audio / video data transmission module 40 can ensure the synchronization of audio / video data through audio waveform dynamic stretching (such as adjusting the duration of audio data or video frame rate through dynamic time warping algorithm, inserting interpolation frames, etc.) to keep the synchronization error of audio / video data within an acceptable threshold range.

[0181] Optionally, the audio / video data transmission module 40 can also set a timeout timer (such as 200 ms) for each NACK message. In the case of timeout without receiving retransmitted data at the participant terminal, a second NACK message request is triggered, and the transmission priority of the lost data block is reduced to avoid blocking subsequent data transmission.

[0182] Optionally, the audio-video data transmission module 40 dynamically stretches the audio data or compresses the video data based on the audio waveform, in a case that the delay of the audio data and the video data in the audio-video data is greater than a preset delay threshold, so as to synchronize the transmission of the audio data and the video data.

[0183] For example, when the delay difference of the audio data and the video data exceeds a preset threshold (such as ±80 ms), the audio-video data transmission module 40 realizes the accurate synchronization of the audio data and the video data through dynamic stretching / compression of the audio waveform or adjustment of the video frame rate.

[0184] It can be understood that the audio-video transmission system can also be used for the participant terminal, and details are not repeated here.

[0185] According to another aspect of the present application, the present application further provides an electronic device. The electronic device comprises: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors can implement the audio-video transmission method as described above.

[0186] According to another aspect of the present application, the present application further provides a non-volatile computer readable storage medium. The storage medium stores a computer program, and the computer program can implement the audio-video transmission method as described above when executed by a processor.

[0187] According to another aspect of the present application, the present application further provides a computer program product. The computer program product comprises: a computer program stored on a computer readable storage medium; the computer program comprises program instructions, when the program instructions are executed by a computer, the computer executes the audio-video transmission method as described above.

[0188] Finally, it should be noted that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the above describes the present application in detail, those skilled in the art can still modify the technical solutions of the above embodiments or make equivalent replacements to some technical features. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An audio and video transmission method for online conference, characterized in that, The online conference includes a host terminal and a participant terminal, and the audio and video transmission method includes: According to a preset priority rule, the transmission priority of different important content of the audio and video data of the host terminal is configured, and the configuration is that according to the preset priority rule, the sound signal of the host and the shared screen page are configured as a first transmission priority, the real-time video picture of the host is configured as a second transmission priority, and the background picture including other background pictures is configured as a third transmission priority; Based on a network state prediction model, a network prediction state of the host terminal in a preset future time period is predicted; In the case that the network prediction state is a weak network state, a transmission channel corresponding to the transmission priority and a dynamic coding strategy corresponding to the transmission priority are determined, the transmission channel includes a high-reliability transmission channel and a low-delay transmission channel, and the dynamic coding strategy includes a dynamic code rate adjustment strategy; Based on the dynamic coding strategy, the audio and video data corresponding to the transmission priority are transmitted to the participant terminal through the corresponding transmission channel, including: A dynamic code rate adjustment strategy corresponding to the transmission priority is determined, and the audio and video data of each transmission priority corresponds to a transmission code rate, so that the audio and video data are transmitted based on the transmission code rate.

2. The audio-video transmission method of claim 1, wherein, The network prediction state of the host terminal in a preset future time period is predicted based on a network state prediction model, including: Determine the historical network data information of a preset historical time period; Based on the network state prediction model, the prediction bandwidth information of the preset future time period is predicted according to the historical network data information; In the case that the prediction bandwidth information is less than a preset threshold, the network prediction state is determined as the weak network state.

3. The audio-video transmission method of claim 2, wherein, Based on the prediction bandwidth information, the forward error correction redundancy of the audio and video data is dynamically adjusted. The audio and video transmission method further includes:

4. The audio-video transmission method of claim 1, wherein, Based on visual tracking, the visual attention area of the user of the participant terminal is dynamically determined; In the case that the network prediction state is the weak network state, the display resolution of the visual attention area is improved to a preset resolution. The audio and video transmission method further includes:

5. The audio-video transmission method of claim 1, wherein, In the case that the audio and video data received by the participant terminal is detected to be lost frames, the lost frame data is determined based on the front frame data and the rear frame data of the audio and video data received by the participant terminal. The audio and video transmission method further includes:

6. The audio-video transmission method of claim 1, wherein, In the case that the delay of the audio data and the video data in the audio and video data is greater than a preset delay threshold, the audio data is dynamically stretched based on the audio waveform or the video data is compressed or speeded up, so that the audio data and the video data are transmitted synchronously. The online conference includes a host terminal and a participant terminal, and the audio and video transmission method includes:

7. An audio and video transmission method for an online conference, characterized by, ​ According to the preset priority rule, the transmission priorities of different importance contents of the audio and video data of the participant terminal are configured, and the configuration is that according to the preset priority rule, the sound signal of the main speaker and the shared screen page are configured as a first transmission priority, the real-time video picture of the main speaker is configured as a second transmission priority, and the other background picture is configured as a third transmission priority; The network prediction state of the participant terminal in a preset future time period is predicted based on a network state prediction model; In the case that the network prediction state is a weak network state, the transmission channel corresponding to the transmission priority and the dynamic encoding strategy corresponding to the transmission priority are determined, the transmission channel includes a high-reliability transmission channel and a low-delay transmission channel, and the dynamic encoding strategy includes a dynamic code rate adjustment strategy; The audio and video data corresponding to the transmission priority are synchronously transmitted to the main speaker through the corresponding transmission channel based on the dynamic encoding strategy, including: The dynamic code rate adjustment strategy corresponding to the transmission priority is determined, and the audio and video data of each transmission priority corresponds to a transmission code rate, so that the audio and video data are transmitted based on the transmission code rate.

8. An audio and video transmission system for online meetings, characterized by, The online conference includes a main speaker and a participant, and the audio and video transmission system is used to execute the audio and video transmission method in any one of claims 1-6, and the audio and video transmission system includes: A content classification determination module is configured to configure transmission priorities of different importance contents of audio and video data of the main speaker according to a preset priority rule, and the configuration is that according to the preset priority rule, the sound signal of the main speaker and the shared screen page are configured as a first transmission priority, the real-time video picture of the main speaker is configured as a second transmission priority, and the other background picture is configured as a third transmission priority; A network state prediction module is configured to predict a network prediction state of the main speaker in a preset future time period based on a network state prediction model; A channel and strategy control module is configured to determine a transmission channel corresponding to the transmission priority and a dynamic encoding strategy corresponding to the transmission priority in the case that the network prediction state is a weak network state. An audio and video data transmission module is configured to synchronously transmit audio and video data corresponding to the transmission priority to the participant through the corresponding transmission channel based on the dynamic encoding strategy.

9. An electronic device, comprising: One or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the audio and video transmission method in any one of claims 1-7. The computer program is executed by the processor to implement the audio and video transmission method in any one of claims 1-7.

10. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that, ​

Images