Audio and video data processing method and system, electronic equipment and storage medium
By obtaining the terminal information of the target conference terminal on the video conferencing client and sending anti-packet loss encoding instructions, the problem of lag caused by network packet loss in video conferencing is solved, the transmission quality and reliability of audio and video data is improved, and the transmission quality and reliability of audio and video data is improved. User experience and meeting stability.
Patent Information
- Application Number
- CN202410992664.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-12-03
AI Technical Summary
During video conferencing, data packets are lost due to the complexity and instability of the network environment, which affects the transmission of audio and video data, resulting in lag and conference results.
Obtain the terminal information of the target conference terminal on the video conferencing client, and send the target conference terminal anti-packet loss encoding instruction based on this information, so that it can perform anti-packet loss encoding processing, so as to perform redundant information when the data packet is lost Data recovery ensures the continuity and integrity of audio and video data.
It effectively reduces the impact of packet loss on audio and video data transmission, improves the stability and fluency of video conferences, reduces lag, improves user experience, and flexibly responds to changes in network conditions, reducing the overhead of retransmission.
Smart Images

Figure CN119071589A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video conferencing, and in particular to a method for processing audio and video data, a system for processing audio and video data, an electronic device, and a computer-readable storage medium. Background Art
[0002] In a video conferencing system, smooth transmission of video and audio is the key to ensure a smooth conference. Video conferencing usually relies on network connections to transmit real-time video and audio data over the network. However, due to the complexity and instability of the network environment, video conferencing may face various technical challenges, one of which is the loss of data packets, also known as packet loss.
[0003] Packet loss refers to the failure of a data packet to reach its destination or loss during transmission. Once packet loss occurs, it may cause the display of audio and video data to become stuck, seriously affecting the video conference effect. Summary of the invention
[0004] In view of the above problems, embodiments of the present invention are proposed to provide a method for processing audio and video data, a system for processing audio and video data, an electronic device and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.
[0005] In order to solve the above problems, an embodiment of the present invention discloses a method for processing audio and video data, which is applied to a video conferencing client. The method includes: obtaining terminal information of a target conference terminal from a first conference terminal; sending an anti-packet loss coding enable instruction to the target conference terminal according to the terminal information, so that the target conference terminal performs anti-packet loss coding processing on the audio and video data of a first code stream according to the anti-packet loss coding enable instruction; and transmitting the processed audio and video data of the first code stream to the first conference terminal.
[0006] Optionally, acquiring the terminal information of the target conference terminal from the first conference terminal includes: acquiring terminal identification information, packet loss rate, and encoding information of the target conference terminal from the first conference terminal.
[0007] Optionally, sending an instruction to enable anti-packet loss coding to the target conference terminal according to the terminal information includes: sending an instruction to enable forward error correction coding to the target conference terminal according to the terminal information.
[0008] Optionally, transmitting the processed audio and video data of the first code stream to the first conference terminal includes: sending a stream adjustment notification to the video conference server, so that the video conference server transmits the processed audio and video data of the first code stream from the autonomous server to the first conference terminal according to the stream adjustment notification.
[0009] Optionally, before obtaining the terminal information of the target conference terminal from the first conference terminal, the method also includes: in response to an anti-packet loss activation operation, adding the first conference terminal to a monitored packet loss list; detecting whether a packet loss problem occurs at a conference terminal in the monitored packet loss list; and if a packet loss problem occurs at the first conference terminal, executing the operation of obtaining the terminal information of the target conference terminal from the first conference terminal.
[0010] Optionally, after detecting whether a conference terminal in the monitoring packet loss list has a packet loss problem, the method further includes: if a second conference terminal in the monitoring packet loss list does not have a packet loss problem, transmitting the unprocessed audio and video data of the second code stream to the second conference terminal; wherein the resolution of the audio and video data of the second code stream is greater than the resolution of the audio and video data of the first code stream.
[0011] Optionally, after transmitting the processed audio and video data of the first code stream to the first conference terminal, the method further includes: sending a stream switching instruction to the first conference terminal, so that the first conference terminal switches the unprocessed audio and video data of the first code stream to the processed audio and video data of the first code stream according to the stream switching instruction.
[0012] An embodiment of the present invention also discloses a system for processing audio and video data, which is applied to a video conferencing client. The system includes: a terminal information acquisition module, which is used to acquire terminal information of a target conference terminal from a first conference terminal; an anti-packet loss encoding module, which is used to send an anti-packet loss encoding enable instruction to the target conference terminal based on the terminal information, so that the target conference terminal performs anti-packet loss encoding processing on the audio and video data of the first code stream according to the anti-packet loss encoding enable instruction; and an audio and video transmission module, which is used to transmit the processed audio and video data of the first code stream to the first conference terminal.
[0013] Optionally, the terminal information acquisition module is used to acquire terminal identification information, packet loss rate, and encoding information of the target conference terminal from the first conference terminal.
[0014] Optionally, the anti-packet loss encoding module is used to send a forward error correction encoding start instruction to the target conference terminal according to the terminal information.
[0015] Optionally, the audio and video transmission module is used to send a stream adjustment notification to the video conferencing server, so that the video conferencing server transmits the processed audio and video data of the first code stream from the autonomous server to the first conference terminal according to the stream adjustment notification.
[0016] Optionally, the system also includes: a terminal adding module, which is used to add the first conference terminal to a monitoring packet loss list in response to an anti-packet loss activation operation before the terminal information acquisition module acquires the terminal information of the target conference terminal from the first conference terminal; a packet loss detection module, which is used to detect whether a conference terminal in the monitoring packet loss list has a packet loss problem; and the terminal information acquisition module, which is used to acquire the terminal information of the target conference terminal from the first conference terminal if a packet loss problem occurs to the first conference terminal.
[0017] Optionally, the system also includes: an unencoded module, which is used to transmit the unprocessed audio and video data of the second code stream to the second conference terminal after the packet loss detection module detects whether the conference terminal in the monitored packet loss list has a packet loss problem. If the second conference terminal in the monitored packet loss list does not have a packet loss problem, the resolution of the audio and video data of the second code stream is greater than the resolution of the audio and video data of the first code stream.
[0018] Optionally, the system also includes: an audio and video switching module, which is used to send a switching instruction to the first conference terminal after the audio and video transmission module transmits the processed audio and video data of the first code stream to the first conference terminal, so that the first conference terminal switches the unprocessed audio and video data of the first code stream to the processed audio and video data of the first code stream according to the switching instruction.
[0019] An embodiment of the present invention further discloses an electronic device, comprising: one or more processors; and one or more machine-readable media storing instructions thereon, which, when executed by the one or more processors, enables the electronic device to execute the method for processing audio and video data as described above.
[0020] The embodiment of the present invention further discloses a computer-readable storage medium, which stores a computer program that enables a processor to execute the above-mentioned method for processing audio and video data.
[0021] The embodiments of the present invention include the following advantages:
[0022] The audio and video data processing solution provided by the embodiment of the present invention is applied to a video conference client, which obtains the terminal information of the target conference terminal from the first conference terminal, and then sends an anti-packet loss coding start instruction to the target conference terminal according to the terminal information, so that the target conference terminal performs anti-packet loss coding processing on the audio and video data of the first code stream according to the anti-packet loss coding start instruction. Then, the processed audio and video data of the first code stream is transmitted to the first conference terminal.
[0023] The embodiment of the present invention can effectively reduce the impact of packet loss on audio and video data transmission by enabling anti-packet loss coding processing at the target conference terminal. Anti-packet loss coding processing can recover data through redundant information in the case of data packet loss, thereby maintaining the continuity and integrity of audio and video. This measure significantly improves the stability and fluency of video conferencing and reduces the jamming phenomenon. By obtaining the terminal information of the target conference terminal and sending an anti-packet loss coding instruction to the target conference terminal according to the terminal information, the target conference terminal can perform adaptive anti-packet loss coding processing according to the specific network conditions. This targeted processing method improves the transmission reliability of audio and video data and ensures that a high transmission quality can be maintained under different network environments. By reducing the negative impact of packet loss on audio and video quality, the overall effect of video conferencing is improved. Users will experience fewer problems such as blurred images and intermittent sounds during the conference, thereby improving the user experience of video conferencing. The anti-packet loss coding strategy can be dynamically adjusted according to the terminal information to flexibly respond to changes in network conditions. This dynamic adjustment mechanism ensures that when the network environment changes, the quality of audio and video data can still be guaranteed through appropriate coding methods, avoiding the decline in conference quality due to sudden changes in network conditions. Through anti-packet loss coding processing, data can be restored when data packets are lost, reducing the need for data retransmission caused by packet loss. This not only reduces network load, but also reduces transmission delays, further improving the real-time and interactive nature of the meeting.
[0024] In summary, the embodiments of the present invention effectively improve the transmission quality and reliability of audio and video data by enabling anti-packet loss coding processing at the target conference terminal, improve the user experience of video conferencing, flexibly respond to changes in network conditions, and reduce the overhead of retransmission. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 is a flowchart of a method for processing audio and video data according to an embodiment of the present invention;
[0026] Figure 2 It is a schematic diagram of the principle of a video stream processing solution for a video conference according to an embodiment of the present invention;
[0027] Figure 3It is a flowchart of a video stream processing solution for a video conference according to an embodiment of the present invention;
[0028] Figure 4 It is a structural block diagram of an audio and video data processing system according to an embodiment of the present invention. DETAILED DESCRIPTION
[0029] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0030] The audio and video data processing scheme proposed in the embodiment of the present invention is intended to improve the stability and fluency of the video conferencing system. Specifically, the scheme is applied on the video conferencing client, by obtaining the terminal information of the target conference terminal from the first conference terminal, and sending an anti-packet loss coding instruction to the target conference terminal based on the terminal information, so that the target conference terminal performs anti-packet loss coding processing on its audio and video data according to the instruction. The audio and video data processed by the anti-packet loss coding is then transmitted back to the first conference terminal. This process effectively reduces the impact of data packet loss on audio and video transmission, improves the reliability and quality of transmission, reduces jamming, and improves user experience. At the same time, it flexibly responds to changes in network conditions by dynamically adjusting the coding strategy, providing an efficient and low-cost solution.
[0031] Reference Figure 1 , shows a flowchart of the steps of a method for processing audio and video data according to an embodiment of the present invention. The method for processing audio and video data can be applied to a video conferencing client, and the method for processing audio and video data can specifically include the following steps:
[0032] Step 101: Acquire terminal information of a target conference terminal from a first conference terminal.
[0033] In the embodiment of the present invention, when a video conference is started, each conference terminal needs to exchange data in real time through a network connection. In order to ensure efficient data transmission, the video conference system (referred to as the system) needs to understand the specific situation of each conference terminal. For example, if a conference terminal is located in an environment with poor network quality, the system can take measures in advance to alleviate potential problems.
[0034] In actual application, the first conference terminal can be the initiator or host terminal of the video conference. It is responsible for managing and coordinating the entire video conference process. When a new conference terminal joins the conference, the first conference terminal will request and receive detailed information of the conference terminal. These detailed information, namely terminal information, are exchanged through a predefined signaling protocol to ensure the accuracy and timeliness of the terminal information. Alternatively, the first conference terminal can be any conference terminal in the video conference.
[0035] For example: suppose that in an international video conference, a conference terminal B located in a remote area needs to join the video conference. After conference terminal B connects to the video conference, the first conference terminal A immediately sends an information request to conference terminal B to obtain its terminal information. After receiving the request, conference terminal B returns detailed information such as its IP address, current network bandwidth, latency, hardware configuration (such as camera and microphone models), and currently used encoding settings. After receiving this data, the first conference terminal A records and analyzes this data to prepare for subsequent steps.
[0036] Step 102: Send an anti-packet loss coding start instruction to the target conference terminal according to the terminal information, so that the target conference terminal performs anti-packet loss coding processing on the audio and video data of the first code stream according to the anti-packet loss coding start instruction.
[0037] In the embodiment of the present invention, anti-packet loss coding is a method for improving data transmission reliability by adding redundant information. When data packets are lost, it can recover data through redundant information, thereby reducing the impact of packet loss on audio and video quality. After analyzing the network status of the first conference terminal, the system can determine whether it is necessary to enable anti-packet loss coding for the target conference terminal and what specific coding strategy to use.
[0038] The process of sending the command to enable anti-packet loss coding involves the use of signaling protocols. The video conferencing client generates a special signaling message containing the command to enable anti-packet loss coding and sends it to the target conference terminal through the network. After receiving the command, the target conference terminal sets up according to the command and starts anti-packet loss coding for the audio and video data.
[0039] Example: Continuing with the example in the previous step, assume that the network condition between conference terminal B and the first conference terminal A is poor, with high latency and unstable bandwidth. The video conference client decides that it is necessary to enable anti-packet loss coding for conference terminal B. The video conference client generates a signaling message to instruct conference terminal B to enable anti-packet loss coding, and sends the message to conference terminal B through the network. After receiving the instruction, conference terminal B starts anti-packet loss coding, processes the audio and video data it transmits, and adds necessary redundant information so that it can recover when data packets are lost.
[0040] Step 103: Transmit the processed audio and video data of the first code stream to the first conference terminal.
[0041] In an embodiment of the present invention, the audio and video data after anti-packet loss coding processing contains additional redundant information for recovery when data is lost. When sending audio and video data, the target conference terminal will send this redundant information in addition to the original audio and video data. After receiving these data, the first conference terminal can recover the data packet based on the redundant information, thereby ensuring the integrity and quality of the audio and video. The transmission of audio and video data still depends on the network, but with anti-packet loss coding, even if the network conditions are poor, the impact of packet loss on audio and video quality can be effectively reduced.
[0042] For example: Assume that conference terminal B has now turned on anti-packet loss encoding processing. It encodes each frame of video data to generate two parts: original data packets and redundant data packets. During the transmission process, if some original data packets are lost, the first conference terminal A can use the corresponding redundant data packets to recover them. For example, conference terminal B sent a video data packet containing 10 data packets, of which 2 data packets were lost during the transmission process. After receiving the remaining 8 data packets and related redundant data, the first conference terminal A successfully recovered the lost 2 data packets using the anti-packet loss algorithm, thereby ensuring the integrity of the entire video frame.
[0043] The audio and video data processing solution provided by the embodiment of the present invention is applied to a video conference client, which obtains the terminal information of the target conference terminal from the first conference terminal, and then sends an anti-packet loss coding start instruction to the target conference terminal according to the terminal information, so that the target conference terminal performs anti-packet loss coding processing on the audio and video data of the first code stream according to the anti-packet loss coding start instruction. Then, the processed audio and video data of the first code stream is transmitted to the first conference terminal.
[0044] The embodiment of the present invention can effectively reduce the impact of packet loss on audio and video data transmission by enabling anti-packet loss coding processing at the target conference terminal. Anti-packet loss coding processing can recover data through redundant information in the case of data packet loss, thereby maintaining the continuity and integrity of audio and video. This measure significantly improves the stability and fluency of video conferencing and reduces the jamming phenomenon. By obtaining the terminal information of the target conference terminal and sending an anti-packet loss coding instruction to the target conference terminal according to the terminal information, the target conference terminal can perform adaptive anti-packet loss coding processing according to the specific network conditions. This targeted processing method improves the transmission reliability of audio and video data and ensures that a high transmission quality can be maintained under different network environments. By reducing the negative impact of packet loss on audio and video quality, the overall effect of video conferencing is improved. Users will experience fewer problems such as blurred images and intermittent sounds during the conference, thereby improving the user experience of video conferencing. The anti-packet loss coding strategy can be dynamically adjusted according to the terminal information to flexibly respond to changes in network conditions. This dynamic adjustment mechanism ensures that when the network environment changes, the quality of audio and video data can still be guaranteed through appropriate coding methods, avoiding the decline in conference quality due to sudden changes in network conditions. Through anti-packet loss coding processing, data can be restored when data packets are lost, reducing the need for data retransmission caused by packet loss. This not only reduces network load, but also reduces transmission delays, further improving the real-time and interactive nature of the meeting.
[0045] In summary, the embodiments of the present invention effectively improve the transmission quality and reliability of audio and video data by enabling anti-packet loss coding processing at the target conference terminal, improve the user experience of video conferencing, flexibly respond to changes in network conditions, and reduce the overhead of retransmission.
[0046] In an exemplary embodiment of the present invention, an implementation method of obtaining the terminal information of the target conference terminal from the first conference terminal is to obtain the terminal identification information, packet loss rate, and coding information of the target conference terminal from the first conference terminal. The terminal identification information includes a unique identifier of the target conference terminal, such as an IP address or a MAC address. This information is essential for the system to identify and distinguish different conference terminals. In a video conference, each conference terminal needs to communicate through a network, and the system must be able to accurately identify each conference terminal in order to perform correct data transmission and processing. Packet loss rate (PLR) is an important indicator for measuring network transmission quality. It indicates the proportion of data packets lost during data transmission. By obtaining the packet loss rate of the target conference terminal, the system can determine the network status of the target conference terminal. If the packet loss rate is high, it means that the network transmission quality is poor, which may cause the loss of video and audio data, thereby affecting the effect of the conference. The coding information includes the audio and video coding format and parameters currently used by the target conference terminal. This information is very important for the system to perform targeted optimization processing. For example, different coding formats have different performances in anti-packet loss capability and transmission efficiency. By obtaining coding information, the system can adjust the coding strategy according to actual conditions to improve transmission quality.
[0047] By acquiring the packet loss rate and coding information, the embodiment of the present invention can determine the network status and take corresponding anti-packet loss measures to improve the transmission reliability of audio and video data and reduce the phenomenon of freezing and distortion. According to the coding information of the target conference terminal, the system can dynamically adjust the coding strategy. For example, when the network condition is poor, anti-packet loss coding can be enabled to improve the success rate of data transmission. By continuously monitoring and acquiring terminal information, the system can flexibly respond to changes in different network environments and improve the adaptability and stability of the entire video conferencing system.
[0048] In an exemplary embodiment of the present invention, an implementation method of sending an anti-packet loss coding instruction to a target conference terminal according to terminal information is to send an instruction to start forward error correction coding to the target conference terminal according to the terminal information. Forward error correction (FEC) coding is a technology that realizes error detection and correction by adding redundant information during data transmission. Specifically, when sending data, the sender not only sends the original data but also sends redundant data, which can help the receiver recover the lost data when the data packet is lost, thereby ensuring the integrity of the data. FEC technology is widely used in various scenarios that require high-reliability data transmission, such as video conferencing, satellite communications, etc. In the previous step, the terminal information of the target conference terminal has been obtained from the first conference terminal. These terminal information include terminal identification information, packet loss rate, coding information, etc. Based on this information, the network status of the target conference terminal and the currently used coding strategy can be judged, so as to decide whether to enable FEC coding. After deciding to enable FEC coding, it is necessary to send an instruction to enable FEC coding to the target conference terminal. This instruction contains specific FEC coding parameters, such as redundancy (indicating the proportion of redundant data), coding algorithm, etc. After receiving the command, the target conference terminal will perform FEC encoding settings according to the parameters in the command, thereby adding redundant information in subsequent data transmission and improving the reliability of data transmission.
[0049] For example: Assume that in an international video conference, the first conference terminal A finds through monitoring and analysis that the packet loss rate of the target conference terminal B is high and the network condition is unstable. In order to ensure the smooth progress of the conference, the video conference client decides to send an instruction to the target conference terminal B to enable FEC encoding. The active information of the target conference terminal B includes but is not limited to: terminal identification information, packet loss rate and encoding information. For example, the IP address of the target conference terminal B is 192.168.1.100, the packet loss rate is 5%, and the current encoding format is H.264. According to the packet loss rate and current encoding information of the target conference terminal B, the video conference client determines that FEC encoding needs to be enabled to improve the reliability of data transmission of the target conference terminal B. The video conference client generates an instruction containing FEC encoding parameters, instructing the target conference terminal B to enable FEC encoding. For example, the instruction contains a redundancy of 20% and a coding algorithm of Reed-Solomon encoding. After receiving the instruction to enable FEC encoding, the target conference terminal B sets FEC encoding according to the parameters in the instruction. The target conference terminal B starts to perform FEC encoding on its audio and video data to increase redundant data.
[0050] The embodiment of the present invention sends an FEC coding start instruction to the target conference terminal according to the terminal information, so that data recovery can be performed when data packets are lost, thereby ensuring the integrity of the data. The impact of packet loss on audio and video quality is effectively reduced, and the reliability of transmission is improved. Through FEC coding, the target conference terminal can still provide continuous audio and video streams when the network condition is poor, reduce the jamming phenomenon caused by packet loss, and improve the user's viewing experience. It can dynamically decide whether to enable FEC coding according to the network conditions and coding strategies of different conference terminals, so as to flexibly respond to different network environments. The video conferencing system can maintain high-quality transmission in various complex network environments. FEC coding can recover data through redundant data when data is lost, reducing the need for data retransmission. This not only reduces the network load, but also reduces transmission delays and improves the real-time nature of the meeting.
[0051] In an exemplary embodiment of the present invention, an implementation method of transmitting the processed audio and video data of the first code stream to the first conference terminal is to send a stream adjustment notification to the video conference server, so that the video conference server transmits the processed audio and video data of the first code stream from the autonomous server to the first conference terminal according to the stream adjustment notification. The stream adjustment notification is a signaling message used to notify the video conference server to adjust and transmit the data stream. Through the stream adjustment notification, the system can dynamically adjust the path and method of the data stream to adapt to the current network conditions and terminal requirements. The stream adjustment notification contains specific instructions and parameters, such as the terminal identification information of the target conference terminal, the terminal identification information of the first conference terminal, the data stream type, the encoding method, etc. After receiving the stream adjustment notification, the video conference server will perform corresponding operations according to the instructions of the notification, instructing the autonomous server to transmit the audio and video data processed by anti-packet loss encoding to the first conference terminal.
[0052] For example: Assume that in an international video conference, the first conference terminal A needs to receive audio and video data processed by anti-packet loss coding from the target conference terminal B. The video conference client generates a stream adjustment notification, which includes the terminal identification information of the target conference terminal B, the data stream type (such as audio and video data), the encoding method (such as FEC encoding), and the transmission path. The stream adjustment notification is sent to the video conference server through the network. After receiving the stream adjustment notification, the video conference server identifies the encoded data to be transmitted and adjusts the data stream according to the instructions in the notification. The video conference server instructs the autonomous server to obtain the encoded data of the target conference terminal B, that is, the audio and video data processed by anti-packet loss coding, and transmits these data to the first conference terminal A according to the instructions in the stream adjustment notification. After receiving the audio and video data processed by anti-packet loss coding, the first conference terminal A decodes and displays the audio and video content. Because the audio and video data are processed by FEC encoding, the first conference terminal A can recover through redundant data when data packets are lost, ensuring the continuity and integrity of the audio and video.
[0053] The embodiment of the present invention adjusts and transmits data streams through a video conferencing server. The system can select the best transmission path according to the current network conditions, reducing data loss and delay caused by network problems. By using FEC coding technology, redundant information is added during data transmission, so that even when the network conditions are poor, it can be restored through redundant data to ensure the quality and continuity of audio and video. By dynamically adjusting the path and mode of data streams, the system can more efficiently utilize network resources, avoid network congestion and transmission bottlenecks, and improve overall transmission efficiency.
[0054] In an exemplary embodiment of the present invention, before obtaining the terminal information of the target conference terminal from the first conference terminal, an implementation method is to add the first conference terminal to the monitoring packet loss list in response to the anti-packet loss start operation; detect whether the conference terminal in the monitoring packet loss list has a packet loss problem; if the first conference terminal has a packet loss problem, perform the operation of obtaining the terminal information of the target conference terminal from the first conference terminal. The anti-packet loss start operation refers to starting the detection and processing mechanism for the packet loss problem in the video conferencing system to improve the reliability of data transmission. By starting the anti-packet loss operation, the system can timely discover and handle the packet loss problem in network transmission to ensure the smooth progress of the meeting. After starting the anti-packet loss operation, the video conferencing client can add the first conference terminal to a list specifically used to monitor packet loss. This list is the monitoring packet loss list, which contains all conference terminals that need to be detected for packet loss. By regularly monitoring the transmission of these conference terminals, the system can timely discover and handle packet loss problems. Regularly detect the conference terminals in the monitoring packet loss list and analyze the quality of their data transmission, including indicators such as packet loss rate, delay and jitter. Through these indicators, it can be judged whether the network status of the conference terminal is stable and whether there is a packet loss problem. When a packet loss problem is detected at the first conference terminal, the operation of obtaining the terminal information of the target conference terminal will be performed. The terminal information includes terminal identification information, packet loss rate, encoding information, etc. By obtaining this information, further measures can be taken to handle packet loss, such as adjusting the encoding strategy, enabling FEC, etc.
[0055] For example: Assume that in a large video conference, the user of the first conference terminal A reports the video and audio freeze phenomenon to the video conference client. The user of the video conference client, that is, the system administrator, starts the anti-packet loss operation and adds the first conference terminal A to the monitoring packet loss list. At this time, the transmission status of the first conference terminal A begins to be monitored. The conference terminals in the monitoring packet loss list are regularly detected to analyze the data transmission quality of each conference terminal. For the first conference terminal A, its packet loss rate, delay, and jitter are detected. If it is found that the packet loss rate of the audio and video data obtained by the first conference terminal A from the target conference terminal B is high, there is a problem with the video and audio data transmission. At this time, the operation of obtaining the terminal information of the target conference terminal B from the first conference terminal A is triggered. The active information of the target conference terminal B includes terminal identification information (IP address, MAC address), current packet loss rate and encoding information (such as H.264 encoding, using FEC). Based on the active information obtained, the video conference client decides to take further anti-packet loss measures, such as adjusting the encoding strategy, enabling or enhancing FEC encoding, to improve the reliability of data transmission.
[0056] The embodiment of the present invention can timely detect packet loss problems and avoid the degradation of video and audio quality caused by packet loss by adding the first conference terminal to the monitoring packet loss list and performing regular detection. After the packet loss problem is discovered, the terminal information of the target conference terminal can be quickly obtained, and corresponding anti-packet loss measures can be taken, such as adjusting the encoding strategy and enabling FEC encoding, so as to improve the reliability of data transmission. Through an effective packet loss detection and processing mechanism, it is possible to reduce freezes and distortions, ensure the smooth progress of video conferencing, and enhance the user's viewing and usage experience. By dynamically monitoring and processing packet loss problems, it is possible to flexibly respond to changes in different network environments and enhance the adaptability and stability of the video conferencing system. By timely detecting and processing packet loss problems, it is possible to avoid the waste of network resources due to data retransmission and improve the utilization efficiency of network resources.
[0057] In an exemplary embodiment of the present invention, after detecting whether a conference terminal in the monitoring packet loss list has a packet loss problem, an implementation method is that if the second conference terminal in the monitoring packet loss list does not have a packet loss problem, the audio and video data of the second code stream that has not been processed by anti-packet loss coding is transmitted to the second conference terminal; wherein the resolution of the audio and video data of the second code stream is greater than the resolution of the audio and video data of the first code stream. In a video conferencing system, the monitoring packet loss list is used to monitor the transmission status of each conference terminal. The packet loss rate, delay, and jitter of the conference terminals in the monitoring packet loss list are regularly detected to determine the stability and quality of their network transmission. When it is detected that the second conference terminal in the monitoring packet loss list (i.e., another conference terminal different from the first conference terminal) does not have a packet loss problem, the transmission mode of the audio and video data will be dynamically adjusted according to the transmission capacity of the second conference terminal. Specifically, the audio and video data of the second code stream of the target conference terminal that has not been processed by anti-packet loss coding can be transmitted to the second conference terminal. The resolution of the audio and video data of the second code stream here is greater than the resolution of the audio and video data of the first code stream, which means providing higher quality audio and video content.
[0058] In actual application, the network status and transmission information of the first conference terminal, the second conference terminal and other conference terminals are obtained, and these conference terminals are added to the monitoring packet loss list. The transmission quality of these conference terminals is regularly checked. The packet loss rate, delay, jitter and other indicators of the conference terminals in the monitoring packet loss list are detected to determine the stability of their network transmission. If the second conference terminal does not have a packet loss problem, the optimization of audio and video data transmission will be considered. For the second conference terminal that does not have a packet loss problem, the video conferencing client transmits the audio and video data of the second code stream of the target conference terminal that has not undergone anti-packet loss encoding processing to the second conference terminal. The resolution of the audio and video data of the second code stream is greater than the resolution of the audio and video data of the first code stream, providing a higher quality audio and video experience.
[0059] For example: Assume that in an enterprise video conference, the user of the first conference terminal A reported that the video quality of the target conference terminal B was poor, and at the same time, the user of the second conference terminal C required higher quality video content. The system administrator obtains the network status and transmission information from the first conference terminal A and the second conference terminal C, and adds them to the monitoring packet loss list. If it is detected that there is a packet loss problem with the first conference terminal A, but the network transmission of the second conference terminal C is very stable and there is no packet loss. In view of the good network status of the second conference terminal C, it is decided to transmit the high-resolution audio and video data of the target conference terminal B that has not been processed by anti-packet loss encoding to the second conference terminal C. The audio and video data received by the second conference terminal C has a higher resolution, providing a clearer and smoother video experience.
[0060] The embodiments of the present invention can dynamically adjust the transmission mode of audio and video data according to the network status of each conference terminal, and flexibly respond to transmission requirements under different network environments. For conference terminals with good network status, higher resolution audio and video data can be transmitted, providing a clearer and smoother audio and video experience, and improving user viewing and usage satisfaction. By dynamically adjusting the transmission mode, it is possible to avoid anti-packet loss encoding processing for all conference terminals, thereby saving system resources and improving transmission efficiency. By providing customized transmission solutions for different conference terminal network conditions, it is possible to reduce freezes and distortions, ensure the smooth conduct of video conferencing, and improve the overall user experience.
[0061] In an exemplary embodiment of the present invention, after transmitting the audio and video data of the first code stream processed by anti-packet loss coding to the first conference terminal, an implementation method is to send a stream switching instruction to the first conference terminal, so that the first conference terminal switches the audio and video data of the first code stream that has not been processed by anti-packet loss coding to the audio and video data of the first code stream that has been processed by anti-packet loss coding according to the stream switching instruction. In video conferencing systems, anti-packet loss coding technology is often used to optimize the transmission stability and quality of audio and video data. When the system detects a change in the network environment or some conference terminals require higher quality audio and video transmission, a signal can be sent to the conference terminal through a stream switching instruction to switch it to a data stream processed by anti-packet loss coding. This method can dynamically adjust the data transmission method according to the real-time network status and terminal requirements to provide a better user experience and video conferencing effect.
[0062] In actual application, the audio and video data of the first code stream that has been processed with anti-packet loss coding are transmitted to the first conference terminal. These audio and video data have been processed with anti-packet loss coding and have strong fault tolerance, and can better cope with packet loss and delay problems during network transmission. When the system detects that the target conference terminal needs to improve the transmission quality or the first conference terminal requires higher quality audio and video, a stream switching instruction is sent to the first conference terminal. The stream switching instruction generally includes adjusting the encoding method and parameters of the data stream so that the first conference terminal can switch the data stream that has not been processed with anti-packet loss coding to the data stream that has been processed with anti-packet loss coding. After the first conference terminal receives the stream switching instruction, it switches the audio and video data of the first code stream that has not been processed with anti-packet loss coding to the audio and video data of the first code stream that has been processed with anti-packet loss coding according to the instruction. This switching can be performed without interrupting the video conference, ensuring that users continue to obtain high-quality audio and video experience in the video conference.
[0063] For example: Assume that in a multinational corporate video conference, the first conference terminal A is located in an area with a relatively poor network environment and may often encounter network packet loss problems, affecting the smoothness of the conference and the quality of audio and video. On the other hand, the second conference terminal B is located in an area with good network conditions and needs to obtain high-quality audio and video transmission. During operation, the system detects that the network conditions of the first conference terminal A are poor. In order to improve its audio and video transmission quality, the system sends a stream switching instruction to the first conference terminal A. After receiving the instruction, the first conference terminal A automatically switches the data stream that has not been processed by anti-packet loss coding to the data stream that has been processed by anti-packet loss coding. Even in the case of high network packet loss, it can still provide stable and clear audio and video content.
[0064] The embodiment of the present invention switches to a data stream that has been processed with anti-packet loss coding, which can significantly reduce audio and video freezes and quality degradation caused by network packet loss, and improves the fluency and viewing experience of video conferencing. Users can continue to obtain high-quality audio and video content during video conferencing, and can ensure that the conference goes smoothly and effectively regardless of the network environment they are in. By dynamically adjusting the coding method of the data stream, it is possible to reasonably utilize system resources while ensuring audio and video quality, thereby improving the efficiency and stability of the system.
[0065] Based on the above description of an embodiment of a method for processing audio and video data, a video stream processing solution for a video conference is introduced below. Figure 2, showing a schematic diagram of the principle of a video stream processing solution for a video conference according to an embodiment of the present invention. The video stream processing solution for the video conference involves a video conference server, a video conference client, and a conference terminal. Among them, the conference terminal may include a service sending terminal and a service receiving terminal. For the conference terminal in the video conference, a network access signaling retransmission mechanism is added. If packet loss occurs at the service receiving terminal, the video conference client may send an FEC switch instruction to the service sending terminal, and the service sending terminal performs FEC encoding on the video stream according to the FEC switch instruction, and then sends the FEC video stream. The service receiving terminal receives the FEC video stream and then displays the FEC video stream.
[0066] Reference Figure 3 , showing a flow chart of a video stream processing solution for a video conference according to an embodiment of the present invention.
[0067] The conference controller in the business turns on the anti-packet loss switch in the video conference client, and drags one or more conference terminals whose packet loss is to be monitored into the monitored packet loss list to enable the packet loss reporting function of the conference terminals.
[0068] When a conference terminal receives audio and video data from a speaker's conference terminal and packet loss occurs, the conference terminal (referred to as a packet loss terminal) reports information such as the terminal number, encoding address, and channel of the speaker's conference terminal to the video conference client.
[0069] The video conference client sends an instruction to enable anti-packet loss encoding to the conference terminal of the speaker.
[0070] The video conference client informs the video conference server, and the autonomous server sends the sub-stream of the speaker's conference terminal to the packet loss terminal.
[0071] The packet loss terminal receives the sub-stream audio and video data from the speaker's conference terminal after anti-packet loss encoding and switches. After the switching, the sub-stream image and audio can be played smoothly without freezing or black screen.
[0072] For conference terminals that receive audio and video data from the speaker's conference terminal without packet loss (referred to as non-packet loss terminals), the speaker's conference terminal sends the main stream audio and video data that has not undergone anti-packet loss encoding to the non-packet loss terminal.
[0073] The embodiment of the present invention can respond to and process the packet loss problem in a timely manner by having the conference controller turn on the anti-packet loss switch and monitor the packet loss situation, thereby significantly reducing the audio and video freezes and black screen phenomena caused by network packet loss. The packet loss terminal can report the packet loss situation to the video conference client in a timely manner, and the video conference client then turns on the anti-packet loss coding to ensure the stability and smoothness of the audio and video data during the transmission process. For the packet loss terminal, once the sub-stream audio and video data that has been anti-packet loss encoded is received from the speaker's conference terminal, it can be quickly switched and played smoothly, avoiding the user's bad experience caused by audio and video freezes during the meeting. This real-time packet loss processing mechanism can ensure the continuity and effect of the meeting. The application of anti-packet loss coding makes the transmission of audio and video data between conference terminals more reliable and efficient. The non-packet loss terminal can receive the main stream audio and video data that has not been anti-packet loss encoded, ensuring the high definition and original quality of the audio and video, while ensuring the real-time and synchronization of the audio and video data, and enhancing the interactive effect between conference terminals. The video conferencing client demonstrates the system's adaptability and intelligent processing capabilities to complex network environments by monitoring the packet loss list and processing the requests of the packet loss terminal in real time. The system can dynamically adjust the encoding method of the data stream according to the actual situation to ensure that each conference terminal can obtain the best audio and video experience according to its network conditions. By processing the packet loss terminal and the non-packet loss terminal differently, the system effectively utilizes the network bandwidth and resources, avoiding unnecessary data retransmission and bandwidth waste. This optimization can improve the overall efficiency of the system and ensure the stability and reliability of the video conferencing system when it is used on multiple terminals and on a large scale.
[0074] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0075] Reference Figure 4 , shows a structural block diagram of an audio and video data processing system according to an embodiment of the present invention. The audio and video data processing system can be applied to a video conference client, and the audio and video data processing system can specifically include the following modules.
[0076] The terminal information acquisition module 41 is used to acquire the terminal information of the target conference terminal from the first conference terminal;
[0077] The anti-packet loss coding module 42 is used to send an anti-packet loss coding start instruction to the target conference terminal according to the terminal information, so that the target conference terminal performs anti-packet loss coding processing on the audio and video data of the first code stream according to the anti-packet loss coding start instruction;
[0078] The audio and video transmission module 43 is used to transmit the processed audio and video data of the first code stream to the first conference terminal.
[0079] In an exemplary embodiment of the present invention, the terminal information acquisition module 41 is used to acquire the terminal identification information, packet loss rate, and coding information of the target conference terminal from the first conference terminal.
[0080] In an exemplary embodiment of the present invention, the anti-packet loss coding module 42 is used to send a forward error correction coding start instruction to the target conference terminal according to the terminal information.
[0081] In an exemplary embodiment of the present invention, the audio and video transmission module 43 is used to send a stream adjustment notification to the video conferencing server, so that the video conferencing server transmits the processed audio and video data of the first code stream from the autonomous server to the first conference terminal according to the stream adjustment notification.
[0082] In an exemplary embodiment of the present invention, the system further comprises:
[0083] A terminal adding module, configured to add the first conference terminal to a monitoring packet loss list in response to an anti-packet loss activation operation before the terminal information acquisition module 41 acquires the terminal information of the target conference terminal from the first conference terminal;
[0084] A packet loss detection module is used to detect whether a conference terminal in the monitoring packet loss list has a packet loss problem;
[0085] The terminal information acquisition module 41 is configured to acquire the terminal information of the target conference terminal from the first conference terminal if a packet loss problem occurs at the first conference terminal.
[0086] In an exemplary embodiment of the present invention, the system further comprises:
[0087] an uncoded module, configured to transmit the unprocessed audio and video data of the second code stream to the second conference terminal if the second conference terminal in the monitored packet loss list does not have a packet loss problem after the packet loss detection module detects whether the conference terminal in the monitored packet loss list has a packet loss problem;
[0088] The resolution of the audio and video data of the second bitstream is greater than the resolution of the audio and video data of the first bitstream.
[0089] In an exemplary embodiment of the present invention, the system further comprises:
[0090] The audio and video stream switching module is used to send a stream switching instruction to the first conference terminal after the audio and video transmission module 43 transmits the processed audio and video data of the first code stream to the first conference terminal, so that the first conference terminal switches the unprocessed audio and video data of the first code stream to the processed audio and video data of the first code stream according to the stream switching instruction.
[0091] As for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0092] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0093] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0094] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0095] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0097] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0098] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0099] The above is a detailed introduction to a method for processing audio and video data and a system for processing audio and video data provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A method for processing audio and video data, characterized in that: Applied to a video conference client, the method includes: Acquiring terminal information of a target conference terminal from the first conference terminal; Sending an anti-packet loss coding start instruction to the target conference terminal according to the terminal information, so that the target conference terminal performs anti-packet loss coding processing on the audio and video data of the first code stream according to the anti-packet loss coding start instruction; The processed audio and video data of the first code stream is transmitted to the first conference terminal.
2. The method according to claim 1, characterized in that The acquiring terminal information of the target conference terminal from the first conference terminal includes: The terminal identification information, the packet loss rate, and the encoding information of the target conference terminal are acquired from the first conference terminal.
3. The method according to claim 1, characterized in that The sending an instruction to enable anti-packet loss encoding to the target conference terminal according to the terminal information includes: A forward error correction coding start instruction is sent to the target conference terminal according to the terminal information.
4. The method according to claim 1, characterized in that: The step of transmitting the processed audio and video data of the first code stream to the first conference terminal includes: A stream adjustment notification is sent to the video conference server, so that the video conference server transmits the processed audio and video data of the first code stream from the autonomous server to the first conference terminal according to the stream adjustment notification.
5. The method according to claim 1, characterized in that Before acquiring the terminal information of the target conference terminal from the first conference terminal, the method further includes: In response to an anti-packet loss enabling operation, adding the first conference terminal to a monitored packet loss list; Detect whether a conference terminal in the monitoring packet loss list has a packet loss problem; If the first conference terminal has a packet loss problem, the operation of acquiring the terminal information of the target conference terminal from the first conference terminal is performed.
6. The method according to claim 5, characterized in that After detecting whether a packet loss problem occurs at a conference terminal in the monitored packet loss list, the method further includes: If the second conference terminal in the monitoring packet loss list does not have a packet loss problem, transmitting the unprocessed audio and video data of the second code stream to the second conference terminal; The resolution of the audio and video data of the second bitstream is greater than the resolution of the audio and video data of the first bitstream.
7. The method according to claim 1, characterized in that After transmitting the processed audio and video data of the first code stream to the first conference terminal, the method further includes: A stream switching instruction is sent to the first conference terminal, so that the first conference terminal switches the unprocessed audio and video data of the first code stream to the processed audio and video data of the first code stream according to the stream switching instruction.
8. A system for processing audio and video data, characterized in that: Applied to a video conference client, the system includes: A terminal information acquisition module, used to acquire terminal information of a target conference terminal from the first conference terminal; an anti-packet loss encoding module, configured to send an anti-packet loss encoding start instruction to the target conference terminal according to the terminal information, so that the target conference terminal performs anti-packet loss encoding processing on the audio and video data of the first code stream according to the anti-packet loss encoding start instruction; The audio and video transmission module is used to transmit the processed audio and video data of the first code stream to the first conference terminal.
9. An electronic device, characterized in that: include: one or more processors; and One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, enable the electronic device to execute the method for processing audio and video data as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer program stored therein enables the processor to execute the method for processing audio and video data as described in any one of claims 1 to 7.
Citation Information
Cited By
Feature adaptive hierarchical protection and packet loss repair method for real-time video session
CN122534227A
