Audio data processing method, server, and program product

Abstract

Embodiments of the present application provide an audio data processing method, a server and a program product. The method is applied to a server, and the method comprises: performing redundant coding processing on to-be-processed audio data to obtain coded audio data, in a case where it is determined based on network state information that the redundant coding process is to be performed; and performing a first redundancy processing operation on the coded audio data to obtain processed audio data. The method significantly improves the quality of audio transmission in a weak network environment, reduces audio delay, freezing and distortion, and improves user experience.

Description

Technical Field

[0001] This application relates to communication technology, and more particularly to an audio data processing method, server, and program product. Background Technology

[0002] With the rapid development of communication and cloud computing technologies, cloud desktops, as a technology that virtualizes desktop environments and provides them to end users via the network, have been widely adopted. Cloud desktops allow users to access their virtual desktops from any device within a network environment and enjoy an operating experience almost identical to that of a local desktop. This technology is widely used in education, finance, healthcare, and many other industries, demonstrating unique advantages, especially in scenarios such as remote work and virtual classrooms.

[0003] However, cloud desktop applications are highly dependent on network quality. Especially in weak network environments—where packet loss is severe, latency and jitter are high, and network bandwidth is limited—the user experience of cloud desktop services is often significantly impacted. Audio transmission is one of the key factors affecting user experience. As a fundamental means of communication, the quality of audio directly determines the user's interactive experience in scenarios such as virtual meetings and online teaching. However, weak network environments can lead to the loss, delay, and out-of-order delivery of audio data packets, resulting in distorted decoded audio, affecting call quality, and ultimately preventing the effective transmission of information. Therefore, ensuring audio quality in weak network environments for cloud desktops is a pressing issue that needs to be addressed. Summary of the Invention

[0004] This application provides an audio data processing method, server, and program product to improve the audio transmission quality in weak network environments.

[0005] In a first aspect, embodiments of this application provide an audio data processing method applied to a server, the method comprising:

[0006] If the redundant encoding process is determined to be executed based on network status information, the audio data to be processed is subjected to redundant encoding to obtain encoded audio data.

[0007] The encoded audio data is subjected to a first redundancy processing operation to obtain the processed audio data.

[0008] In one possible implementation, the network status information indicates network congestion and packet loss status; before performing redundant encoding on the audio data to be processed to obtain the encoded audio data, the method further includes:

[0009] The decision to execute the redundancy coding process is based on the network status information.

[0010] In one possible implementation, the network state information includes at least uplink packet loss rate information. Determining to perform the redundancy coding process based on the network state information includes:

[0011] If the uplink packet loss rate is greater than or equal to the packet loss rate threshold, the redundant coding process is determined to be executed.

[0012] In one possible implementation, the redundancy encoding process performed on the audio data to be processed to obtain encoded audio data includes:

[0013] For each audio frame in the audio data, the main bitstream and the sub-bitstream of the audio frame are encoded according to a preset first redundancy ratio to obtain an encoded audio frame; the first redundancy ratio is the proportion of the sub-bitstream in the audio frame.

[0014] In one possible implementation, the method further includes:

[0015] If, based on network status information, it is determined that the redundant encoding process should not be performed, the audio data to be processed is subjected to non-redundant encoding to obtain the encoded audio data.

[0016] In one possible implementation, the non-redundant encoding process performed on the audio data to be processed to obtain encoded audio data includes:

[0017] For each audio frame in the audio data, all the audio data of the audio frame is used as the main bitstream, and the main bitstream is encoded to obtain the encoded audio frame.

[0018] In one possible implementation, performing a first redundancy processing operation on the encoded audio data to obtain processed audio data includes:

[0019] For any of the encoded audio frames, determine whether the previous audio frame of the encoded audio frame contains a sub-bitstream;

[0020] If the previous audio frame contains a sub-bitstream, the sub-bitstream in the previous audio frame is appended to the main bitstream of the encoded audio frame to obtain the processed audio frame, and the flag bit in the processed audio frame is set to a first value.

[0021] If the previous audio frame does not contain a sub-bitstream, a processed audio frame is generated based on the main bitstream of the encoded audio frame, and the flag bit in the processed audio frame is set to a second value.

[0022] In one possible implementation, the method further includes:

[0023] Based on the network status information, determine whether to perform the second redundancy processing operation;

[0024] If it is determined that the second redundancy processing operation will be performed, a target redundancy processing method supported by the terminal device communicating with the server is determined, and the second redundancy processing operation is performed on the processed audio data using the target redundancy processing method.

[0025] In one possible implementation, the network state information includes at least round-trip delay; the step of determining whether to perform a second redundancy processing operation based on the network state information includes:

[0026] If the round-trip delay is greater than or equal to the delay threshold, the second redundancy processing operation is performed.

[0027] In one possible implementation, the target redundancy processing method includes: a first redundancy processing method and a second redundancy processing method; the device performance of the terminal device supporting the second redundancy processing method is higher than that of the terminal device supporting the first redundancy processing method; determining the target redundancy processing method supported by the terminal device communicating with the server includes:

[0028] If at least one of the multiple terminal devices communicating with the server does not support the second redundancy processing method, the first redundancy processing method is determined to be the target redundancy processing method.

[0029] If multiple terminal devices communicating with the server all support the second redundancy processing method, then the second redundancy processing method is determined to be the target redundancy processing method.

[0030] In one possible implementation, the first redundancy processing method is the target redundancy processing method, and the second redundancy processing operation performed on the processed audio data using the target redundancy processing method includes:

[0031] For any target audio frame in the audio data, the first n audio frames of the target audio frame are appended to the target audio frame to obtain an aggregated data packet; the length of the next audio frame is set between adjacent audio frames in the data packet; n is obtained according to the second redundancy ratio, and n is an integer greater than 0.

[0032] In one possible implementation, the second redundancy ratio is obtained based on round-trip delay and a preset packet loss tolerance threshold.

[0033] In one possible implementation, the second redundancy processing method is the target redundancy processing method, and the step of performing the second redundancy processing operation on the processed audio data using the target redundancy processing method includes:

[0034] For any first audio frame group in the audio data, at least one redundant audio frame is generated corresponding to the first audio frame group according to the third redundancy ratio; the first audio frame group includes at least one original audio frame.

[0035] The at least one redundant audio frame and the second audio frame group are aggregated to obtain an aggregated data packet; the second audio frame group is at least one audio frame group following the first audio frame group, and the length of the next audio frame is set between adjacent audio frames in the aggregated data packet.

[0036] In one possible implementation, the third redundancy ratio is obtained based on the number of redundant frames and the number of original audio frames in the first audio frame group, wherein the number of redundant frames is obtained based on the number of original audio frames in the first audio frame group, the uplink packet loss rate, and the no-packet-loss threshold.

[0037] In one possible implementation, the aggregated data packet has a header, which includes at least one of the following: data packet type, timestamp, sequence number, redundancy flag, group number, group size, and the number of redundant audio frames corresponding to the first audio frame group; the redundancy flag is used to indicate whether the data packet contains redundant data.

[0038] In one possible implementation, before performing redundant encoding on the audio data to be processed to obtain the encoded audio data, the method further includes:

[0039] Based on the audio acquisition parameters and acquisition driving method supported by the server, audio data is acquired and written into the buffer.

[0040] Audio data is extracted from the buffer according to preset audio format parameters to obtain the audio data to be processed; the audio format parameters include at least one of the following: frame length, sampling rate, number of channels, and bit depth.

[0041] In one possible implementation, before acquiring audio data based on audio acquisition parameters and acquisition driving mode, the method further includes:

[0042] The server is initialized, and the audio acquisition parameters include at least one of the following: sampling rate, number of channels, and bit depth;

[0043] The acquisition driving method supported by the server is determined, and the acquisition driving method includes at least one of the following: time-driven method or event-driven method.

[0044] In one possible implementation, before determining whether to perform the redundancy coding process based on the network state information, the method further includes:

[0045] Obtain the network status information;

[0046] The network status information is smoothed.

[0047] In one possible implementation, smoothing the network state information includes:

[0048] The uplink packet loss rate in the network status information is smoothed using an exponential filtering algorithm;

[0049] The round-trip delay in the network status information is smoothed using a percentile filtering algorithm.

[0050] In one possible implementation, after performing the first redundancy processing operation on the encoded audio data to obtain processed audio data, or after performing the second redundancy processing operation on the processed audio data using the target redundancy processing method, the method further includes:

[0051] The processed audio data is backed up and transmitted through multiple available transmission channels.

[0052] Secondly, embodiments of this application provide an audio data processing apparatus, comprising:

[0053] The processing module is used to perform redundant encoding on the audio data to be processed when the redundant encoding process is determined to be executed based on network status information, so as to obtain the encoded audio data.

[0054] The processing module is further configured to perform a first redundancy processing operation on the encoded audio data to obtain processed audio data.

[0055] Thirdly, embodiments of this application provide a server, including: a memory and a processor;

[0056] The memory stores computer-executed instructions;

[0057] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0058] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0059] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0060] The audio data processing method, server, and program product provided in this application can dynamically determine whether to execute a redundant encoding process by adjusting the redundancy strategy based on network status information. When it is determined to execute the encoding process based on network status information, the audio data to be processed is subjected to redundant encoding to obtain encoded audio data. Furthermore, a first redundancy processing operation is performed on the encoded audio data, that is, redundant information is added to the encoded audio data, which makes the decoding accuracy higher, improves the audio quality in weak network environments, improves the reliability of audio data transmission, reduces latency and stuttering problems in audio data transmission, and improves the user experience in unstable network environments. Attached Figure Description

[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0062] Figure 1 A schematic diagram illustrating a scenario for the audio data processing method provided in this application;

[0063] Figure 2 Flowchart of the audio data processing method provided in this application Figure 1 ;

[0064] Figure 3a Flowchart of the audio data processing method provided in this application Figure 2 ;

[0065] Figure 3b Flowchart 3 of the audio data processing method provided in this application;

[0066] Figure 4 Flowchart of the audio data processing method provided in this application Figure 4 ;

[0067] Figure 5 Flowchart of the audio data processing method provided in this application Figure 5 ;

[0068] Figure 6Schematic diagram of the audio data processing method provided in this application Figure 1 ;

[0069] Figure 7 Schematic diagram of the audio data processing method provided in this application Figure 2 ;

[0070] Figure 8 Flowchart of the audio data processing method provided in this application Figure 5 ;

[0071] Figure 9 A schematic diagram of the audio data processing device provided in this application;

[0072] Figure 10 This is a schematic diagram of the server structure provided in this application.

[0073] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0074] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0075] First, let me explain the terms used in this application:

[0076] Deinitialization is a series of operations performed when an object or system is destroyed. Its main purpose is to clean up the resources occupied by the object or system and release the allocated memory.

[0077] Currently, the following technical solutions are mainly used to ensure audio quality in weak network scenarios:

[0078] 1. Redundant Encoding of Data (RED)

[0079] By adding redundant information to data packets, lost audio data frames can be recovered at the receiving end based on this redundancy. The advantage of this method is that it can reduce the impact of packet loss on audio quality to some extent; the disadvantage is that it increases terminal resource overhead and network bandwidth consumption.

[0080] 2. Automatic Repeat Request (ARQ) mechanism

[0081] When the receiving end detects lost audio data, it initiates a retransmission request to the sending end. The advantage is that it can selectively retransmit data packets that do not arrive as expected, thus ensuring data integrity to the greatest extent. The disadvantage is that the delay caused by retransmission can significantly affect the real-time performance of the audio, especially in high-latency network environments, resulting in a poor user experience.

[0082] 3. Packet Loss Concealment (PLC)

[0083] Techniques based on statistical or deep learning models are used to predict and compensate for lost data. The advantage is that this technique can generate lost data packets "out of thin air" from existing data packets, mitigating the impact of packet loss on audio quality to some extent. The disadvantage is that the compensation effect depends heavily on the accuracy of the model, and the effect is very limited, especially in cases of continuous packet loss. Using deep learning models also significantly increases the computational burden on the terminal.

[0084] 4. Jitter Buffer (JB)

[0085] Jitter buffering is an important technique for ensuring audio quality in weak network environments. It temporarily stores data packets at the receiving end to smooth out latency and jitter during network transmission. Simultaneously, the buffer allows for retransmission requests when data loss is detected, recovering packets that did not arrive as expected. The advantage is that it balances network jitter, stably providing continuous audio data to subsequent modules, thereby reducing stuttering and interruptions during audio playback and improving the overall audio quality experience. The disadvantage is that the buffer increases link latency, affecting the real-time performance of cloud desktop audio.

[0086] The specific application scenario of this application can be an audio data processing method in a cloud desktop scenario to optimize audio playback performance.

[0087] In some embodiments, a cloud desktop is a flexible virtualized desktop environment provided by a cloud desktop server to a terminal device, which obtains audio data by accessing the cloud desktop server.

[0088] Figure 1 A schematic diagram of the scenario for the audio data processing method provided in this application, such as... Figure 1 As shown, the scenario includes a server 11 and at least one terminal device 12. The server 11 and the terminal device 12 are connected via a network. Optionally, the terminal device can be a personal computer, a mobile phone, a tablet computer, etc.

[0089] Optionally, the methods of this application embodiment can also be applied to other network environments, including but not limited to Bluetooth, mobile networks, high-latency cross-border networks, and satellite communications. Furthermore, some of the technologies involved in the solutions of this application embodiment can be directly applied to data transmission of other media forms (including but not limited to video, Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR)) to enhance overall transmission quality.

[0090] Based on the above scenarios, it can be seen that in related technologies, weak network environments may lead to the loss, delay, and out-of-order delivery of audio data packets, resulting in audio distortion after decoding, affecting call quality, and consequently preventing information from being effectively transmitted. Therefore, how to ensure audio quality in a weak network environment on a cloud desktop has become an urgent problem to be solved.

[0091] The audio data processing method provided in this application performs redundant encoding processing based on network status information when it is determined to perform a redundant encoding process, and performs a first redundant processing operation after encoding, which can improve the audio transmission quality and user experience in weak network environments.

[0092] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0093] Figure 2 Flowchart of the audio data processing method provided in this application Figure 1 The method in this embodiment is applied to a server. For example... Figure 2 As shown, the method includes:

[0094] S201. If the redundant encoding process is determined based on the network state information, the audio data to be processed is subjected to redundant encoding to obtain the encoded audio data.

[0095] Specifically, before transmitting audio data to the terminal device, the server performs redundancy processing on the audio data. Based on the network status information, it determines to perform redundancy encoding on the audio data to be processed. Redundancy encoding means adding redundant information during the encoding process of the audio data.

[0096] In this scheme, the decision to perform redundant coding can be dynamically determined based on network status information. If it is determined that redundant coding should be performed, then redundant coding is carried out. This means that the redundant coding decision is made dynamically based on the network status. For example, network status information is used to indicate network congestion and packet loss.

[0097] S202. Perform the first redundancy processing operation on the encoded audio data to obtain the processed audio data.

[0098] Specifically, the encoded audio data undergoes a first redundancy processing operation, which involves performing a first redundancy processing operation on the real-time audio bitstream. For example, redundant information can be added to certain audio frames to obtain the processed audio data.

[0099] Optionally, the first redundancy processing operation refers to further redundancy processing of the encoded audio data, that is, the processing operation of adding redundant information.

[0100] The audio data processing method provided in this application adjusts the redundancy strategy according to network status information, which can dynamically determine whether to execute the redundancy encoding process. When it is determined to execute the encoding process based on the network status information, the audio data to be processed is subjected to redundancy encoding to obtain encoded audio data. Furthermore, a first redundancy processing operation is performed on the encoded audio data, that is, redundant information is added to the encoded audio data, which makes the decoding accuracy higher, improves the audio quality in weak network environments, improves the reliability of audio data transmission, reduces latency and stuttering problems in audio data transmission, and improves the user experience in unstable network environments.

[0101] In some embodiments, network status information indicates network transmission congestion and packet loss status, such as... Figure 3a As shown, before S201, it also includes:

[0102] S200. Determine whether to execute the redundancy coding process based on the network status information.

[0103] Specifically, the system can dynamically determine whether to perform a redundant coding process based on network status information. If it is determined to perform a redundant coding process, then redundant coding is performed.

[0104] Optionally, the network status information includes information on at least one of the following network parameters: uplink packet loss rate, round-trip delay, and uplink bandwidth.

[0105] In some embodiments, before determining whether to perform the redundancy coding process based on the network state information, the method further includes:

[0106] Obtain the network status information;

[0107] The network status information is smoothed.

[0108] Specifically, network status information may include information on multiple network parameters, including but not limited to at least one of the following: uplink packet loss rate (loss_rate), round-trip time (RTT), and uplink bandwidth.

[0109] Optionally, the acquired network status information can be smoothed to make the processed data more accurate and reliable.

[0110] For example, during server system initialization, the upper-layer processing component registers a callback function with the lower-layer transmission component. This callback function is used to receive feedback on network status information. Every set time interval (e.g., 250ms), the lower-layer transmission component uses the callback mechanism to feed back the collected network status information for each transmission channel to the upper-layer processing component. The upper-layer processing component then performs redundant processing on the audio data.

[0111] In some embodiments, the network state information is smoothed, which can be achieved in the following ways:

[0112] The uplink packet loss rate in the network status information is smoothed using an exponential filtering algorithm;

[0113] The round-trip delay in the network status information is smoothed using a percentile filtering algorithm.

[0114] Specifically, smoothing algorithms include, but are not limited to, exponential filtering and percentile filtering algorithms, which are used to smooth the real-time network state information.

[0115] Optionally, the uplink packet loss rate can be smoothed using an exponential filtering algorithm.

[0116] For example, the uplink packet loss rate is set to α = 0.9999 (time constant is...). Exponential smoothing of ) is expressed by the following formula:

[0117] smoothed_loss t =α exp ×smoothed_loss t-1 +(1-α exp )×loss_rate t

[0118] Among them, smoothed_loss tLet be the packet loss rate at time t after smoothing, and let be the smoothed_loss. t-1 The loss rate is the smoothed packet loss rate at time t-1. t Let be the estimated uplink packet loss rate obtained from the lower-layer transmission component at time t, and exp be the time interval (milliseconds) between time t and time t-1. At the initial time (time 0), smoothed_loss0 = loss_rate0, that is, the smoothed packet loss smoothed_loss0 is equal to the estimated uplink packet loss rate loss_rate0 obtained from the lower-layer transmission component at this time.

[0119] Optionally, the round-trip delay can be smoothed using a percentile filtering algorithm.

[0120] For example, percentile filtering with a window size of W (e.g., W = 20) is applied to the round-trip delay. Specifically, for the round-trip delay estimates rtt0, rtt1, rtt2, ... obtained from the lower-layer transmission components at different times, the data contained in the window at time t is S. t ={rtt t-W+1 ,rtt t-W+2 ,…,rtt t}. For that moment, the calculation window S t The p-th percentile of the data (e.g., p = 95%) is used as the smoothed round-trip delay at that moment. t The formula is:

[0121] smoothed_rtt t =Percentile(S t ,p)

[0122] Where Percentile(·) is a function that, for a given window S, t Sort the data in ascending order and return the value at position p in the sorted data.

[0123] In the above embodiments, by smoothing the network status information, the referenced network status becomes more accurate and has higher reliability.

[0124] In some embodiments, the network state information includes at least uplink packet loss rate information. Determining whether to perform the redundancy coding process based on the network state information includes:

[0125] If the uplink packet loss rate is greater than or equal to the packet loss rate threshold, the redundant coding process is determined to be executed.

[0126] Specifically, when the uplink packet loss rate is smoothed_loss tGreater than or equal to the packet loss rate threshold (e.g., λ) encoder =0), i.e., smoothed_loss t ≥λ encoder The process of performing redundant coding is determined, meaning that redundant coding is allowed; otherwise, redundant coding is not performed.

[0127] Optionally, the first redundancy ratio of the redundant coding can be a preset fixed ratio (such as 25%). For example, when the actual coding bit rate is 64kbps, 48kbps is allocated for the main bitstream coding and 16kbps is allocated for redundancy.

[0128] In some embodiments, such as Figure 3a As shown, the method also includes:

[0129] S201a. If it is determined based on network state information that a redundant encoding process will not be performed, the audio data to be processed is subjected to non-redundant encoding to obtain the encoded audio data.

[0130] Specifically, if it is determined based on network status information that a redundant encoding process should not be performed, the audio data to be processed is subjected to non-redundant encoding. Non-redundant encoding means that no redundant information needs to be added during the encoding process of the audio data.

[0131] In some embodiments, S201 can be implemented in the following manner:

[0132] For each audio frame in the audio data, the main bitstream and the sub-bitstream of the audio frame are encoded according to a preset first redundancy ratio to obtain an encoded audio frame; the first redundancy ratio is the proportion of the sub-bitstream in the audio frame.

[0133] Specifically, the audio frame to be encoded is acquired, and based on the redundancy encoding on / off state determined in step S201, it is determined whether to perform redundancy encoding:

[0134] like Figure 3b As shown, when performing redundant encoding, the encoder simultaneously encodes the main bitstream and the sub-bitstream of the audio frame, that is, the main bitstream and the sub-bitstream of the audio frame are encoded separately.

[0135] The actual bitrate allocation between the main stream and the secondary stream is based on the first redundancy ratio determined in the preceding steps. The first redundancy ratio refers to the proportion of the secondary stream in the total bitrate.

[0136] Optionally, the main stream generally accounts for a larger proportion of the total bitstream than the secondary stream.

[0137] In some embodiments, S201a can be implemented in the following manner:

[0138] For each audio frame in the audio data, all the audio data of the audio frame is used as the main bitstream, and the main bitstream is encoded to obtain the encoded audio frame.

[0139] Specifically, such as Figure 3b As shown, when no redundant encoding is performed, the audio frame is encoded using only the main bitstream through the encoder. That is, all audio data of the audio frame is used as the main bitstream for encoding, and the main bitstream accounts for 100% of the total bitstream.

[0140] In some embodiments, performing a first redundancy processing operation on the encoded audio data to obtain processed audio data includes:

[0141] For any of the encoded audio frames, determine whether the previous audio frame of the encoded audio frame contains a sub-bitstream;

[0142] If the previous audio frame contains a sub-bitstream, the sub-bitstream in the previous audio frame is appended to the main bitstream of the encoded audio frame to obtain the processed audio frame, and the flag bit in the processed audio frame is set to a first value.

[0143] If the previous audio frame does not contain a sub-bitstream, a processed audio frame is generated based on the main bitstream of the encoded audio frame, and the flag bit in the processed audio frame is set to a second value.

[0144] Specifically, such as Figure 3b As shown, for any encoded audio frame, check whether the previous audio frame contains a sub-bitstream: if the previous audio frame contains a sub-bitstream, then append the sub-bitstream of the previous audio frame to the main bitstream of the current audio frame to obtain the processed audio frame, that is, the sub-bitstream of the previous audio frame is used as redundant information, and the flag bit of the processed audio frame is set to the first value, such as "true"; if the previous audio frame does not contain a sub-bitstream, then only the main bitstream of the current audio frame is output, that is, the processed audio frame is formed only based on the main bitstream of the current audio frame, and the flag bit of the processed audio frame is set to the second value, such as "false".

[0145] In the above implementation, by determining whether the previous audio frame contains a sub-bitstream, and then performing frame assembly based on different methods, if a sub-bitstream is contained, the sub-bitstream of the previous audio frame is used as redundant information for frame assembly; if no sub-bitstream is contained, only the main bitstream of the current audio frame is output. Through redundancy processing, the audio quality in a weak network environment of the cloud desktop can be improved, thereby improving the reliability of cloud desktop audio transmission.

[0146] In some embodiments, such as Figure 4 As shown, the method also includes:

[0147] S401. Based on the network status information, determine whether to perform the second redundancy processing operation;

[0148] S402. If it is determined that the second redundancy processing operation will be performed, the target redundancy processing method supported by the terminal device communicating with the server is determined, and the second redundancy processing operation is performed on the processed audio data using the target redundancy processing method.

[0149] Specifically, based on network status information, it can also be determined whether to perform a second redundancy processing operation. Optionally, the order of steps S401 and S200 is not important.

[0150] If it is determined that a second redundancy processing operation will be performed, the target redundancy processing method supported by the terminal device communicating with the server is determined. Optionally, a redundancy processing method supported by multiple terminal devices communicating with the server is determined. Optionally, different redundancy processing methods have different requirements for device performance.

[0151] Furthermore, a second redundancy processing operation is performed on the audio data obtained from the first redundancy processing operation using the target redundancy processing method.

[0152] If it is determined that a second redundancy processing operation will not be performed, the encoded audio data can be transmitted.

[0153] Optionally, the second redundancy processing operation refers to performing redundancy processing again after the first redundancy processing operation, adding redundant information. This can improve the decoding accuracy and audio transmission quality in weak network environments. Optionally, the specific implementation methods of the first and second redundancy processing operations are different.

[0154] In the above embodiments, multiple redundancy schemes are combined and applied, such as redundant coding, first redundancy processing operation, and second redundancy processing operation, to improve the reliability of cloud desktop audio transmission and solve the problems of poor audio quality and large latency in cloud desktop weak network environments.

[0155] In some embodiments, the network state information includes at least round-trip delay information, and determining whether to perform a second redundancy processing operation based on the network state information includes:

[0156] If the round-trip delay is greater than or equal to the delay threshold, the second redundancy processing operation is performed.

[0157] Specifically, when the round-trip delay is smoothed_rtt t Greater than or equal to the delay threshold (e.g., λ) red =40), i.e., smoothed_rtt t≥λ red If the second redundancy processing operation is to be performed, then the second redundancy processing operation is allowed; otherwise, the second redundancy processing operation is not allowed.

[0158] In some embodiments, the target redundancy processing method includes: a first redundancy processing method and a second redundancy processing method; the device performance of the terminal device supporting the second redundancy processing method is higher than that of the terminal device supporting the first redundancy processing method; determining the target redundancy processing method supported by the terminal device communicating with the server includes:

[0159] If at least one of the multiple terminal devices communicating with the server does not support the second redundancy processing method, the first redundancy processing method is determined to be the target redundancy processing method.

[0160] If multiple terminal devices communicating with the server all support the second redundancy processing method, then the second redundancy processing method is determined as the target redundancy processing method.

[0161] Specifically, such as Figure 5 As shown, based on the device performance of the terminal device, the redundancy processing method supported by the terminal device is determined, i.e., a redundancy processing method decision is made. Device performance can be determined, for example, based on the processor architecture and model. For instance, high-performance terminal devices (e.g., those using x86 or ARM architecture and not listed in the blacklist) support the first redundancy processing method (…). Figure 5 Method 1 (abbreviated as Method 1) and the second redundancy processing method ( Figure 5 (referred to as Method 2 in Chinese), while low-performance terminal devices only support the first redundancy processing method.

[0162] Optionally, during network communication, the second redundancy processing method is selected only if all participating terminal devices support it; otherwise, it automatically reverts to the first redundancy processing method. Optionally, the terminal devices negotiate through lower-layer transmission components to determine the redundancy processing method commonly supported by all participating terminal devices.

[0163] In the above embodiments, a suitable redundancy processing method is automatically selected for terminal devices with different performance levels, which provides good compatibility with devices of different performance levels.

[0164] In some embodiments, the first redundancy processing method is a target redundancy processing method, and the second redundancy processing operation performed on the processed audio data using the target redundancy processing method includes:

[0165] For any target audio frame in the audio data, the first n audio frames of the target audio frame are appended to the target audio frame to obtain an aggregated data packet; the length of the next audio frame is set between adjacent audio frames in the data packet; n is obtained according to the second redundancy ratio, and n is an integer greater than 0.

[0166] Specifically, by using the first redundancy processing method, multiple audio frames can be aggregated by appending the first n audio frames to the current audio frame to obtain the aggregated data packet.

[0167] To reduce decoding complexity and improve decoding accuracy, two adjacent audio frames are separated by the length of the audio frame. For example, the length of the next audio frame is set between adjacent audio frames. Optionally, the length of the audio frame can be set before the first audio frame in the data packet, or a packet header can also be set.

[0168] Optionally, the second redundancy ratio is obtained based on the round-trip delay and a preset packet loss tolerance threshold.

[0169] For example, the number of data packets sent at any given time is determined based on round-trip delay and packet loss tolerance threshold, and a second redundancy ratio is determined based on this number of packets sent.

[0170] For example, the number of data packets sent at time t (including the original data) is count. t The choice is to satisfy The smallest positive integer, λ loss The preset packet loss tolerance threshold, such as λ. loss =0.08. The second redundancy ratio is expressed as

[0171] Optionally, the number of transmissions can be further limited by preset minimum and maximum values.

[0172] In the above implementation, data frames are merged using frame aggregation technology, which reduces the amount of data transmitted over the network without exacerbating the impact of packet loss, thereby reducing bandwidth consumption and improving bandwidth utilization.

[0173] For example, for lengths L respectively t-2 ,L t-1 ,L t ,L t+1 ,L t+2 ,L t+3 The input data packet P of ... t-2 ,P t-1 ,P t ,P t+1 ,P t+2 ,P t+3..., based on the second redundancy ratio decided in the preceding steps and the number of data packets sent (count) t-2 count t-1 count t count t+1 count t+2 count t+3 ..., copy each package separately (count) t-2 -1, count t-1 -1, count t -1, count t+1 -1, count t+2 -1, count t+3 -1, ... times, and appended sequentially to several consecutive audio frames, separated by the length of the audio frame. Taking a data packet sending count of 3 as an example (in actual use, this value will dynamically change depending on network conditions and the second redundancy ratio), the resulting output data packet is as follows: Figure 6 As shown. This redundancy handling method does not increase latency when the network is good and there is no packet loss. When packet loss occurs, the maximum increase in latency is frame_length_ms × (count). max -1), where frame_length_ms is the length of the audio frame within each input data packet, and count max This represents the maximum number of data packets that can be sent.

[0174] In some embodiments, such as Figure 5 As shown, the second redundancy processing method is a target redundancy processing method. The step of performing the second redundancy processing operation on the processed audio data using the target redundancy processing method includes:

[0175] For any first audio frame group in the audio data, at least one redundant audio frame is generated corresponding to the first audio frame group according to the third redundancy ratio; the first audio frame group includes at least one original audio frame.

[0176] The at least one redundant audio frame and the second audio frame group are aggregated to obtain an aggregated data packet; the second audio frame group is at least one audio frame group following the first audio frame group, and the length of the next audio frame is specified between adjacent audio frames in the aggregated data packet.

[0177] Specifically, the size of the first audio frame group can be preset or predefined by the protocol. The first audio frame group includes at least one original audio frame. The original audio frames are encoded in units of the first audio frame group to obtain redundant audio frames. The redundant audio frames are then appended to the original audio frames in subsequent audio frame groups to obtain the aggregated data packets.

[0178] Optionally, the number of second audio frame groups is one. For any original audio frame in the second audio frame group, at least one redundant audio frame is appended to the original audio frame to obtain an aggregated data packet. The length of the redundant audio frame is provided between the original audio frame and the redundant audio frame in the aggregated data packet.

[0179] Optionally, the redundant audio frames are appended sequentially after the original audio frames in the second audio frame group in the order they appear.

[0180] In the above embodiments, audio frames are merged by frame aggregation technology (such as cross-group frame aggregation), which reduces the amount of data transmitted over the network without aggravating the impact of packet loss, thereby reducing bandwidth consumption and improving bandwidth utilization.

[0181] Optionally, the third redundancy ratio is obtained based on the number of redundant frames and the number of original audio frames in the first audio frame group, wherein the number of redundant frames is obtained based on the number of original audio frames in the first audio frame group, the uplink packet loss rate, and the no-packet-loss threshold.

[0182] Optionally, the aggregated data packet has a header, which includes at least one of the following: data packet type, timestamp, sequence number, redundancy flag, group number, group size, and the number of redundant audio frames corresponding to the first audio frame group; the redundancy flag is used to indicate whether the data packet contains redundant data.

[0183] For example, the size of each audio frame group (the number of original audio frames in each group) is original_count (e.g., 5), and at time t, each group generates recovery_count. t A completely new redundant audio frame. recovery_count t Take the smallest positive integer that satisfies the following formula:

[0184] send_count=origional_count+recovery_count t ;

[0185]

[0186] Where send_count is the number of audio frames sent by each packet to the lower-level transmission component, C represents the number of combinations, and the no-loss threshold λ is... noloss =0.98. The third redundancy ratio is expressed as

[0187] Optionally, the value `recovery_count` can also be adjusted using preset minimum and maximum values ​​for the number of redundant audio frames, as well as the ratio of total bandwidth to encoding bitrate. t Further restrictions will be imposed.

[0188] For example, the input raw audio frames are grouped according to a predefined grouping size (origional_count), and the number of redundant audio frames (recovery_count) is calculated in the previous step. t Generate a specified number of redundant audio frames. Specifically, taking original audio frames P1, P2, P3, P4, and P5 as inputs with original_count = 5 and recovery_count = 3, the redundant audio frames are calculated according to the following formula:

[0189]

[0190] The leftmost matrix in the equation consists of an identity matrix and a Cauchy matrix. The elements of the Cauchy matrix... Where x i -y j ≠0, 1≤i≤recovery_count t ,1≤j≤origional_count. x i and y j All are Galois domains GF(2) w Elements in R. a ,R b ,R c These are redundant audio frames generated through encoding.

[0191] like Figure 7 As shown, several original audio frames within a group are encoded to generate several redundant audio frames. All redundant audio frames generated in the current group are appended to one or more subsequent groups of original audio frames for framing and output. For example, the redundant audio frames R generated in the first group... a ,R b ,R c Each of the original audio frames P6, P7, and P8 in the second group is used to create a new redundant audio frame R. d ,R e ,R f Each of the three groups of original audio frames P 11 ,P 12 ,P 13 Perform frame assembly.

[0192] For example, when data packets are lost during network transmission: using the original audio frames P1, P2, P3, P4, P5, and redundant audio frames R...a ,R b ,R c For example, if P3, P4, and P5 are missing, then P1, P2, and R... a ,R b ,R c Arrived successfully. The method to recover the lost audio frames is to solve the following equation:

[0193]

[0194] remember Cauchy matrices have the property that they remain invertible even after deleting any row or column, therefore matrix B... ′ It is reversible. Multiply both sides of the equation by B on the left. ′ The inverse of the equation can be used to solve for and recover the original audio frames P1, P2, P3, P4, and P5:

[0195]

[0196] In some embodiments, prior to S201, the following operations may also be performed:

[0197] Based on the audio acquisition parameters and acquisition driving method supported by the server, audio data is acquired and written into the buffer.

[0198] Audio data is extracted from the buffer according to preset audio format parameters to obtain the audio data to be processed; the audio format parameters include at least one of the following: frame length, sampling rate, number of channels, and bit depth.

[0199] Specifically, audio data is acquired based on audio acquisition parameters and acquisition drive method; that is, the audio data to be sent from the server to the terminal device is continuously written into the buffer. If the buffer is full, it is processed according to a preset strategy (such as discarding the oldest data).

[0200] Optionally, audio frames are extracted from the buffer according to a preset frame length, and the audio frames are format-converted according to at least one of the sampling rate, number of channels, and bit depth.

[0201] Specifically, audio data is retrieved from the buffer according to a preset frame length (e.g., 10ms). The number of audio frames retrieved from the buffer each time is determined based on the sampling rate supported by the server and the preset frame length.

[0202] Furthermore, each audio frame is converted into audio data that conforms to preset audio format parameters, such as at least one of the following: sample rate (sample_rate_hz), number of channels, and bit depth (bit_depth).

[0203] In the above implementation, audio data is acquired according to the audio acquisition parameters and acquisition driving method supported by the server, and the acquired audio data is written into a buffer. Then, the audio data to be processed is extracted from the buffer. The audio data acquisition efficiency is high and the complexity is low.

[0204] In some embodiments, before acquiring audio data according to audio acquisition parameters and acquisition driving mode, the method further includes:

[0205] The server is initialized, and the audio acquisition parameters include at least one of the following: sampling rate, number of channels, and bit depth;

[0206] The acquisition driving method supported by the server is determined, and the acquisition driving method includes at least one of the following: time-driven method or event-driven method.

[0207] Specifically, the server needs to be initialized, which means initializing the server's audio virtual device. The audio virtual device can be a driver for a real hardware device, or it can be a virtual audio input / output device based on the internal mechanisms of the operating system.

[0208] The device identifier is obtained using the operating system's Application Programming Interface (API), and then the supported sampling rate (sample_rate_hz) of the virtual device is determined based on the device identifier. device Hertz, number of channels device bit depth device Bits (e.g., 44100, 2, 16 respectively).

[0209] Determine the supported acquisition driver method, such as time-driven or event-driven. Optionally, the supported acquisition driver method can be determined based on operating system information, such as the operating system model and type. If an event-driven method is used, an event is triggered after the virtual device generates audio data when it becomes available, and the corresponding callback function is called to acquire the audio data. If a time-driven method is used, the API is called to acquire audio data when each time slice arrives.

[0210] For example, such as Figure 8 As shown, the audio data acquisition process is as follows:

[0211] S81. Virtual Device Initialization: First, initialize the audio virtual device. The virtual device can be a driver for a real hardware device or a virtual audio input / output device based on the internal mechanisms of the operating system.

[0212] S82. Configure audio acquisition parameters: Use the operating system API to obtain the device identifier, and then obtain the sampling rate (sample_rate_hz) supported by the virtual device. device Hertz, number of channels device bit depth device Bits (e.g., 44100, 2, 16 respectively).

[0213] S83. Acquisition Driver Selection: Check the supported acquisition driver methods, such as time-driven or event-driven.

[0214] S84. Acquisition and Acquisition Buffer: Acquire audio data according to the above audio acquisition parameters and acquisition drive method, and continuously write the acquired audio data into the circular buffer. If the buffer is full, it is processed according to a preset strategy (such as discarding the oldest data).

[0215] S85, Frame Segmentation: Data is retrieved from the acquisition buffer according to the preset frame length frame_length_ms (e.g., 10ms). The number of audio frames, bits, and bytes retrieved from the acquisition buffer each time are as follows:

[0216]

[0217] bits_per_frame=samples_per_frame×bit_depth device ×

[0218] channel device ;

[0219]

[0220] S86. Resampling: Using a third-party library (such as Speex), each audio frame is converted to meet the preset sampling rate (sample_rate_hz) and channel count (channel). device bit depth device The data is then delivered to subsequent processing components in the specified format.

[0221] S87. Repeat steps S84 to S86 until data acquisition stops, the virtual device is deinitialized, and related resources are released.

[0222] In summary, the method described in this application can dynamically adjust redundancy strategies, taking into account both network conditions and terminal performance, to improve audio transmission quality in weak network environments. Simultaneously, it can reduce latency, stuttering, and distortion issues in audio transmission, enhancing the user experience in unstable network environments.

[0223] In some embodiments, after performing a first redundancy processing operation on the encoded audio data to obtain processed audio data, or after performing a second redundancy processing operation on the processed audio data using the target redundancy processing method, the method further includes:

[0224] The processed audio data is backed up and transmitted through multiple available transmission channels.

[0225] Specifically, when sending audio data, multiple available transmission channels are used for redundancy and transmission occurs simultaneously. Examples include Wi-Fi + mobile data, and Transmission Control Protocol (TCP) traffic + User Datagram Protocol (UDP) traffic. Even if some channels experience network fluctuations, other channels can still ensure that data packets arrive as expected, thereby improving the overall reliability and stability of communication.

[0226] In the above embodiments, multiple redundancy schemes are combined, such as combining encoding redundancy, first redundancy processing operations and backup transmission, or combining second redundancy processing operations and backup transmission, to improve the reliability of cloud desktop audio transmission.

[0227] Figure 9 A schematic diagram of the audio data processing device provided in this application is shown below. Figure 9 As shown, the audio data processing device provided in this embodiment includes:

[0228] Processing module 901 is used to perform redundant encoding processing on the audio data to be processed when the redundant encoding process is determined to be executed based on network status information, so as to obtain encoded audio data.

[0229] The processing module 901 is further configured to perform a first redundancy processing operation on the encoded audio data to obtain processed audio data.

[0230] In one possible implementation, the network status information indicates the congestion and packet loss status of network transmission; the device further includes: a determination module 902, used to determine whether to perform the redundancy coding process based on the network status information.

[0231] In one possible implementation, the network state information includes at least uplink packet loss rate information, and the determining module 902 is specifically used for:

[0232] If the uplink packet loss rate is greater than or equal to the packet loss rate threshold, the redundant coding process is determined to be executed.

[0233] In one possible implementation, the determining module 902 is further configured to:

[0234] Based on the network status information, determine whether to perform the second redundancy processing operation;

[0235] The processing module 901 is further configured to:

[0236] If it is determined that the second redundancy processing operation will be performed, a target redundancy processing method supported by the terminal device communicating with the server is determined, and the second redundancy processing operation is performed on the processed audio data using the target redundancy processing method.

[0237] In one possible implementation, the network state information includes at least round-trip delay information; the determining module 902 is specifically used for:

[0238] If the round-trip delay is greater than or equal to the delay threshold, the second redundancy processing operation is performed.

[0239] In one possible implementation, the processing module 901 is specifically used for:

[0240] For each audio frame in the audio data, the main bitstream and the sub-bitstream of the audio frame are encoded according to a preset first redundancy ratio to obtain an encoded audio frame; the first redundancy ratio is the proportion of the sub-bitstream in the audio frame.

[0241] In one possible implementation, the processing module 901 is further configured to:

[0242] If, based on network status information, it is determined that the redundant encoding process should not be performed, the audio data to be processed is subjected to non-redundant encoding to obtain the encoded audio data.

[0243] In one possible implementation, the processing module 901 is specifically used for:

[0244] For each audio frame in the audio data, all the audio data of the audio frame is used as the main bitstream, and the main bitstream is encoded to obtain the encoded audio frame.

[0245] In one possible implementation, the processing module 901 is specifically used for:

[0246] For any of the encoded audio frames, determine whether the previous audio frame of the encoded audio frame contains a sub-bitstream;

[0247] If the previous audio frame contains a sub-bitstream, the sub-bitstream in the previous audio frame is appended to the main bitstream of the encoded audio frame to obtain the processed audio frame, and the flag bit in the processed audio frame is set to a first value.

[0248] If the previous audio frame does not contain a sub-bitstream, a processed audio frame is generated based on the main bitstream of the encoded audio frame, and the flag bit in the processed audio frame is set to a second value.

[0249] In one possible implementation, the target redundancy processing method includes: a first redundancy processing method and a second redundancy processing method; the device performance of the terminal device supporting the second redundancy processing method is higher than that of the terminal device supporting the first redundancy processing method; the processing module 901 is specifically used for:

[0250] If at least one of the multiple terminal devices communicating with the server does not support the second redundancy processing method, the first redundancy processing method is determined to be the target redundancy processing method.

[0251] If multiple terminal devices communicating with the server all support the second redundancy processing method, then the second redundancy processing method is determined to be the target redundancy processing method.

[0252] In one possible implementation, the first redundancy processing method is the target redundancy processing method, and the processing module 901 is specifically used for:

[0253] For any target audio frame in the audio data, the first n audio frames of the target audio frame are appended to the target audio frame to obtain an aggregated data packet; the length of the next audio frame is set between adjacent audio frames in the data packet; n is obtained according to the second redundancy ratio, and n is an integer greater than 0.

[0254] In one possible implementation, the second redundancy ratio is obtained based on the round-trip delay and a preset packet loss tolerance threshold.

[0255] In one possible implementation, the second redundancy processing method is the target redundancy processing method, and the processing module 901 is specifically used for:

[0256] For any first audio frame group in the audio data, at least one redundant audio frame is generated corresponding to the first audio frame group according to the third redundancy ratio; the first audio frame group includes at least one original audio frame.

[0257] The at least one redundant audio frame and the second audio frame group are aggregated to obtain an aggregated data packet; the second audio frame group is at least one audio frame group following the first audio frame group, and the length of the next audio frame is specified between adjacent audio frames in the aggregated data packet.

[0258] In one possible implementation, the third redundancy ratio is obtained based on the number of redundant frames and the number of original audio frames in the first audio frame group, wherein the number of redundant frames is obtained based on the number of original audio frames in the first audio frame group, the uplink packet loss rate, and the no-packet-loss threshold.

[0259] In one possible implementation, the aggregated data packet has a header, which includes at least one of the following: data packet type, timestamp, sequence number, redundancy flag, group number, group size, and the number of redundant audio frames corresponding to the first audio frame group; the redundancy flag is used to indicate whether the data packet contains redundant data.

[0260] In one possible implementation, the processing module 901 is further configured to:

[0261] Based on the audio acquisition parameters and acquisition driving method supported by the server, audio data is acquired and written into the buffer.

[0262] Audio data is extracted from the buffer according to preset audio format parameters to obtain the audio data to be processed; the audio format parameters include at least one of the following: frame length, sampling rate, number of channels, and bit depth.

[0263] In one possible implementation, the processing module 901 is further configured to:

[0264] Before acquiring audio data according to the audio acquisition parameters and acquisition driving method, the server is initialized. The audio acquisition parameters include at least one of the following: sampling rate, number of channels, and bit depth.

[0265] The acquisition driving method supported by the server is determined, and the acquisition driving method includes at least one of the following: time-driven method or event-driven method.

[0266] In one possible implementation, the processing module 901 is further configured to:

[0267] The network state information is obtained before determining whether to perform the redundancy coding process based on the network state information;

[0268] The network status information is smoothed.

[0269] In one possible implementation, the processing module 901 is specifically used for:

[0270] The uplink packet loss rate in the network status information is smoothed using an exponential filtering algorithm;

[0271] The round-trip delay in the network status information is smoothed using a percentile filtering algorithm.

[0272] In one possible implementation, the processing module 901 is further configured to:

[0273] After performing the first redundancy processing operation on the encoded audio data to obtain the processed audio data, or after performing the second redundancy processing operation on the processed audio data using the target redundancy processing method, the processed audio data is backed up and transmitted through multiple available transmission channels.

[0274] The audio data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0275] Figure 10 This is a schematic diagram of the server structure provided in this application. Figure 10 As shown, the electronic device 100 provided in this embodiment includes at least one processor 1001 and a memory 1002. Optionally, the device 100 further includes a communication component 1003. The processor 1001, memory 1002, and communication component 1003 are connected via a bus.

[0276] In a specific implementation, at least one processor 1001 executes computer execution instructions stored in memory 1002, causing at least one processor 1001 to perform the above-described method.

[0277] The specific implementation process of processor 1001 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0278] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0279] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0280] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0281] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0282] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0283] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0284] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0285] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0286] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0287] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0288] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to related technologies, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0289] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0290] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An audio data processing method, characterized in that, Applied to a server, the method includes: When the redundant encoding process is determined based on network state information, the audio data to be processed is subjected to redundant encoding to obtain the encoded audio data. The encoded audio data is subjected to a first redundancy processing operation to obtain the processed audio data.

2. The method according to claim 1, characterized in that, The network status information indicates the congestion and packet loss status of network transmission; before performing redundant encoding on the audio data to be processed to obtain the encoded audio data, the process further includes: The decision to execute the redundancy coding process is based on the network status information.

3. The method according to claim 2, characterized in that, The network state information includes at least uplink packet loss rate information. Determining whether to execute the redundancy coding process based on the network state information includes: If the uplink packet loss rate is greater than or equal to the packet loss rate threshold, the redundant coding process is determined to be executed.

4. The method according to any one of claims 1-3, characterized in that, The process of performing redundant encoding on the audio data to be processed to obtain encoded audio data includes: For each audio frame in the audio data, the main bitstream and the sub-bitstream of the audio frame are encoded according to a preset first redundancy ratio to obtain an encoded audio frame; the first redundancy ratio is the proportion of the sub-bitstream in the audio frame.

5. The method according to any one of claims 1-3, characterized in that, The method further includes: If, based on network status information, it is determined that the redundant encoding process should not be performed, the audio data to be processed is subjected to non-redundant encoding to obtain the encoded audio data.

6. The method according to any one of claims 1-3, characterized in that, The method further includes: Based on the network status information, determine whether to perform the second redundancy processing operation; If it is determined that the second redundancy processing operation will be performed, a target redundancy processing method supported by the terminal device communicating with the server is determined, and the second redundancy processing operation is performed on the processed audio data using the target redundancy processing method.

7. The method according to claim 6, characterized in that, The network status information includes at least round-trip delay information, and the step of determining whether to perform a second redundancy processing operation based on the network status information includes: If the round-trip delay is greater than or equal to the delay threshold, the second redundancy processing operation is performed.

8. The method according to claim 6, characterized in that, The target redundancy processing method includes: a first redundancy processing method and a second redundancy processing method; the device performance of the terminal device supporting the second redundancy processing method is higher than that of the terminal device supporting the first redundancy processing method; determining the target redundancy processing method supported by the terminal device communicating with the server includes: If at least one of the multiple terminal devices communicating with the server does not support the second redundancy processing method, the first redundancy processing method is determined to be the target redundancy processing method. If multiple terminal devices communicating with the server all support the second redundancy processing method, then the second redundancy processing method is determined to be the target redundancy processing method.

9. The method according to claim 8, characterized in that, The second redundancy processing method is the target redundancy processing method. The step of performing the second redundancy processing operation on the processed audio data using the target redundancy processing method includes: For any first audio frame group in the audio data, at least one redundant audio frame is generated corresponding to the first audio frame group according to the third redundancy ratio; the first audio frame group includes at least one original audio frame. The at least one redundant audio frame and the second audio frame group are aggregated to obtain an aggregated data packet; the second audio frame group is at least one audio frame group following the first audio frame group, and the length of the next audio frame is specified between adjacent audio frames in the aggregated data packet.

10. The method according to claim 9, characterized in that, The third redundancy ratio is obtained based on the number of redundant frames and the number of original audio frames in the first audio frame group. The number of redundant frames is obtained based on the number of original audio frames in the first audio frame group, the uplink packet loss rate, and the no-packet-loss threshold.

11. The method according to any one of claims 9-10, characterized in that, The aggregated data packet has a header, which includes at least one of the following: data packet type, timestamp, sequence number, redundancy flag, group number, group size, and the number of redundant audio frames corresponding to the first audio frame group; the redundancy flag is used to indicate whether the data packet contains redundant data.

12. The method according to any one of claims 1-3, characterized in that, Before performing redundant encoding on the audio data to be processed to obtain the encoded audio data, the process further includes: Based on the audio acquisition parameters and acquisition driving method supported by the server, audio data is acquired and written into the buffer. Audio data is extracted from the buffer according to preset audio format parameters to obtain the audio data to be processed; the audio format parameters include at least one of the following: frame length, sampling rate, number of channels, and bit depth.

13. The method according to claim 12, characterized in that, Before acquiring audio data based on audio acquisition parameters and acquisition driving method, the process also includes: The server is initialized, and the audio acquisition parameters include at least one of the following: sampling rate, number of channels, and bit depth; The acquisition driving method supported by the server is determined, and the acquisition driving method includes at least one of the following: time-driven method or event-driven method.

14. The method according to claim 6, characterized in that, After performing the first redundancy processing operation on the encoded audio data to obtain processed audio data, or after performing the second redundancy processing operation on the processed audio data using the target redundancy processing method, the method further includes: The processed audio data is backed up and transmitted through multiple available transmission channels.

15. An audio data processing apparatus, characterized in that, include: The processing module is used to perform redundant encoding on the audio data to be processed, based on network status information, to obtain the encoded audio data. The processing module is further configured to perform a first redundancy processing operation on the encoded audio data to obtain processed audio data.

16. A server, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-14.

17. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-14.

Images