Data processing method and device, communication device, storage medium and program product

Abstract

Embodiments of the present application disclose a data processing method and device, communication equipment, a storage medium and a program product. The method comprises: receiving, by a cloud desktop client, a data packet, the data packet comprising a timestamp, a frame number and audio data; determining network state information based on the timestamp and / or the frame number; the network state information comprising one or more of the following: transmission delay, network jitter, packet loss rate; determining a data processing strategy corresponding to the network state information, and processing the audio data according to the data processing strategy.

Description

Technical Field

[0001] This application relates to communication technology, and more particularly to a data processing method, apparatus, communication equipment, storage medium, and program product. Background Technology

[0002] Cloud desktop transmission protocols are a crucial component of cloud desktop solutions. They enable efficient data transmission with limited resources, achieving low-latency, high-fidelity audio transmission and maintaining a superior user experience even in high-density user scenarios. The user experience of audio transmission in cloud desktops primarily focuses on sound quality, real-time audio performance, and continuity. Deficiencies in any of these aspects can significantly impact the actual audio experience.

[0003] Currently, network status detection mainly utilizes Internet Control Message Protocol (ICMP) technology. This method requires sending additional detection messages, which incurs additional network overhead. Summary of the Invention

[0004] This application provides a data processing method, apparatus, communication device, storage medium, and program product.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides a data processing method applied to a cloud desktop client. The method includes: receiving a data packet, the data packet including a timestamp, a frame number, and audio data; determining network status information based on the timestamp and / or the frame number; the network status information including one or more of the following: transmission latency, network jitter, and packet loss rate; determining a data processing strategy corresponding to the network status information; and processing the audio data according to the data processing strategy.

[0006] In the above scheme, determining network status information based on the timestamp and / or the frame number includes one or more of the following: The transmission delay of the data packet is determined based on the timestamp in the data packet and the reception time of the data packet; Network jitter is determined based on the transmission delay of multiple data packets; The packet loss rate is determined based on the frame number in each of the multiple data packets.

[0007] In the above scheme, determining the data processing strategy corresponding to the network status information and processing the audio data according to the data processing strategy includes: adjusting a first factor based on the network status information, determining a first data processing strategy or a second data processing strategy based on the adjusted first factor, and processing the audio data according to the first data processing strategy or the second data processing strategy; wherein, the first data processing strategy is a data processing strategy corresponding to an abnormal network status, and the second data processing strategy is a data processing strategy corresponding to a normal network status.

[0008] In the above scheme, the first factor includes one or more of the following: latency factor, packet loss factor, jitter factor; the step of adjusting the first factor according to the network state information, and determining the first data processing strategy or the second data processing strategy according to the adjusted first factor, includes: In response to the transmission delay being greater than or equal to a first threshold, the delay factor is increased; or, in response to the transmission delay being less than the first threshold, the delay factor is decreased; wherein the delay factor is adjusted starting from a first preset initial value; in response to the adjusted delay factor being greater than or equal to a second threshold, a first data processing strategy is determined; or, in response to the adjusted delay factor being the first preset initial value, a second data processing strategy is determined. In response to the packet loss rate being greater than or equal to a third threshold, the packet loss factor is increased; or, in response to the packet loss rate being less than the third threshold, the packet loss factor is decreased; wherein, the packet loss factor is adjusted starting from a second preset initial value; in response to the adjusted packet loss factor being greater than or equal to a fourth threshold, a first data processing strategy is determined; or, in response to the adjusted packet loss factor being the second preset initial value, a second data processing strategy is determined. In response to the network jitter being greater than or equal to a fifth threshold, the jitter factor is increased; or, in response to the network jitter being less than the fifth threshold, the jitter factor is decreased; wherein the jitter factor is adjusted starting from a third preset initial value; in response to the adjusted jitter factor being greater than or equal to a sixth threshold, a first data processing strategy is determined; or in response to the adjusted jitter factor being the third preset initial value, a second data processing strategy is determined.

[0009] In the above scheme, processing the audio data according to the first data processing strategy or the second data processing strategy includes: In response to the case that the first factor is a delay factor, the audio data is processed according to the first compression rate and / or the first bit rate included in the first data processing strategy, or the audio data is processed according to the second compression rate and / or the second bit rate included in the second data processing strategy, wherein the first compression rate is greater than the second compression rate and the first bit rate is less than the second bit rate; In response to the case where the first factor is a packet loss factor, the previous data packet is recovered using redundant data in the current data packet according to the first data processing strategy, or the audio data in the current data packet is processed according to the second data processing strategy; wherein, the redundant data is key data of the audio data in the previous data packet carried in the current data packet. In response to the case where the first factor is a jitter factor, a preset number of data packets are cached according to the first data processing strategy, and the cached data packets are processed; or, the received data packets are processed directly according to the second data processing strategy.

[0010] This application also provides a data processing method applied to a cloud desktop server. The method includes: encoding audio data sequentially to obtain encoded data; adding frame numbers and timestamps to the encoded data to generate data packets; and sending the generated data packets to a cloud desktop client. The timestamps and / or the frame numbers are used by the cloud desktop client to determine network status information, so that the cloud desktop client processes the audio data according to the data processing strategy corresponding to the network status information. The network status information includes one or more of the following: transmission latency, network jitter, and packet loss rate.

[0011] In the above scheme, the step of encoding the audio data in time sequence to obtain encoded data includes: acquiring audio time sequence data, which includes multiple audio data ordered by time; collecting audio data of a preset duration from the audio time sequence data; encoding the audio data of the preset duration and redundant data to obtain encoded data; wherein, the redundant data is key data of the audio data corresponding to the frame number preceding the frame number.

[0012] This application also provides a communication device, which includes: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the data processing method described in this application.

[0013] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed, implements the steps of the data processing method described in this application.

[0014] This application also provides a computer program product, including computer program instructions that cause a computer to perform the steps of the data processing method described in this application.

[0015] The data processing method, apparatus, communication device, storage medium, and program product provided in this application embodiment include: a cloud desktop client receiving a data packet, the data packet including a timestamp, a frame number, and audio data; determining network status information based on the timestamp and / or the frame number; the network status information including one or more of the following: transmission latency, network jitter, and packet loss rate; determining a data processing strategy corresponding to the network status information; and processing the audio data according to the data processing strategy. By using the technical solution of this application embodiment to detect network status using the timestamp and / or frame number carried by the audio data packet, on the one hand, additional network overhead is avoided by using network status detection such as Internet Control Message Protocol (ICMP) messages, thus saving network overhead; on the other hand, this application embodiment can dynamically adjust the corresponding data processing strategy for various network conditions, effectively improving the real-time audio readability of the cloud desktop. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the system architecture for the data processing method described in this application embodiment; Figure 2 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 1 ; Figure 3 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 2 ; Figure 4 This is a schematic diagram of the data packet format in the data processing method provided in the embodiments of this application; Figure 5 This is a flowchart illustrating the process of determining the corresponding data processing strategy based on transmission delay in the data processing method provided in this application embodiment; Figure 6 This is a flowchart illustrating the process of determining the corresponding data processing strategy based on the packet loss rate in the data processing method provided in the embodiments of this application. Figure 7 This is a flowchart illustrating the process of determining the corresponding data processing strategy based on network jitter in the data processing method provided in this application embodiment; Figure 8 This is a schematic diagram of the buffer in the data processing method provided in the embodiments of this application; Figure 9This is a schematic diagram of the overall processing procedure of the data processing method provided in the embodiments of this application; Figure 10 This is a schematic diagram of the composition structure of the data processing device provided in the embodiments of this application. Figure 1 ; Figure 11 This is a schematic diagram of the composition structure of the data processing device provided in the embodiments of this application. Figure 2 ; Figure 12 This is a schematic diagram of the hardware composition structure of the communication device provided in the embodiments of this application. Detailed Implementation

[0017] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0018] The technical solutions of this application embodiment can be applied to various communication systems, such as GSM (Global System of Mobile communication), LTE (Long Term Evolution), or 5G systems. Optionally, a 5G system or 5G network can also be referred to as a New Radio (NR) system or NR network.

[0019] It should be understood that devices with communication functions in the network / system of this application embodiment can be referred to as communication devices. Communication devices may include network devices and terminals with communication functions.

[0020] It should be understood that the terms "system" and "network" in the embodiments of this application are often used interchangeably herein. The term "and / or" in this document merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Furthermore, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0021] The terms "first," "second," etc., used in the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] Figure 1 This is a schematic diagram of the system architecture for the data processing method described in this application embodiment; as follows: Figure 1 As shown, the system may include a cloud desktop server 10 and a cloud desktop client 20. The cloud desktop server 10 can be deployed on a server cluster in a data center and may consist of multiple core components. Its functions include, but are not limited to: hardware resource virtualization (such as creating and running desktop virtual machines and allocating resources), image creation (such as creating system images and generating user desktops), desktop management (such as account binding, permission policies, session management, etc.), user access scheduling, etc. The cloud desktop client 20 can be deployed on user equipment (UE, User Equipment), such as on a user terminal (such as a computer). For example, the cloud desktop client 20 can run on a user terminal as software, and the user 30 can log in to the cloud desktop client 20 by entering an account and password.

[0023] For example, a communication process between the cloud desktop server 10 and the cloud desktop client 20 may include: the user initiating a login request; after the cloud desktop server 10 completes authentication, it allocates or finds the virtual machine corresponding to the user, and completes all calculations through the virtual machine to obtain the cloud desktop screen; the cloud desktop screen data is compressed and sent to the cloud desktop client via the network; the cloud desktop client decodes the cloud desktop screen, renders and displays it through a display device, and plays sound through an audio device; the cloud desktop client 20 captures keyboard and / or mouse input operations, encapsulates the input operation data into data packets, and sends them to the cloud desktop server 10; the cloud desktop server 10 hands over the input operation data to the virtual machine for execution.

[0024] based on Figure 1 The system shown in this application provides a data processing method, which is applied to a cloud desktop client. Figure 2 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 1 ;like Figure 2 As shown, the method includes: Step 101: Receive a data packet, which includes a timestamp, frame number, and audio data; Step 102: Determine network status information based on the timestamp and / or the frame number; the network status information includes one or more of the following: transmission delay, network jitter, and packet loss rate; Step 103: Determine the data processing strategy corresponding to the network status information, and process the audio data according to the data processing strategy.

[0025] Accordingly, this application also provides a data processing method, which is applied to a cloud desktop server. Figure 3 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 2 ;like Figure 3 As shown, the method includes: Step 201: Encode the audio data sequentially to obtain encoded data; Step 202: Add a frame number and a timestamp to the encoded data to generate a data packet, and send the generated data packet to the cloud desktop client; wherein, the timestamp and / or the frame number are used by the cloud desktop client to determine network status information, so that the cloud desktop client processes the audio data according to the data processing strategy corresponding to the network status information; the network status information includes one or more of the following: transmission latency, network jitter, and packet loss rate.

[0026] In this embodiment, the cloud desktop server uses transmission technology and a virtual audio driver to acquire the raw audio data. It then preprocesses the audio data sequentially, including but not limited to noise reduction, data packaging, compression, and header encapsulation. After preprocessing, the cloud desktop server sends the data packet to the cloud desktop client. The cloud desktop client uses the timestamp and / or frame number carried in the data packet to determine the network status information, and then determines the corresponding data processing strategy based on the network status information, processing the audio data accordingly.

[0027] In this embodiment, the data packet includes a timestamp, frame number, and audio data; the cloud desktop client can perform self-detection of network conditions based on the timestamp and / or frame number carried in the data packet, thereby determining network status information.

[0028] Traditional solutions typically use the Internet Control Message Protocol (ICMP) to obtain network status. For example, the sender sends an ICMP request, and the receiver replies with an ICMP response. In this process, the connectivity between the sender and receiver is determined by whether the receiver responds with an ICMP response; the transmission delay between the sending of an ICMP request and the receiving of an ICMP response can be determined by the time interval between them; and the packet loss rate between the sender and receiver can be determined by the number of ICMP responses received after the sender sends multiple ICMP requests. However, this method of obtaining network status via ICMP is susceptible to being blocked by firewalls or host operating systems, leading to the obstruction of ICMP message sending or receiving. Furthermore, because ICMP messages have low priority, they are easily dropped during network congestion, rendering network diagnostic tools that rely on ICMP messages ineffective.

[0029] Since audio data is packaged according to a certain duration (e.g., 10 milliseconds, 20 milliseconds, etc.) or quantity (e.g., sampling points); for example, for audio data with a sampling rate of N Hertz (Hz), assuming that the sampling points of one frame are M, the number of data packets that need to be transmitted per second is N / M. Using a common sampling rate of 48 kHz and 960 sampling points per frame, this means 50 data packets need to be sent per second, with each data packet corresponding to a timestamp of 20 milliseconds (ms). The number of data packets and the timing accuracy both meet the requirements of network probing. Therefore, this embodiment can directly obtain network status information by probing the transmitted data packets. This avoids the additional network overhead associated with using ICMP messages. Furthermore, by utilizing the self-probing of network status by audio data packets, this embodiment can fully leverage the characteristics of audio data packets themselves to adjust dynamic data processing strategies.

[0030] In this embodiment, during the encoding and packaging of audio data, the cloud desktop server adds a frame number and a timestamp to the data packet and sends it to the cloud desktop client. The timestamp represents the sending time of the data packet. The frame number represents the sequence number (or time sequence number) of the data packet. For example, the cloud desktop server acquires audio data sequentially, forming audio time-series data. It then sequentially acquires audio data from the audio time-series data according to a preset duration (e.g., 10 milliseconds, 20 milliseconds, etc.), packages and encodes it to generate a data packet. Before sending the data packet, a frame number and a timestamp are added to it; alternatively, the frame number is added to the data packet during audio data packaging and encoding, and the timestamp is added before sending the data packet. The frame number increments sequentially according to the generation sequence of the data packet or the encoding sequence of the audio data. For example, when a process on the cloud desktop server uses audio, the transmission protocol notifies the audio module to work through driver layer signals, initializing the initial sequence number (i.e., frame number) corresponding to the audio data. For example, the initial sequence number (i.e., frame number) is initialized to 0. It can be understood that the frame number starts from 0 and increments sequentially. When the audio data meets the data volume corresponding to a data packet, the audio module encodes the audio data to generate a data packet. When sending this data packet, the frame number (e.g., frame number 0) and timestamp are added to the data packet for encapsulation, and the encapsulated data packet is sent. Similarly, when the audio data meets the data volume corresponding to the next data packet, the audio module encodes the audio data to generate a data packet and increments the frame number. When sending this data packet, the frame number (e.g., frame number 1) and timestamp are added to the data packet for encapsulation, and the encapsulated data packet is sent. The processing of subsequent data packets is similar to the above process.

[0031] For example, Figure 4 This is a schematic diagram of the data packet format in the data processing method provided in the embodiments of this application; as shown... Figure 4As shown, in addition to encoded data (such as encoded audio data), the data packet in this embodiment also includes a frame number and a timestamp; wherein, the frame number and the timestamp can be as follows: Figure 4 As shown, the frame number and timestamp are added before the encoded data as the header of the data packet. This means that during the data packet encapsulation process, the frame number and timestamp can be encapsulated in the header before transmission. In other alternative embodiments, the frame number and timestamp can also be added to the tail of the data packet, for example, by adding them to the tail of the encoded data before encapsulation.

[0032] In this embodiment, the timestamp represents the sending time of the data packet, and the transmission delay of the data packet can be calculated based on the timestamp. The transmission delay represents the total time required for the data packet to travel from the sending end to the receiving end, and may include sending, transmission, processing, queuing delays, etc. The transmission delay indicates the response speed. Network jitter can be determined based on the transmission delay. The network jitter (also known as latency jitter) represents the difference in latency between consecutive data packets in the data stream, and the network jitter indicates the smoothness of transmission. The purpose of carrying the frame number in the data packet is to calculate the packet loss situation. It can be understood that the network status information in this embodiment includes, but is not limited to, one or more of the following: transmission delay, network jitter, and packet loss rate.

[0033] In some optional embodiments, for cloud desktop clients, determining network status information based on the timestamp and / or the frame number includes one or more of the following: determining the transmission delay of the data packet based on the timestamp in the data packet and the reception time of the data packet; determining network jitter based on the transmission delay of multiple data packets; and determining the packet loss rate based on the frame number in each of the multiple data packets.

[0034] In this embodiment, the cloud desktop client can calculate the transmission delay of the data packet based on the timestamp carried in the data packet and the data packet reception time. For example, if the timestamp is represented as t1 and the data packet reception time is represented as t2, then the transmission delay is equal to t2-t1.

[0035] In this embodiment, the cloud desktop client can calculate network jitter based on the transmission latency corresponding to each data packet. As an example, network jitter can be obtained by subtracting the maximum and minimum transmission latency. As another example, network jitter can also be obtained using the statistical dispersion method based on the transmission latency of multiple data packets. This embodiment does not specifically limit the method for obtaining network jitter; any method for calculating network jitter can be used.

[0036] In this embodiment, the cloud desktop client can determine the number of lost data packets within a preset number of data packets based on the frame numbers carried in the data packets. For example, if there are discontinuous frame numbers within 1000 data packets, the cloud desktop client can determine the number of lost data packets based on the missing frame numbers. Furthermore, the packet loss rate can be determined based on the formula: packet loss rate = (number of lost data packets / total number of data packets) × 100%.

[0037] It should be noted that the cloud desktop client can determine one or more of the network status information, such as transmission latency, network jitter, and packet loss rate, based on the implementation method of the above embodiments, and then dynamically determine the corresponding data processing strategy according to one or more of the network status information.

[0038] In some optional embodiments, for a cloud desktop client, determining the data processing strategy corresponding to the network status information and processing the audio data according to the data processing strategy includes: adjusting a first factor based on the network status information, determining a first data processing strategy or a second data processing strategy based on the adjusted first factor, and processing the audio data according to the first data processing strategy or the second data processing strategy; wherein, the first data processing strategy is a data processing strategy corresponding to an abnormal network status, and the second data processing strategy is a data processing strategy corresponding to a normal network status.

[0039] In this embodiment, a first factor is set for network status information. The first factor is dynamically adjusted based on the network status information. Then, a first data processing strategy corresponding to an abnormal network status or a second data processing strategy corresponding to a normal network status is determined based on the adjusted first factor. By introducing the first factor, the actual network status is smoothed out, avoiding excessively frequent changes in data processing strategies due to frequent network status switching, thus improving the user experience.

[0040] In some optional embodiments, for a cloud desktop client, the first factor includes one or more of the following: latency factor, packet loss factor, jitter factor; the step of adjusting the first factor according to the network state information, and determining a first data processing strategy or a second data processing strategy according to the adjusted first factor, includes: In response to the transmission delay being greater than or equal to a first threshold, the delay factor is increased; or, in response to the transmission delay being less than the first threshold, the delay factor is decreased; wherein the delay factor is adjusted starting from a first preset initial value; in response to the adjusted delay factor being greater than or equal to a second threshold, a first data processing strategy is determined; or, in response to the adjusted delay factor being the first preset initial value, a second data processing strategy is determined. In response to the packet loss rate being greater than or equal to a third threshold, the packet loss factor is increased; or, in response to the packet loss rate being less than the third threshold, the packet loss factor is decreased; wherein, the packet loss factor is adjusted starting from a second preset initial value; in response to the adjusted packet loss factor being greater than or equal to a fourth threshold, a first data processing strategy is determined; or, in response to the adjusted packet loss factor being the second preset initial value, a second data processing strategy is determined. In response to the network jitter being greater than or equal to a fifth threshold, the jitter factor is increased; or, in response to the network jitter being less than the fifth threshold, the jitter factor is decreased; wherein the jitter factor is adjusted starting from a third preset initial value; in response to the adjusted jitter factor being greater than or equal to a sixth threshold, a first data processing strategy is determined; or in response to the adjusted jitter factor being the third preset initial value, a second data processing strategy is determined.

[0041] In this embodiment, a delay factor is introduced to address transmission delay. A bounded increment / decrement control method is used to determine whether to adopt a first data processing strategy or a second data processing strategy related to transmission delay. For example, when the transmission delay is greater than or equal to a first threshold, the delay factor is increased; when the transmission delay is less than the first threshold, the delay factor is decreased. The initial value of the delay factor is a first preset initial value. For example, the first preset initial value can be 0, which can be understood as the delay factor being at least the first preset initial value (e.g., 0). When the delay factor is greater than or equal to a second threshold, i.e., when the delay factor exceeds the second threshold, a delay anomaly strategy (i.e., the first data processing strategy) is triggered. When the network returns to normal, i.e., when the delay factor returns to the first preset initial value, such as when the delay factor returns to 0, the original default data processing strategy (i.e., the second data processing strategy) is restored. It should be noted that if the current delay factor is 0, and the delay factor is currently decreased, then the delay factor changes from 0 to 0, i.e., the delay factor is at least 0.

[0042] In some optional embodiments, the amount of data that increases or decreases the latency factor is related to the magnitude of the transmission delay, specifically to the difference between the transmission delay and the first threshold. As an example, when the transmission delay is greater than or equal to the first threshold, the larger the difference between the transmission delay and the first threshold, the greater the increase in the latency factor; conversely, the smaller the difference between the transmission delay and the first threshold, the smaller the increase in the latency factor. As another example, when the transmission delay is less than the first threshold, the larger the difference between the first threshold and the transmission delay, the greater the decrease in the latency factor; conversely, the smaller the difference between the first threshold and the transmission delay, the smaller the decrease in the latency factor. In practical applications, for example, the correspondence between the difference between each transmission delay and the first threshold, and the change in the latency factor, can be pre-configured; then the cloud desktop client can determine the change in the latency factor by looking up this correspondence based on the difference between the transmission delay and the first threshold, or the difference between the first threshold and the transmission delay. Alternatively, a linear relationship between the difference between the transmission delay and the first threshold and the change in the delay factor can be pre-configured. This linear relationship can be expressed by an expression with a linear relationship. The cloud desktop client can then substitute the difference between the transmission delay and the first threshold, or the difference between the first threshold and the transmission delay, into this linear relationship to obtain the change in the delay factor.

[0043] In this embodiment, a packet loss factor is introduced to address the packet loss rate. A bounded increment / decrease control method is used to determine whether to adopt a first data processing strategy or a second data processing strategy related to the packet loss rate. For example, when the packet loss rate is greater than or equal to a third threshold, the packet loss factor is increased; when the packet loss rate is less than the third threshold, the packet loss factor is decreased. The initial value of the packet loss factor is a second preset initial value. For example, the second preset initial value can be 0, which can be understood as the minimum packet loss factor being the second preset initial value (e.g., 0). When the packet loss factor is greater than or equal to a fourth threshold, i.e., when the packet loss factor exceeds the fourth threshold, a packet loss anomaly strategy (i.e., the first data processing strategy) is triggered. When the network returns to normal, i.e., when the packet loss factor returns to the second preset initial value, such as when the packet loss factor returns to 0, the original default data processing strategy (i.e., the second data processing strategy) is restored. It should be noted that if the current packet loss factor is 0, and the packet loss factor is currently decreased, then the packet loss factor changes from 0 to 0, i.e., the minimum packet loss factor is 0.

[0044] In some optional embodiments, the amount of data that increases or decreases in the packet loss factor is related to the magnitude of the packet loss rate, specifically to the magnitude of the difference between the packet loss rate and the third threshold. As an example, when the packet loss rate is greater than or equal to the third threshold, the larger the difference between the packet loss rate and the third threshold, the greater the increase in the packet loss factor; conversely, the smaller the difference between the packet loss rate and the third threshold, the smaller the increase in the packet loss factor. As another example, when the packet loss rate is less than the third threshold, the larger the difference between the third threshold and the packet loss rate, the greater the decrease in the packet loss factor; conversely, the smaller the difference between the third threshold and the packet loss rate, the smaller the decrease in the packet loss factor. In practical applications, for example, the correspondence between the difference between each packet loss rate and the third threshold, and the change in the packet loss factor, can be pre-configured; then the cloud desktop client can determine the change in the packet loss factor by looking up this correspondence based on the difference between the packet loss rate and the third threshold, or the difference between the third threshold and the packet loss rate. Alternatively, a linear relationship between the difference between the packet loss rate and the third threshold and the change in the packet loss factor can be pre-configured. This linear relationship can be expressed by an expression with a linear relationship. The cloud desktop client can then input the difference between the packet loss rate and the third threshold, or the difference between the third threshold and the packet loss rate, into this linear relationship to obtain the change in the packet loss factor.

[0045] In this embodiment, a jitter factor is introduced to address network jitter. A bounded increment / decrement control method is used to determine whether to adopt a first or second data processing strategy related to network jitter. For example, when network jitter is greater than or equal to a fifth threshold, the jitter factor is increased; when network jitter is less than the fifth threshold, the jitter factor is decreased. The initial value of the jitter factor is a third preset initial value, which can be 0, meaning the jitter factor is at least the third preset initial value (e.g., 0). When the jitter factor is greater than or equal to a sixth threshold (i.e., when the jitter factor exceeds the sixth threshold), a jitter anomaly strategy (i.e., the first data processing strategy) is triggered. When the network returns to normal, i.e., when the jitter factor returns to the third preset initial value, such as when the jitter factor returns to 0, the original default data processing strategy (i.e., the second data processing strategy) is restored. It should be noted that if the current jitter factor is 0, and the jitter factor is currently decreased, the jitter factor changes from 0 to 0, meaning the minimum jitter factor is 0.

[0046] In some optional embodiments, the amount of data that increases or decreases the jitter factor is related to the magnitude of the network jitter, specifically to the difference between the network jitter and the fifth threshold. As an example, when the network jitter is greater than or equal to the fifth threshold, the larger the difference between the network jitter and the fifth threshold, the greater the increase in the jitter factor; conversely, the smaller the difference between the network jitter and the fifth threshold, the smaller the increase in the jitter factor. As another example, when the network jitter is less than the fifth threshold, the larger the difference between the fifth threshold and the network jitter, the greater the decrease in the jitter factor; conversely, the smaller the difference between the fifth threshold and the network jitter, the smaller the decrease in the jitter factor. In practical applications, for example, the correspondence between the difference between each network jitter and the fifth threshold, and the change in the jitter factor, can be pre-configured; then the cloud desktop client can determine the change in the jitter factor by looking up this correspondence based on the difference between the network jitter and the fifth threshold, or the difference between the fifth threshold and the network jitter. Alternatively, a linear relationship between the difference between network jitter and the fifth threshold and the change in the jitter factor can be pre-configured. This linear relationship can be expressed by an expression with a linear relationship. The cloud desktop client can then input the difference between network jitter and the fifth threshold, or the difference between the fifth threshold and network jitter, into this linear relationship to obtain the change in the jitter factor.

[0047] In this embodiment, corresponding data processing strategies are provided for different network status indicators (such as transmission latency, network jitter, packet loss rate, etc.), and the specific processing strategy depends on the corresponding network status. In addition, when it is determined that the current network is in a normal or abnormal state based on the network status information, there are corresponding data processing strategies, which are respectively denoted as the first data processing strategy corresponding to the abnormal network status or the second data processing strategy corresponding to the normal network status.

[0048] In some alternative embodiments, for a cloud desktop client, processing the audio data according to the first data processing strategy or the second data processing strategy includes: In response to the case that the first factor is a delay factor, the audio data is processed according to the first compression rate and / or the first bit rate included in the first data processing strategy, or the audio data is processed according to the second compression rate and / or the second bit rate included in the second data processing strategy, wherein the first compression rate is greater than the second compression rate and the first bit rate is less than the second bit rate; In response to the case where the first factor is a packet loss factor, the previous data packet is recovered using redundant data in the current data packet according to the first data processing strategy, or the audio data in the current data packet is processed according to the second data processing strategy; wherein, the redundant data is key data of the audio data in the previous data packet carried in the current data packet. In response to the case where the first factor is a jitter factor, a preset number of data packets are cached according to the first data processing strategy, and the cached data packets are processed; or, the received data packets are processed directly according to the second data processing strategy.

[0049] In this embodiment, the first and second data processing strategies, in addition to the aforementioned network condition-specific processing strategies, can include any processing strategy from data packets to their playback by a speaker. For example, the first and second data processing strategies may include, but are not limited to, one or more of the following processing strategies: decapsulation, audio decoding, resampling, audio frame preprocessing, audio output frame, digital-to-analog conversion, etc. For example, when a cloud desktop client receives data packets, typically streaming media encapsulated according to a specified protocol, it can first perform decapsulation processing to remove the protocol header and obtain the compressed bitstream (such as compressed audio data). Audio decoding processing is then performed on the compressed bitstream to restore it to Pulse Code Modulation (PCM) to obtain sampled data that the user terminal where the cloud desktop client is located can process. If the PCM sampling rate is not equal to the sampling rate of the user terminal's hardware, resampling processing of the PCM is also required. Optionally, audio frame preprocessing can also be performed on the PCM, such as gain control, noise suppression, etc. Finally, the PCM is transmitted to the audio output frame, converting the PMC into an analog electrical signal to drive the hardware to produce sound. It can be understood that the processing of audio data according to the second data processing strategy described in the embodiments of this application may include processing according to preset or default parameters, or using one or more of the above processing strategies.

[0050] In this embodiment, regarding the network status information of transmission delay, i.e., in response to the case where the first factor is a delay factor, since transmission delay is usually caused by network congestion, the surge in data volume causes data packets to be unable to arrive faster. Therefore, in this case, it is necessary to reduce the amount of data in the network. Based on this, in this scenario, the cloud desktop client determines a first data processing strategy corresponding to the transmission delay anomaly, i.e., processing the audio data according to a first compression ratio and / or a first bitrate; after the transmission delay returns to normal, the delay factor decreases because the transmission delay has returned to normal. When the delay factor decreases to a first preset initial value, a second data processing strategy is determined, i.e., processing the audio data according to a second compression ratio and / or a second bitrate; wherein, the first compression ratio is greater than the second compression ratio, and the first bitrate is less than the second bitrate. This can be understood as, when the transmission delay is abnormal, increasing the compression ratio of the audio data and / or decreasing the bitrate of the audio data, and processing the audio data according to the increased compression ratio and / or decreased bitrate.

[0051] For example, Figure 5 This is a flowchart illustrating the process of determining a corresponding data processing strategy based on transmission delay in the data processing method provided in this application embodiment; as follows: Figure 5 As shown, the process of determining the corresponding data processing strategy based on transmission delay mainly includes: Step 301: The cloud desktop client receives the data packet.

[0052] Step 302: Calculate the transmission delay based on the timestamp carried in the data packet and the data packet reception time.

[0053] Step 303: Determine whether the transmission delay is greater than or equal to the first threshold; when the transmission delay is greater than or equal to the first threshold, execute step 304: increase the delay factor; when the transmission delay is less than the first threshold, execute step 307: decrease the delay factor.

[0054] For example, the initial value of the delay factor can be 0, meaning the delay factor can be adjusted starting from 0. If step 307 is executed when the delay factor is 0, reducing the delay factor, then the delay factor will remain at 0 and will not decrease further.

[0055] After step 304, step 305 is executed: determine whether the delay factor is greater than or equal to the second threshold; when the delay factor is greater than or equal to the second threshold, step 306 is executed: enable the first data processing strategy, and then re-execute step 301; otherwise, if the delay factor is less than the second threshold, then step 301 is re-executed.

[0056] After step 307, step 308 is executed: determine whether the delay factor is 0; when the delay factor is 0, step 309 is executed: enable the second data processing strategy, and then step 301 is executed again; otherwise, if the delay factor is not 0, step 301 is executed again.

[0057] In this embodiment, the first data processing strategy is a strategy that, based on the second processing strategy, increases the compression rate of the audio data and / or reduces the bitrate of the audio data. For example, during the audio decoding process of the compressed bitstream, the corresponding compression rate and / or bitrate can be used for processing.

[0058] In this embodiment, regarding the network status information of the packet loss rate, that is, in response to the case where the first factor is the packet loss factor, in this scenario, the cloud desktop client determines the first data processing strategy corresponding to the packet loss anomaly, that is, to use the redundant information carried in the data packet to restore the information of the previous data packet; after the packet loss rate returns to normal, the packet loss factor decreases because the packet loss rate returns to normal; when the packet loss factor decreases to the second preset initial value, the second data processing strategy is determined, that is, to directly process the received data packet.

[0059] For example, Figure 6 This is a flowchart illustrating the process of determining the corresponding data processing strategy based on the packet loss rate in the data processing method provided in this application embodiment; as follows: Figure 6 As shown, determining the corresponding data processing strategy based on packet loss rate mainly includes: Step 401: The cloud desktop client receives the data packet.

[0060] Step 402: Calculate the packet loss rate based on the frame number carried in the data packet.

[0061] Step 403: Determine whether the packet loss rate is greater than or equal to the third threshold; when the packet loss rate is greater than or equal to the third threshold, execute step 404: increase the packet loss factor; when the packet loss rate is less than the third threshold, execute step 407: decrease the packet loss factor.

[0062] For example, the initial value of the packet loss factor can be 0, meaning the packet loss factor can be adjusted starting from 0. If step 407 is executed to reduce the packet loss factor when the packet loss factor is 0, the packet loss factor will remain at 0 and will not decrease further.

[0063] After step 404, step 405 is executed: determine whether the packet loss factor is greater than or equal to the fourth threshold; when the packet loss factor is greater than or equal to the fourth threshold, step 406 is executed: enable the first data processing strategy, and then step 401 is executed again; otherwise, if the packet loss factor is less than the fourth threshold, step 401 is executed again.

[0064] After step 407, step 408 is executed: determine whether the packet loss factor is 0; when the packet loss factor is 0, step 409 is executed: enable the second data processing strategy, and then step 401 is executed again; otherwise, if the packet loss factor is not 0, step 401 is executed again.

[0065] In this embodiment, the data packet not only contains the audio data corresponding to the current frame number, but also redundant information from the previous frame. For example, the redundant information may be key information from the previous frame's audio data; this key information could be, for example, low-frequency information (because the human ear is more sensitive to low frequencies (such as 0-4kHz fundamental frequency encoding)). When the cloud desktop client activates the first data processing strategy, it can process the audio data of the current frame contained in the data packet, and simultaneously recover the previous frame's data packet based on the redundant information carried in the data packet. This allows for recovery of the previous frame's data packet in the event of packet loss, based on the redundant information carried in the current frame. This improves the readability of the audio data and ensures audio quality.

[0066] In this embodiment, regarding network jitter information, specifically in response to the first factor being a jitter factor, since network jitter is usually caused by incomplete packet connections, high jitter results in discontinuous audio data with gaps in the audio. These gaps cause noise in the audio data, affecting audio quality. In this scenario, the cloud desktop determines a first data processing strategy corresponding to the network jitter anomaly, which employs a dynamic buffer strategy. For example, a preset number of data packets are first cached, and then the cached data packets are processed accordingly. After the network jitter returns to normal, the jitter factor decreases. When the network jitter decreases to a third preset initial value, a second data processing strategy is determined, which eliminates the need for data packet caching and directly processes the received data packets.

[0067] For example, Figure 7 This is a flowchart illustrating the process of determining the corresponding data processing strategy based on network jitter in the data processing method provided in this application embodiment; as follows: Figure 7 As shown, the main data processing strategies determined based on network jitter include: Step 501: The cloud desktop client receives the data packet.

[0068] Step 502: Calculate the transmission delay based on the timestamp carried in the data packet and the data packet reception time, and calculate the network jitter based on the data packet transmission delay.

[0069] Step 503: Determine if the network jitter is greater than or equal to the fifth threshold; if the network jitter is greater than or equal to the fifth threshold, proceed to step 504: increase the jitter factor; if the network jitter is less than the fifth threshold, proceed to step 507: decrease the jitter factor.

[0070] For example, the initial value of the jitter factor can be 0, meaning the jitter factor can be adjusted starting from 0. If step 507 is executed when the jitter factor is 0, reducing the jitter factor, then the jitter factor will remain at 0 and will not decrease further.

[0071] After step 504, step 505 is executed: determine whether the jitter factor is greater than or equal to the sixth threshold; when the jitter factor is greater than or equal to the sixth threshold, step 506 is executed: enable the first data processing strategy, and then step 501 is executed again; otherwise, if the jitter factor is less than the sixth threshold, step 501 is executed again.

[0072] After step 507, step 508 is executed: determine whether the jitter factor is 0; when the jitter factor is 0, step 509 is executed: enable the second data processing strategy, and then step 501 is executed again; otherwise, if the jitter factor is not 0, step 501 is executed again.

[0073] Figure 8 This is a schematic diagram of the buffer in the data processing method provided in the embodiments of this application; as shown Figure 8 As shown, in high-jitter scenarios, audio data is discontinuous, with gaps in the audio stream. These large gaps cause a rapid decline in audio quality, such as... Figure 8 The first line in the text. In this application embodiment, a buffer is used to cache a preset number of data packets first, and then the cached data packets are processed accordingly; for example... Figure 8 The second line uses a 5-frame buffer, which can first buffer 5 data packets and then process them accordingly. Although the audio latency increases, the gaps between audio data are significantly reduced, and the readability of the audio data is significantly improved.

[0074] In this embodiment, the cloud desktop client can dynamically adjust one or more of the following network status information: transmission latency, network jitter, and packet loss rate. Based on this, it can determine a data processing strategy corresponding to one or more of these factors. For example, after adjusting the latency factor based on transmission latency, if the latency factor exceeds a second threshold, a first data processing strategy corresponding to the transmission latency can be determined. Similarly, after adjusting the jitter factor based on network jitter, if the jitter factor exceeds a sixth threshold, a first data processing strategy corresponding to the network jitter can be determined. In this case, the cloud desktop client can, on the one hand, improve the compression rate and / or reduce the bitrate of audio data processing according to the first data processing strategy for transmission latency; on the other hand, it can, according to the first data processing strategy corresponding to network jitter, first cache a preset number of data packets, and then process the cached data packets accordingly (e.g., process audio data according to the improved compression rate and / or reduced bitrate).

[0075] It should be noted that the cloud desktop client can determine one or more network status information such as transmission delay, network jitter, and packet loss rate based on the timestamp and / or frame number carried in the data packet. Then, it can determine one or more data processing strategies corresponding to one or more network status information and process the audio data according to one or more data processing strategies.

[0076] In some optional embodiments, for the cloud desktop server, the step of encoding the audio data in time sequence to obtain encoded data includes: acquiring audio time sequence data in time sequence, wherein the audio time sequence data includes multiple audio data ordered by time; collecting a preset number of audio data from the audio time sequence data; encoding the preset number of audio data and redundant data to obtain encoded data; wherein the redundant data is key data of the audio data corresponding to the frame number preceding the frame number.

[0077] In this embodiment, during the process of packaging and encoding audio data, each data packet of the cloud desktop server contains not only the audio data corresponding to the current frame number, but also redundant information of the audio data of the previous frame; wherein, for example, the redundant information may be key information of the audio data of the previous frame; such key information is, for example, low-frequency information.

[0078] Figure 9 This is a schematic diagram of the overall processing procedure of the data processing method provided in the embodiments of this application; as shown below. Figure 9As shown, the data processing method in this embodiment can be implemented through a cloud desktop server and a cloud desktop client. Specifically, the cloud desktop server can use a virtual audio driver to acquire raw audio data (i.e., audio data acquisition), and preprocess the audio data sequentially. The preprocessing includes, but is not limited to, noise reduction, data packaging, compression, and header encapsulation. After preprocessing, the cloud desktop server can send data packets to the cloud desktop client. The cloud desktop client uses the timestamp and / or frame number carried in the data packets to detect the network status, thereby determining the network status information. Then, based on the determined network status information and the data processing strategy, the client preprocesses the audio data in the received data packets, renders the preprocessed audio data, and outputs it. This embodiment can detect the network status in real time based on the received data packets, and can then dynamically adjust the data processing strategy based on the network status information.

[0079] The technical solution of this application avoids additional network overhead caused by using network status detection such as ICMP messages, thus saving network overhead. On the other hand, this application can dynamically adjust the corresponding data processing strategy for various network conditions, which can effectively improve the real-time audio readability of the cloud desktop. Furthermore, in the face of complex network conditions, this application introduces a first factor (such as latency factor, packet loss factor, jitter factor, etc.) to dynamically adjust the data processing strategy, smoothing the actual network conditions and avoiding frequent changes in audio strategy due to frequent state switching, thereby achieving real-time smooth audio processing and improving the user experience.

[0080] Based on the above embodiments, this application also provides a data processing device, which is applied to a cloud desktop client. Figure 10 This is a schematic diagram of the composition structure of the data processing device provided in the embodiments of this application. Figure 1 ;like Figure 10 As shown, the device includes a first communication unit 11, a network status detection unit 12, a policy determination unit 13, and a first processing unit 14; wherein, The first communication unit 11 is used to receive data packets, the data packets including timestamps, frame numbers and audio data; The network status detection unit 12 is used to determine network status information based on the timestamp and / or the frame number; the network status information includes one or more of the following: transmission delay, network jitter, and packet loss rate. The strategy determination unit 13 is used to determine the data processing strategy corresponding to the network status information; The first processing unit 14 is used to process the audio data according to the data processing strategy.

[0081] In some optional embodiments of this application, the network state detection unit 12 is configured to perform one or more of the following: determine the transmission delay of the data packet based on the timestamp in the data packet and the reception time of the data packet; determine network jitter based on the transmission delay of multiple data packets; and determine the packet loss rate based on the frame number in each of the multiple data packets.

[0082] In some optional embodiments of this application, the strategy determination unit 13 is used to adjust a first factor according to the network state information, and determine a first data processing strategy or a second data processing strategy according to the adjusted first factor. The first processing unit 14 is used to process the audio data according to the first data processing strategy or the second data processing strategy; wherein, the first data processing strategy is a data processing strategy corresponding to network status abnormality, and the second data processing strategy is a data processing strategy corresponding to network status normality.

[0083] In some optional embodiments of this application, the first factor includes one or more of the following: a delay factor, a packet loss factor, and a jitter factor; the strategy determination unit 13 is configured to perform one or more of the following: in response to the transmission delay being greater than or equal to a first threshold, increase the delay factor, or in response to the transmission delay being less than the first threshold, decrease the delay factor; wherein the delay factor is adjusted starting from a first preset initial value; in response to the adjusted delay factor being greater than or equal to a second threshold, determine a first data processing strategy, or in response to the adjusted delay factor being the first preset initial value, determine a second data processing strategy; In response to the packet loss rate being greater than or equal to a third threshold, the packet loss factor is increased; or, in response to the packet loss rate being less than the third threshold, the packet loss factor is decreased; wherein, the packet loss factor is adjusted starting from a second preset initial value; in response to the adjusted packet loss factor being greater than or equal to a fourth threshold, a first data processing strategy is determined; or, in response to the adjusted packet loss factor being the second preset initial value, a second data processing strategy is determined. In response to the network jitter being greater than or equal to a fifth threshold, the jitter factor is increased; or, in response to the network jitter being less than the fifth threshold, the jitter factor is decreased; wherein the jitter factor is adjusted starting from a third preset initial value; in response to the adjusted jitter factor being greater than or equal to a sixth threshold, a first data processing strategy is determined; or in response to the adjusted jitter factor being the third preset initial value, a second data processing strategy is determined.

[0084] In some optional embodiments of this application, the first processing unit 14 is configured to perform one or more of the following: In response to the case that the first factor is a delay factor, the audio data is processed according to the first compression rate and / or the first bit rate included in the first data processing strategy, or the audio data is processed according to the second compression rate and / or the second bit rate included in the second data processing strategy, wherein the first compression rate is greater than the second compression rate and the first bit rate is less than the second bit rate; In response to the case where the first factor is a packet loss factor, the previous data packet is recovered using redundant data in the current data packet according to the first data processing strategy, or the audio data in the current data packet is processed according to the second data processing strategy; wherein, the redundant data is key data of the audio data in the previous data packet carried in the current data packet. In response to the case where the first factor is a jitter factor, a preset number of data packets are cached according to the first data processing strategy, and the cached data packets are processed; or, the received data packets are processed directly according to the second data processing strategy.

[0085] In this embodiment, the network status detection unit 12, strategy determination unit 13, and first processing unit 14 in the device can all be implemented by a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU), or a field-programmable gate array (FPGA) in practical applications; the first communication unit 11 in the device can be implemented by a communication module (including: basic communication kit, operating system, communication module, standardized interface and protocol, etc.) and transceiver antenna in practical applications.

[0086] This application also provides a data processing device, which is applied to a cloud desktop server. Figure 11 This is a schematic diagram of the composition structure of the data processing device provided in the embodiments of this application. Figure 2 ;like Figure 11 As shown, the device includes: a second processing unit 22 and a second communication unit 21; wherein, The second processing unit 22 is used to encode audio data in a time sequence to obtain encoded data; and to add frame numbers and timestamps to the encoded data to generate data packets; The second communication unit 21 is used to send the generated data packet to the cloud desktop client; wherein, the timestamp and / or the frame number are used by the cloud desktop client to determine network status information, so that the cloud desktop client processes the audio data according to the data processing strategy corresponding to the network status information; the network status information includes one or more of the following: transmission latency, network jitter, and packet loss rate.

[0087] In some optional embodiments of this application, the second processing unit 22 is used to acquire audio timing data, which includes multiple audio data ordered by time; to collect audio data of a preset duration from the audio timing data; and to encode the audio data of the preset duration and redundant data to obtain encoded data; wherein the redundant data is key data of the audio data corresponding to the frame number preceding the frame number.

[0088] In this embodiment of the application, the second processing unit 22 in the device can be implemented by a CPU, DSP, MCU or FPGA in practical applications; the second communication unit 21 in the device can be implemented by a communication module (including: basic communication kit, operating system, communication module, standardized interface and protocol, etc.) and transceiver antenna in practical applications.

[0089] It should be noted that the data processing apparatus provided in the above embodiments is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the apparatus can be divided into different program modules to complete all or part of the processing described above. In addition, the data processing apparatus and data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0090] This application also provides a communication device, which is a cloud platform client or a cloud platform server. Figure 12 This is a schematic diagram of the hardware composition structure of the communication device provided in the embodiments of this application, such as... Figure 12 As shown, the communication device includes a memory 32, a processor 31, and a computer program stored in the memory 32 and executable on the processor 31. When the processor 31 executes the program, it implements the steps of the data processing method of the present application embodiment.

[0091] Optionally, the communication device may also include at least one network interface 33. The various components in the communication device are coupled together via a bus system 34. It is understood that the bus system 34 is used to implement communication between these components. In addition to a data bus, the bus system 34 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 12 The general labeled all buses as Bus System 34.

[0092] It is understood that memory 32 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 32 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0093] The methods disclosed in the embodiments of this application can be applied to processor 31, or implemented by processor 31. Processor 31 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 31 or by instructions in the form of software. The processor 31 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 31 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 32. Processor 31 reads the information in memory 32 and completes the steps of the aforementioned method in combination with its hardware.

[0094] In an exemplary embodiment, the communication device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0095] In an exemplary embodiment, this application also provides a computer-readable storage medium, such as a memory 32 including a computer program, which can be executed by a processor 31 of a communication device to complete the steps described in the foregoing method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM; or it may be various devices including one or any combination of the above-mentioned memories.

[0096] The computer-readable storage medium provided in this application embodiment stores a computer program thereon, which, when executed by a processor, implements the steps of the data processing method of this application embodiment.

[0097] This application also provides a computer program product, including a computer program that can be executed by a communication device (such as the processor 31 of the communication device) to complete the steps of any of the aforementioned data processing methods.

[0098] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0099] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0100] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0102] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0103] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0104] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0105] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0106] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method is applied to a cloud desktop client, and the method includes: Receive data packets, the data packets including timestamps, frame numbers and audio data; Network status information is determined based on the timestamp and / or the frame number; the network status information includes one or more of the following: transmission latency, network jitter, and packet loss rate. Determine the data processing strategy corresponding to the network status information, and process the audio data according to the data processing strategy.

2. The method according to claim 1, characterized in that, The determination of network status information based on the timestamp and / or the frame number includes one or more of the following: The transmission delay of the data packet is determined based on the timestamp in the data packet and the reception time of the data packet; Network jitter is determined based on the transmission delay of multiple data packets; The packet loss rate is determined based on the frame number in each of the multiple data packets.

3. The method according to claim 1, characterized in that, The step of determining the data processing strategy corresponding to the network status information and processing the audio data according to the data processing strategy includes: The first factor is adjusted based on the network status information, and a first data processing strategy or a second data processing strategy is determined based on the adjusted first factor. The audio data is then processed according to the first data processing strategy or the second data processing strategy. The first data processing strategy is the data processing strategy corresponding to an abnormal network status, and the second data processing strategy is the data processing strategy corresponding to a normal network status.

4. The method according to claim 3, characterized in that, The first factor includes one or more of the following: latency factor, packet loss factor, jitter factor; the step of adjusting the first factor according to the network state information, and determining a first data processing strategy or a second data processing strategy according to the adjusted first factor, includes: In response to the transmission delay being greater than or equal to a first threshold, the delay factor is increased; or, in response to the transmission delay being less than the first threshold, the delay factor is decreased; wherein the delay factor is adjusted starting from a first preset initial value; in response to the adjusted delay factor being greater than or equal to a second threshold, a first data processing strategy is determined; or, in response to the adjusted delay factor being the first preset initial value, a second data processing strategy is determined. In response to the packet loss rate being greater than or equal to a third threshold, the packet loss factor is increased; or, in response to the packet loss rate being less than the third threshold, the packet loss factor is decreased; wherein, the packet loss factor is adjusted starting from a second preset initial value; in response to the adjusted packet loss factor being greater than or equal to a fourth threshold, a first data processing strategy is determined; or, in response to the adjusted packet loss factor being the second preset initial value, a second data processing strategy is determined. In response to the network jitter being greater than or equal to a fifth threshold, the jitter factor is increased; or, in response to the network jitter being less than the fifth threshold, the jitter factor is decreased; wherein the jitter factor is adjusted starting from a third preset initial value; in response to the adjusted jitter factor being greater than or equal to a sixth threshold, a first data processing strategy is determined; or, in response to the adjusted jitter factor being the third preset initial value, a second data processing strategy is determined.

5. The method according to claim 3 or 4, characterized in that, The step of processing the audio data according to the first data processing strategy or the second data processing strategy includes: In response to the case that the first factor is a delay factor, the audio data is processed according to the first compression rate and / or the first bit rate included in the first data processing strategy, or the audio data is processed according to the second compression rate and / or the second bit rate included in the second data processing strategy, wherein the first compression rate is greater than the second compression rate and the first bit rate is less than the second bit rate; In response to the case where the first factor is a packet loss factor, the previous data packet is recovered using redundant data in the current data packet according to the first data processing strategy, or the audio data in the current data packet is processed according to the second data processing strategy; wherein, the redundant data is key data of the audio data in the previous data packet carried in the current data packet. In response to the case where the first factor is a jitter factor, a preset number of data packets are cached according to the first data processing strategy, and the cached data packets are processed; or, the received data packets are processed directly according to the second data processing strategy.

6. A data processing method, characterized in that, The method is applied to a cloud desktop server, and the method includes: The audio data is encoded sequentially to obtain the encoded data; A frame number and a timestamp are added to the encoded data to generate a data packet, and the generated data packet is sent to the cloud desktop client; wherein, the timestamp and / or the frame number are used by the cloud desktop client to determine network status information, so that the cloud desktop client processes the audio data according to the data processing strategy corresponding to the network status information; the network status information includes one or more of the following: transmission latency, network jitter, and packet loss rate.

7. The method according to claim 6, characterized in that, The step of encoding audio data sequentially to obtain encoded data includes: Acquire audio time-series data, which includes multiple audio data items sorted by time; Audio data of a preset duration is collected from audio time-series data. The audio data of the preset duration and redundant data are encoded to obtain encoded data. The redundant data is the key data of the audio data corresponding to the frame number preceding the frame number.

8. A communication device, characterized in that, The communication device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it performs the steps of the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, It includes computer program instructions that cause a computer to perform the steps of the method according to any one of claims 1 to 7.

Images