Data processing method for audio and video streams and cloud background, cloud server and medium thereof
By acquiring historical data of cloud machine audio and video communication and adjusting encoding parameters in real time, the problems of stuttering and clarity in cloud phone audio and video streaming transmission under dynamic network environments have been solved, realizing an adaptive optimization transmission strategy and improving user experience.
Patent Information
- Application Number
- CN202311731141.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-15
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2043-12-15
AI Technical Summary
Existing cloud phone audio and video streaming cannot adapt to dynamic network environments, resulting in stuttering, reduced clarity, and a poor user experience.
By acquiring historical data of cloud machine audio and video communication, encoding parameter decisions are made, and the encoding parameters are adjusted in real time based on the current streaming performance data to achieve an adaptive optimization strategy for audio and video stream transmission.
It maximizes the efficiency of audio and video streaming transmission in cloud phones, improves user experience quality, adapts to network fluctuations, and reduces stuttering and clarity issues.
Smart Images

Figure CN117714744B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of audio and video stream processing, and in particular to a data processing method for audio and video streams, a cloud background for audio and video streams, a cloud server and a computer-readable storage medium. Background Art
[0002] At present, cloud technology has been very mature and has been widely used, such as cloud computers, cloud desktops, cloud services, etc. Cloud phones proposed for mobile phones have also been fully developed. Existing cloud phones generally include clients and cloud servers. Clients can include mobile phones, tablets, computers, etc. The basic requirements are that the client APP can be installed and there is a network. The client connects to the cloud server through the network. When in use, the client runs the APP to connect to the cloud server, that is, based on the application of cloud phones, people can use mobile phones to operate several cloud phones with one client and control several mobile phone control interfaces with one client. As the roles played by users themselves are different, they need to deal with various different things in daily life. Many users carry multiple mobile phones at the same time to deal with different things according to different usage needs. For example, the multiple mobile phones carried include mobile phones specifically for making and receiving calls, mobile phones specifically for handling company affairs, and mobile phones specifically for handling private life trivia. However, carrying multiple mobile phones according to different usage needs will cause inconvenience to users, and they cannot obtain all the required files and information from one mobile phone at any time. Different mobile phones must be opened to obtain different files and information. Through cloud phones, the convenience of users can be improved and the security of user data can be protected.
[0003] The terminal data transmission of cloud phones is highly dependent on the network environment. The current network environment changes dynamically, and the cloud server and cloud phone terminals use a single data transmission method that cannot adapt to the current dynamic network environment, resulting in the audio and video streams of cloud phones being stuck and the clarity being reduced, and users having a poor experience using cloud phone services. Summary of the invention
[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a data processing method for audio and video streams, a cloud background for audio and video streams, a cloud server and a computer-readable storage medium, which can adaptively optimize the transmission strategy of audio and video streams and maximize the transmission efficiency of audio and video streams of cloud phones.
[0005] In order to solve at least one of the above technical problems, an embodiment of the present invention provides a data processing method for audio and video streams, which is applied to a terminal-pipe-cloud network big data analysis module of a cloud server, and the data processing method includes:
[0006] Obtain historical data of audio and video communications between cloud machines and make encoding parameter decisions based on the historical data;
[0007] The encoding parameter decision is sent to the cloud machine sending end, and the cloud machine sending end sends a first audio and video stream to the client based on the encoding parameter decision, so that the client receives and plays the first audio and video stream;
[0008] Receive current streaming performance data sent by the client, update the encoding parameter decision according to the current streaming performance data, and send the updated encoding parameter decision to the cloud machine sending end, so that the cloud machine sending end sends a second audio and video stream to the client based on the updated encoding parameter decision.
[0009] Preferably, the current streaming performance data includes the current network status of the client and the current audio and video playback status of the client; the method further comprises: obtaining the current encoding status of the cloud machine sending end;
[0010] The updating of the encoding parameter decision according to the current streaming performance data comprises:
[0011] Performing network mode perception on the current network status of the client to obtain the current network mode;
[0012] Determining an updated bit rate according to the current network mode;
[0013] The encoding parameter decision is updated according to the updated bit rate, the current audio and video playback status of the client, and the current encoding status of the cloud machine sending end.
[0014] Preferably, the updated encoding parameter decision is used to instruct the cloud machine sending end to adjust the encoding rate and sending bit rate so as to send the second audio and video stream to the client using the adjusted encoding rate and sending bit rate.
[0015] Preferably, the historical data includes the historical encoding status of the cloud machine sending end, the historical network status of the client, and the historical audio and video playback status of the client;
[0016] The obtaining of coding parameter decisions based on historical data includes: performing data analysis on the historical coding status, the historical network status, and the historical audio and video playback status to obtain coding parameter decisions.
[0017] A cloud backend for audio and video streams, the cloud backend comprising a cloud server end-to-end cloud network big data analysis module and a cloud machine sending end;
[0018] The end-to-end cloud network big data analysis module obtains historical data of cloud machine audio and video communications, obtains coding parameter decisions based on the historical data, and sends the coding parameter decisions to the cloud machine sending end;
[0019] The cloud machine sending end sends the first audio and video stream to the client based on the encoding parameter decision, so that the client receives and plays the first audio and video stream;
[0020] The end-pipe-cloud network big data analysis module receives the current streaming performance data sent by the client, updates the encoding parameter decision according to the current streaming performance data, and sends the updated encoding parameter decision to the cloud machine sending end;
[0021] The cloud machine sending end decides to send a second audio and video stream to the client based on the updated encoding parameter.
[0022] Preferably, the cloud machine sending end sends the first audio and video stream to the client based on the encoding parameter decision, including:
[0023] The cloud machine sending end obtains the first audio and video stream;
[0024] The cloud machine sending end determines the adjusted coding parameters according to the coding parameter decision, and determines the bit rate according to the adjusted coding parameters;
[0025] The cloud machine transmitting end performs audio and video stream encoding on the first audio and video stream according to the determined bit rate and then performs audio and video streaming to obtain an audio and video stream;
[0026] The cloud machine sending end sends the audio and video stream to the client.
[0027] Preferably, after decoding and playing the audio and video stream, the client collects network status data and playback status data for playing the audio and video stream, and generates current streaming performance data based on the network status data and playback status data.
[0028] Preferably, the end-to-end cloud network big data analysis module obtains the current encoding status of the cloud machine sending end, performs network modality perception on the current network status of the client, obtains the current network mode, determines an updated bit rate based on the current network mode, and updates the encoding parameter decision based on the updated bit rate, the current audio and video playback status of the client, and the current encoding status of the cloud machine sending end.
[0029] A cloud server comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the above embodiments when executing the computer program.
[0030] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in any of the above embodiments.
[0031] The above-mentioned data processing method of an audio and video stream, a cloud background of an audio and video stream, a cloud server and a computer-readable storage medium, include: obtaining historical data of audio and video communication of a cloud machine, and obtaining a coding parameter decision based on the historical data; sending the coding parameter decision to the cloud machine sending end, and the cloud machine sending end sending a first audio and video stream to the client based on the coding parameter decision, so that the client receives and plays the first audio and video stream; receiving the current streaming performance data sent by the client, updating the coding parameter decision according to the current streaming performance data, and sending the updated coding parameter decision to the cloud machine sending end, so that the cloud machine sending end sends a second audio and video stream to the client based on the updated coding parameter decision. Therefore, after obtaining the coding parameter decision based on the historical data, the cloud machine sending end sends the first audio and video stream to the client based on the coding parameter decision. At the same time, the current streaming performance data of the client is received in real time, and the coding parameter decision is updated based on this, and the updated coding parameter decision is used to send the subsequent second audio and video stream to the client, so as to realize the adaptive optimization of the transmission strategy of the audio and video stream and maximize the transmission efficiency of the audio and video stream of the cloud phone. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 It is a flowchart of a method for processing audio and video stream data in an embodiment of the present invention;
[0033] Figure 2 It is a structural block diagram of a cloud background of audio and video streams in an embodiment of the present invention;
[0034] Figure 3 It is a system architecture diagram of a data processing method for audio and video streams in an embodiment of the present invention. DETAILED DESCRIPTION
[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0036] The embodiment of the present invention provides a method for processing audio and video stream data. Figure 1 As shown, a data processing method for audio and video streams is applied to the end-to-end cloud network big data analysis module of the cloud server, comprising the following steps:
[0037] S101, obtaining historical data of audio and video communications between the cloud machine and making coding parameter decisions based on the historical data.
[0038] In one example, the historical data includes the historical encoding status of the cloud machine sending end, the historical network status of the client, and the historical audio and video playback status of the client; the encoding parameter decision is obtained based on the historical data, including: performing data analysis on the historical encoding status, the historical network status, and the historical audio and video playback status to obtain the encoding parameter decision.
[0039] Specifically, the end-pipe-cloud big data analysis module first observes the historical data of previous cloud-machine audio and video communications, that is, the encoding status, network status, and audio and video playback status collected from the video encoder and the receiving end, and uses data analysis to mine the potential characteristics of the video content and network during the encoding and transmission process, and makes decisions on the encoding parameter settings, that is, encoding parameter decisions.
[0040] S102, sending the encoding parameter decision to the cloud machine sending end, and the cloud machine sending end sends a first audio and video stream to the client based on the encoding parameter decision, so that the client receives and plays the first audio and video stream.
[0041] Specifically, the sending end of the audio and video application, the cloud machine sending end, decides to encode and transmit the audio and video based on the encoding parameters. After decoding and playback by the client at the receiving end, a new state is generated, and the current streaming performance data is fed back to the end-to-end cloud network big data analysis module of the cloud server.
[0042] S103, receiving the current streaming performance data sent by the client, updating the encoding parameter decision according to the current streaming performance data, and sending the updated encoding parameter decision to the cloud machine sending end, so that the cloud machine sending end sends a second audio and video stream to the client based on the updated encoding parameter decision.
[0043] Specifically, based on the feedback of current streaming performance data, the end-pipe-cloud network big data analysis module continuously updates the encoding parameter decision and adjusts the recommendation algorithm. Ultimately, by simply observing and learning the encoding status, network status, and audio and video playback status, the encoding parameters of audio and video are adaptively adjusted, effectively combating network fluctuations while effectively improving the quality of user experience.
[0044] In one example, the current streaming performance data includes the current network status of the client and the current audio and video playback status of the client; the method also includes: obtaining the current encoding status of the cloud machine sending end; updating the encoding parameter decision based on the current streaming performance data, including: performing network mode perception on the current network status of the client to obtain the current network mode; determining an updated bit rate based on the current network mode; updating the encoding parameter decision based on the updated bit rate, the current audio and video playback status of the client, and the current encoding status of the cloud machine sending end.
[0045] Among them, the updated encoding parameter decision is used to instruct the cloud machine sending end to adjust the encoding rate and sending bit rate, so as to use the adjusted encoding rate and sending bit rate to send the second audio and video stream to the client.
[0046] The above-mentioned method for processing audio and video stream data includes: obtaining historical data of audio and video communication of the cloud machine, and obtaining a coding parameter decision based on the historical data; sending the coding parameter decision to the cloud machine sending end, and the cloud machine sending end sending a first audio and video stream to the client based on the coding parameter decision, so that the client receives and plays the first audio and video stream; receiving the current streaming performance data sent by the client, updating the coding parameter decision according to the current streaming performance data, and sending the updated coding parameter decision to the cloud machine sending end, so that the cloud machine sending end sends a second audio and video stream to the client based on the updated coding parameter decision. Therefore, after obtaining the coding parameter decision based on the historical data, the cloud machine sending end sends the first audio and video stream to the client based on the coding parameter decision. At the same time, the current streaming performance data of the client is received in real time, and the coding parameter decision is updated based on this, and the updated coding parameter decision is used to send the subsequent second audio and video stream to the client, so as to realize the adaptive optimization of the transmission strategy of the audio and video stream and maximize the transmission efficiency of the audio and video stream of the cloud phone.
[0047] The present invention also provides a cloud background for audio and video streams, such as Figure 2 As shown, the cloud backend includes an end-to-end cloud network big data analysis module of the cloud server and a cloud machine sending end; the end-to-end cloud network big data analysis module obtains historical data of cloud machine audio and video communications, obtains encoding parameter decisions based on the historical data, and sends the encoding parameter decisions to the cloud machine sending end; the cloud machine sending end sends a first audio and video stream to the client based on the encoding parameter decision, so that the client receives and plays the first audio and video stream; the end-to-end cloud network big data analysis module receives current streaming performance data sent by the client, updates the encoding parameter decision according to the current streaming performance data, and sends the updated encoding parameter decision to the cloud machine sending end; the cloud machine sending end sends a second audio and video stream to the client based on the updated encoding parameter decision.
[0048] Specifically, the coding parameter decisions sent by the end-to-end cloud network big data analysis module to the cloud machine transmitter include the coding parameter decisions before the update and the coding parameter decisions after the update. That is, it includes all types of coding parameter decisions sent. The audio and video streams sent by the cloud machine transmitter to the client include the first audio and video stream and the second audio and video stream. The first audio and video stream and the second audio and video stream refer to audio and video streams sent at different times. The audio and video streams sent by the cloud machine transmitter to the client include all audio and video streams sent. The current streaming performance data sent by the client is the streaming performance data obtained at each current moment.
[0049] In one example, the cloud machine transmitter sends a first audio and video stream to a client based on the encoding parameter decision, including: the cloud machine transmitter obtains the first audio and video stream; the cloud machine transmitter determines the adjusted encoding parameters according to the encoding parameter decision, and determines the bit rate according to the adjusted encoding parameters; the cloud machine transmitter encodes the first audio and video stream according to the determined bit rate and then performs audio and video streaming to obtain an audio and video stream; the cloud machine transmitter sends the audio and video stream to the client.
[0050] In one example, after the client decodes and plays the audio and video stream, it collects network status data and playback status data for playing the audio and video stream, and generates current streaming performance data based on the network status data and playback status data.
[0051] In one example, the end-to-end cloud network big data analysis module obtains the current encoding status of the cloud machine sending end, performs network modality perception on the current network status of the client, obtains the current network mode, determines an updated bit rate based on the current network mode, and updates the encoding parameter decision based on the updated bit rate, the current audio and video playback status of the client, and the current encoding status of the cloud machine sending end.
[0052] Specifically, Figure 3As shown in the figure, the end-to-end cloud network big data analysis module introduces network multimodal perception for online analysis and inference coding decisions. After audio and video acquisition, audio and video encoding, and audio and video streaming, the Android cloud machine transmitter sends the audio and video stream to the network. The receiving end receives the audio and video data for decoding and playback, and collects data related to network status and audio and video playback. The network modality perception, bit rate adaptation, and online modeling and inference modules are deployed in the end-to-end cloud network big data analysis module of the backend service. The backend and the cloud machine transmitter have a direct message channel to transmit the coding parameter decision data. First, by analyzing and mining the historical network status characteristics, the current network mode is identified, and the overall network quality is initially perceived. After that, the multimodal bit rate adaptation based on reinforcement learning theory is used to dynamically switch the bit rate adaptation model according to the network mode, so as to finely generate different bit rate adaptation decision algorithms for different network conditions. In the actual new network environment, online analysis and inference coding decisions are applied based on the multimodal bit rate adaptation model, and the model and algorithm are further updated and optimized for the current environment to solve the performance degradation problem of the recommendation model in the new environment. After a bitrate decision is made, the audio and video encoding and audio and video streaming modules at the sending end receive the notification and guide to adjust the encoding bitrate and sending bitrate to resist dynamic network fluctuations.
[0053] A computer-readable storage medium is provided in an embodiment of the present invention, wherein an application is stored on the computer-readable storage medium, and when the program is executed by a processor, a data processing method for an audio and video stream of any one of the above embodiments is implemented. The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the storage device includes any medium that can store or transmit information in a readable form by a device (for example, a computer, a mobile phone), which can be a read-only memory, a disk or an optical disk, etc.
[0054] An embodiment of the present invention further provides a computer application program, which runs on a computer and is used to execute a data processing method for an audio or video stream according to any one of the above embodiments.
[0055] An embodiment of the present invention also provides a cloud server, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the data processing method for an audio or video stream described in any one of the above items are implemented.
[0056] In addition, the above provides a detailed introduction to a data processing method for an audio and video stream, a cloud background for an audio and video stream, a cloud server and a computer-readable storage medium provided in an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for a person skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A method for processing audio and video stream data, characterized in that: The end-to-end cloud network big data analysis module applied to the cloud server, the data processing method includes: Obtain historical data of audio and video communications between cloud machines and make encoding parameter decisions based on the historical data; The encoding parameter decision is sent to the cloud machine sending end, and the cloud machine sending end sends the first audio and video stream to the client based on the encoding parameter decision, so that the client receives and plays the first audio and video stream; the sending end of the audio and video application, the cloud machine sending end, encodes and transmits the audio and video based on the encoding parameter decision, and after the receiving end client decodes and plays, a new state is generated, and the current streaming performance data is fed back to the end-to-end cloud network big data analysis module of the cloud server; Receiving current streaming performance data sent by the client, updating the encoding parameter decision according to the current streaming performance data, and sending the updated encoding parameter decision to the cloud machine sending end, so that the cloud machine sending end sends a second audio and video stream to the client based on the updated encoding parameter decision; The historical data includes the historical encoding status of the cloud machine sender, the historical network status of the client, and the historical audio and video playback status of the client; The obtaining of the coding parameter decision according to the historical data includes: performing data analysis on the historical coding state, the historical network status and the historical audio and video playback status to obtain the coding parameter decision; The end-to-end cloud network big data analysis module first observes the historical data of cloud-machine audio and video communications, namely the encoding status, network status, and audio and video playback status collected from the video encoder and the receiving end, and uses data analysis to mine the potential characteristics of the video content and network during the encoding and transmission process to make decisions on encoding parameter settings, namely encoding parameter decisions; Based on the feedback of current streaming performance data, the end-to-end cloud network big data analysis module continuously updates encoding parameter decisions and adjusts the recommendation algorithm; The current streaming performance data includes the current network status of the client and the current audio and video playback status of the client; the method further includes: obtaining the current encoding status of the cloud machine sending end; The updating of the encoding parameter decision according to the current streaming performance data comprises: Performing network mode perception on the current network status of the client to obtain the current network mode; Determining an updated bit rate according to the current network mode; The encoding parameter decision is updated according to the updated bit rate, the current audio and video playback status of the client, and the current encoding status of the cloud machine sending end.
2. The data processing method according to claim 1, characterized in that: The updated encoding parameter decision is used to instruct the cloud machine sending end to adjust the encoding rate and sending bit rate so as to use the adjusted encoding rate and sending bit rate to send the second audio and video stream to the client.
3. A cloud background for audio and video streaming, characterized in that: Used to execute the data processing method according to any one of claims 1-2, the cloud backend includes a cloud server end-to-end cloud network big data analysis module and a cloud machine sending end; The end-to-end cloud network big data analysis module obtains historical data of cloud machine audio and video communications, obtains coding parameter decisions based on the historical data, and sends the coding parameter decisions to the cloud machine sending end; The cloud machine sending end sends the first audio and video stream to the client based on the encoding parameter decision, so that the client receives and plays the first audio and video stream; The end-pipe-cloud network big data analysis module receives the current streaming performance data sent by the client, updates the encoding parameter decision according to the current streaming performance data, and sends the updated encoding parameter decision to the cloud machine sending end; The cloud machine sending end sends a second audio and video stream to the client based on the updated encoding parameter decision; The historical data includes the historical encoding status of the cloud machine sender, the historical network status of the client, and the historical audio and video playback status of the client; The obtaining of the coding parameter decision according to the historical data includes: performing data analysis on the historical coding state, the historical network status and the historical audio and video playback status to obtain the coding parameter decision; The end-pipe-cloud network big data analysis module first observes the historical data of previous cloud-machine audio and video communications, that is, the encoding status, network status, and audio and video playback status collected from the video encoder and the receiving end, and uses data analysis to mine the potential characteristics of the video content and network during the encoding and transmission process, and makes decisions on the encoding parameter settings, that is, encoding parameter decisions.
4. A cloud server comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 2 are implemented.
5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.
