A data processing method, cloud phone, operating device and medium

Abstract

The present application relates to the field of cloud phone audio and video processing technology, and in particular to a data processing method, cloud phone, operating equipment and medium, including analyzing network information according to a pre-constructed compensation prediction model to obtain a prediction result; when the prediction result is that there is an abnormality, decoding the video information with the abnormality according to the prediction result to obtain first decoding information, performing video data compensation on the first decoding information to obtain first compensation information; when the prediction result is that there is an abnormality, decoding the audio information with the abnormality according to the prediction result to obtain second decoding information, performing audio data compensation on the second decoding information to obtain second compensation information; post-processing the first compensation information and the second compensation information to obtain audio and video information for sending to a receiving device, thereby improving the stability of the cloud phone for video calls, and thereby improving the user experience of the cloud phone for video calls.

Description

Technical Field

[0001] The present application relates to the technical field of cloud phone audio and video processing, and in particular to a data processing method, a cloud phone, an operating device and a medium. Background Art

[0002] Cloud phone technology, also known as virtual phone technology, virtualizes the functions of smartphones through cloud computing technology, allowing users to access and operate virtual phones through remote devices. Cloud phones can not only provide traditional smartphone applications and services, but also achieve an operating environment isolated from physical devices at the hardware and system levels, thereby bringing higher flexibility and scalability. With the popularization of the Internet and the advancement of cloud computing technology, cloud phones are widely used in various scenarios, including games, office work, video conferencing, etc. Among them, video call software has become one of the important applications of cloud phone technology because it can provide high-quality remote communication experience.

[0003] The advantage of cloud phone technology is that it transfers applications and data processing from local devices to cloud servers, which greatly reduces the burden on hardware devices. However, the smoothness, clarity and real-time interactive experience of video call software are highly dependent on network quality and cloud device resources. Network anomalies are changeable. When the network bandwidth is unstable or insufficient, the transmission of video data packets will become unsmooth, resulting in freezes or delays in video images.

[0004] At present, in order to make video calls more stable when using cloud phones, a more stable and faster network is usually used to provide technical support, and more efficient encoding and decoding technology and compression technology are used for data processing. However, this method cannot cope with changing abnormal problems, affecting the stability of cloud phones for video calls and the user experience of cloud phones for video calls. The above problems need to be solved. Summary of the invention

[0005] In order to improve the stability of cloud phones for video calls, and thus improve the user experience of cloud phones for video calls, this application provides a data processing method, a cloud phone, an operating device and a medium, which adopt the following technical solutions:

[0006] In a first aspect, the present application provides a cloud phone data processing method, comprising:

[0007] Get the video and audio information of the running device;

[0008] Obtain network information, analyze the network information according to the pre-built compensation prediction model, and obtain prediction results;

[0009] When the prediction result indicates that an abnormality exists, the video information with the abnormality is decoded according to the prediction result to obtain first decoded information, and video data compensation is performed on the first decoded information to obtain first compensation information;

[0010] When the prediction result indicates that an abnormality exists, the audio information with the abnormality is decoded according to the prediction result to obtain second decoded information, and audio data compensation is performed on the second decoded information to obtain second compensated information;

[0011] The first compensation information and the second compensation information are post-processed to obtain audio and video information for sending to a receiving device.

[0012] Preferably, it also includes:

[0013] When the prediction result is normal, the video information is decoded to obtain third decoded information, and the running page information is adjusted according to the third decoded information;

[0014] The running page image is acquired, and the running page image is encoded and compressed to obtain page image information for feeding back to the running device.

[0015] Preferably, it also includes:

[0016] When the prediction result is normal, the video information and audio information of the running device are sent to the receiving device.

[0017] Preferably, it also includes:

[0018] Historical network information, historical video information, historical audio information and historical compensation information are obtained, and a compensation prediction model is constructed according to the historical network information, historical video information, historical audio information and historical compensation information.

[0019] Preferably, the specific steps of performing video data compensation on the first decoded information to obtain the first compensation information are:

[0020] Performing frame anomaly analysis on the first decoded information to obtain missing frame position information;

[0021] Acquire adjacent frame images of missing frame position information, analyze the previous adjacent frame image and the next adjacent frame image, and obtain compensation frame data;

[0022] The missing frame position is compensated according to the compensation frame data to obtain first compensation information.

[0023] Preferably, the specific steps of post-processing the first compensation information and the second compensation information to obtain the audio and video information for sending to the receiving device are:

[0024] Acquire time information of the first compensation information and the second compensation information, and perform synchronization processing on the first compensation information and the second compensation information according to the time information to obtain synchronization data;

[0025] The synchronous data is encoded and compressed to obtain audio and video information for sending to a receiving device.

[0026] In a second aspect, the present application provides a cloud phone, including:

[0027] An audio and video information acquisition module is used to acquire video information and audio information of a running device;

[0028] A network information acquisition module is used to acquire network information, analyze the network information according to a pre-built compensation prediction model, and obtain a prediction result;

[0029] A first decoding module is used for, when the prediction result indicates that an abnormality exists, decoding the video information with the abnormality according to the prediction result to obtain first decoded information, and performing video data compensation on the first decoded information to obtain first compensation information;

[0030] A second decoding module is used for, when the prediction result indicates that an abnormality exists, decoding the audio information with the abnormality according to the prediction result to obtain second decoded information, and performing audio data compensation on the second decoded information to obtain second compensation information;

[0031] The first information sending module is used to post-process the first compensation information and the second compensation information to obtain audio and video information for sending to a receiving device.

[0032] Preferably, it also includes:

[0033] A third decoding module is used to decode the video information to obtain third decoding information when the prediction result is normal, and adjust the running page information according to the third decoding information;

[0034] The running page image is acquired, and the running page image is encoded and compressed to obtain page image information for feeding back to the running device.

[0035] Preferably, it also includes:

[0036] The second information sending module is used to send the video information and audio information of the running device to the receiving device when the prediction result is normal.

[0037] Preferably, it also includes:

[0038] The model building module is used to obtain historical network information, historical video information, historical audio information and historical compensation information, and build a compensation prediction model based on the historical network information, historical video information, historical audio information and historical compensation information.

[0039] Preferably, the first decoding module performs video data compensation on the first decoded information to obtain the first compensation information in the following specific steps:

[0040] Performing frame anomaly analysis on the first decoded information to obtain missing frame position information;

[0041] Acquire adjacent frame images of missing frame position information, analyze the previous adjacent frame image and the next adjacent frame image, and obtain compensation frame data;

[0042] The missing frame position is compensated according to the compensation frame data to obtain first compensation information.

[0043] Preferably, the first information sending module performs post-processing on the first compensation information and the second compensation information to obtain the audio and video information for sending to the receiving device in the following specific steps:

[0044] Acquire time information of the first compensation information and the second compensation information, and perform synchronization processing on the first compensation information and the second compensation information according to the time information to obtain synchronization data;

[0045] The synchronous data is encoded and compressed to obtain audio and video information for sending to a receiving device.

[0046] In a third aspect, the present application provides a method for processing operating device data, comprising:

[0047] Get real-time collection data;

[0048] The real-time collected data is encoded and compressed to obtain video information and audio information for feedback to the cloud phone;

[0049] Obtain page image information, decode the page image information, and obtain audio and video information acting on the running device.

[0050] Preferably, it also includes:

[0051] Get abnormal instruction information;

[0052] When abnormal instruction information is obtained, the page image information is subjected to screen recognition and extraction to obtain fixed screen information;

[0053] Obtain partial image information of the cloud phone, and construct audio and video information acting on the running device based on real-time collected data, fixed screen information and partial image information.

[0054] Preferably, the real-time collected data includes real-time video data and real-time audio data.

[0055] In a fourth aspect, the present application provides an operating device, including:

[0056] Real-time data collection module, used to obtain real-time collection data;

[0057] The acquisition data processing module is used to encode and compress the real-time acquisition data to obtain video information and audio information for feedback to the cloud phone;

[0058] The page image processing module is used to obtain page image information, decode the page image information, and obtain audio and video information acting on the running device.

[0059] Preferably, it also includes:

[0060] An abnormal data processing module is used to obtain abnormal instruction information;

[0061] When abnormal instruction information is obtained, the page image information is subjected to screen recognition and extraction to obtain fixed screen information;

[0062] Obtain partial image information of the cloud phone, and construct audio and video information acting on the running device based on real-time collected data, fixed screen information and partial image information.

[0063] Preferably, the real-time collected data of the real-time data collection module includes real-time video data and real-time audio data.

[0064] In a fifth aspect, the present application provides a device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method as described above.

[0065] In a sixth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the method as described above when run.

[0066] In summary, compared with the prior art, the technical solution provided by this application has at least the following beneficial effects:

[0067] The present application obtains real-time collected data by running a device, and obtains video information and audio information by encoding and compressing the real-time collected data. The cloud phone analyzes whether the current network situation will cause network abnormalities and thus data packets of the video information or audio information will be lost based on the network information. When the prediction result is that there is an abnormality, the obtained video information and audio information are decoded to obtain first decoding information and second decoding information, and data compensation is performed on the first decoding information and the second decoding information respectively to obtain first compensation information and second compensation information. Audio and video information sent to a receiving device is obtained based on the processing of the first compensation information and the second compensation information, so that video data and audio data with missing data are compensated through prediction, thereby reducing the video or audio freeze of the opposite device, improving the stability of the cloud phone in making video calls, and thereby improving the user experience of the cloud phone in making video calls. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flow chart of a cloud phone data processing method described in an embodiment of the present application.

[0069] Figure 2 It is a module diagram of the cloud phone described in the embodiment of the present application.

[0070] Figure 3 It is a flow chart of a method for processing data of an operating device described in an embodiment of the present application.

[0071] Figure 4 It is a module schematic diagram of the operating device described in the embodiment of the present application.

[0072] Figure 5 It is a module schematic diagram of the electronic device described in the embodiment of the present application.

[0073] Description of reference numerals:

[0074] 1. Audio and video information acquisition module; 2. Network information acquisition module; 3. First decoding module; 4. Second decoding module; 5. First information sending module; 6. Real-time data acquisition module; 7. Acquisition data processing module; 8. Page image processing module. DETAILED DESCRIPTION

[0075] The following combination Figure 1-Figure 5 The present application is described in further detail. The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting.

[0076] Reference Figure 1 , a cloud phone data processing method involved in this application specifically includes:

[0077] Get the video and audio information of the running device;

[0078] Obtain network information, analyze the network information according to the pre-built compensation prediction model, and obtain prediction results;

[0079] When the prediction result indicates that an abnormality exists, the video information with the abnormality is decoded according to the prediction result to obtain first decoded information, and video data compensation is performed on the first decoded information to obtain first compensation information;

[0080] When the prediction result indicates that an abnormality exists, the audio information with the abnormality is decoded according to the prediction result to obtain second decoded information, and audio data compensation is performed on the second decoded information to obtain second compensated information;

[0081] The first compensation information and the second compensation information are post-processed to obtain audio and video information for sending to a receiving device.

[0082] Specifically, the present application obtains real-time acquisition data by running the device, and obtains video information and audio information by encoding and compressing the real-time acquisition data. The cloud phone analyzes whether the current network situation will cause network abnormalities and thus video information or audio information packet loss according to the network information. The impact of the network situation on the video information and audio information is not directly judged by the current network delay, so after obtaining the network information, a compensation prediction model is used to predict and obtain the prediction result. In the case where the prediction result is that there is an abnormality, the acquired video information and audio information are decoded respectively. After decoding to obtain the first decoding information and the second decoding information, the first decoding information and the second decoding information are respectively compensated for data to obtain the first compensation information and the second compensation information. According to the processing of the first compensation information and the second compensation information, the audio and video information sent to the receiving device is obtained, so that the video data and audio data with missing data are compensated by prediction. The present application decodes and compensates the video information and audio information transmitted to the cloud phone by the running device in a targeted manner, so that when the video information and audio information are lost, the video or audio freeze of the opposite receiving device is reduced, the stability of the cloud phone for video calls is improved, and the use experience of the cloud phone for video calls is improved.

[0083] The execution end involved in realizing the video call function of the cloud phone in the embodiment of the present application includes a cloud phone, an operating device and a receiving device. The user opens the corresponding application for running and controlling the cloud phone system through an operating device with a camera and a microphone such as a mobile phone, tablet computer, laptop or desktop computer, and connects to the cloud phone. The operating device and the cloud phone are usually connected wirelessly through the network. At this time, executing a high-quality video call will only occupy the resources of the cloud phone end, while the operating device itself occupies fewer resources and is usually only responsible for sending audio and video information and receiving and processing image information, thereby improving the user experience of video calls.

[0084] For example, network delays or insufficient bandwidth may cause delays or packet loss in the video and audio data captured on the device when transmitted to the cloud phone. If the network is unstable, the camera video or microphone audio may not be transmitted to the cloud phone in a timely or complete manner, resulting in picture or sound loss, which is usually manifested as freeze, silence or unclear images.

[0085] For example, when the cloud phone receives and parses data, network anomalies such as packet loss and delay will cause the data received by the cloud phone to be incomplete or out of order. The cloud phone may encounter problems when parsing video and audio data. If the data loss during the parsing process is serious, the cloud phone will not be able to correctly process the audio and video streams, and the screen may become stuck or the audio and video may be out of sync.

[0086] For example, when the cloud phone sends the parsed audio and video data and the cloud phone sends the parsed and synchronized audio and video data back to the device, network instability, insufficient bandwidth, or packet loss may cause the quality of the audio and video stream sent by the cloud phone to deteriorate.

[0087] For example, when the cloud phone returns the audio and video streams to the device after synchronization, if the network delay or packet loss is severe, the synchronization of video and audio may be disrupted. The asynchrony of video and audio streams will cause the sound to not match the picture.

[0088] This application is mainly aimed at the situation where there is a network anomaly between the running device and the cloud phone. In the process of transmitting video information and audio information that occupy a lot of resources and have high quality, in addition to adopting network technology with faster transmission speed, lower latency usage environment, and more advanced compression and decompression encoding and decoding technology, it also predicts and judges the network information related to the cloud phone, and further processes the acquired audio and video information, so as to achieve targeted data processing.

[0089] Specifically, when the network delay is abnormal, it may have no effect or little effect on the data information transmitted by the running device and the cloud phone, or it may cause data loss during the transmission process, resulting in problems such as freezing and sluggishness after the receiving device receives the information. It can be seen that due to the ever-changing situation during the operation, the derived problems and the degree of problems are different for different usage scenarios and conditions. Before executing the data processing step, the embodiment of the present application needs to analyze and judge the network information, specifically the network delay situation, so as to analyze whether the video information and audio information transmitted from the running device to the cloud phone may be missing or disordered, so as to process and send and receive the video information and audio information more targeted.

[0090] As one implementation mode, it also includes:

[0091] When the prediction result is normal, the video information is decoded to obtain third decoded information, and the running page information is adjusted according to the third decoded information;

[0092] The running page image is acquired, and the running page image is encoded and compressed to obtain page image information for feeding back to the running device.

[0093] When the prediction result is normal, the video information and audio information of the running device are sent to the receiving device.

[0094] Specifically, the running device of the embodiment of the present application communicates with the receiving device through the cloud phone. That is, when the network is normal, the cloud phone will directly forward the encoded and compressed video information and audio information received from the running device to the receiving device, so that the receiving device can receive the data collected by the camera and microphone of the running device, thereby realizing real-time data transmission of the video call.

[0095] While forwarding the encoded and compressed video information and audio information directly to the receiving device, the cloud phone will decompress and decode the encoded and compressed video information and audio information, so that the running page of the cloud phone system can be updated through this part of video data and audio data, and the screen information of the running page, that is, the running page image, will be collected and encoded and compressed, and this part of information will be fed back to the running device, and the display content and the played audio content of the running device will be updated to realize real-time data transmission of video calls.

[0096] The embodiment of the present application can, when the network is normal, copy the video information and audio information and process them synchronously, and forward the video information and audio information that are judged to be normal at the fastest speed, so that the receiving device can receive them in the first time and process them as soon as possible. During the video call, the time spent on transmitting and processing information by the devices at both ends is minimized, and the display screen and audio playback of the receiving device are timely.

[0097] As one implementation mode, it also includes:

[0098] Historical network information, historical video information, historical audio information and historical compensation information are obtained, and a compensation prediction model is constructed according to the historical network information, historical video information, historical audio information and historical compensation information.

[0099] Specifically, the embodiments of the present application use a compensation prediction model to monitor the network status in real time, analyze and predict the network status, and then determine whether data loss or quality degradation will occur during the transmission of video and audio data.

[0100] When constructing a compensation prediction model, it is necessary to call historical information and construct or train a model through the historical information, so as to obtain a compensation prediction model that is constructed and capable of analyzing the current network status.

[0101] Among them, historical network information includes historical network delay, bandwidth, packet loss rate, network fluctuation and other data. Network data can help the model identify potential problems in the network, such as network instability, insufficient bandwidth, etc., which in turn affect the quality of video and audio streams.

[0102] Historical video information includes the transmission quality, frame loss, and video quality of past video data. The decoding and transmission quality of the video are affected by factors such as the network environment and compression quality.

[0103] The quality of historical audio information, packet loss, audio quality fluctuations, etc. will affect the transmission effect of the audio stream.

[0104] The historical compensation information specifically includes the compensation measures that have been taken and the corresponding effects, including data retransmission, error correction, video frame interpolation and other methods.

[0105] Based on historical network information, historical video information, historical audio information and historical compensation information, the compensation prediction model can capture the laws and patterns of video and audio stream loss in different network environments, and then predict whether packet loss or quality degradation will occur under the current network status.

[0106] In the embodiments of the present application, since the compensation prediction model needs to process multi-dimensional data from multiple sources, a machine learning model is usually used for training. The model usually adopts a regression model, a classification model, a deep learning model, a reinforcement learning model, etc.

[0107] Among them, regression models such as linear regression and ridge regression can be used to predict the quality score or packet loss probability of video and audio data under network conditions.

[0108] Classification models such as support vector machines, decision trees, random forests, and neural networks are used to determine whether video and audio streams will experience packet loss under the current network status.

[0109] Deep learning models such as long short-term memory network (LSTM) and convolutional neural network (CNN) are suitable for processing data with time series characteristics, such as network status changes, historical video and audio quality, etc. Deep learning models can automatically extract complex features from data and adapt to dynamically changing network environments.

[0110] By adopting the reinforcement learning model, when the network conditions are changing greatly and uncertain, reinforcement learning can gradually adjust the prediction and compensation strategies according to environmental feedback to find the optimal compensation solution.

[0111] During the training phase, the model learns from a large amount of historical data to establish the relationship between network status and video and audio quality loss, so that it can predict possible future packet loss or data distortion based on the current network status during real-time operation.

[0112] In actual operation of the embodiment of the present application, the compensation prediction model receives the current network information in real time, and predicts the possible loss or quality problems of video and audio data during transmission based on the learning results of the historical data model. When the model predicts that the network condition may cause packet loss, it performs related operations of exception handling.

[0113] As one implementation manner, the specific steps of performing video data compensation on the first decoded information to obtain the first compensation information are:

[0114] Performing frame anomaly analysis on the first decoded information to obtain missing frame position information;

[0115] Acquire adjacent frame images of missing frame position information, analyze the previous adjacent frame image and the next adjacent frame image, and obtain compensation frame data;

[0116] The missing frame position is compensated according to the compensation frame data to obtain first compensation information.

[0117] Specifically, the embodiments of the present application use a prediction mechanism to predict the playback timing of the next video frame or audio packet to address issues such as audio and video delay and packet loss, and to compensate for the delay in the video frame if it is large.

[0118] The data loss in the embodiments of the present application is usually caused by unstable network transmission, packet loss or frame loss. After receiving the video data, the cloud phone first analyzes the decoded data to find out which frame data is lost or damaged. Specifically, the cloud phone will check the video data of each frame or several data segments of a certain data segment size, identify which frames are missing, and the missing frames will be recorded as "missing frame position information". This identification information enables the system to identify which frame was not received normally at a certain moment in the video. Thereby, the frame anomaly analysis of the first decoded information is performed to obtain the missing frame position information.

[0119] Once the missing frame is located, the missing frame is then compensated by the adjacent frames before and after. The images of the adjacent frames can provide enough image information to help the cloud phone infer and compensate for the missing frame data. For example, if the 5th frame is lost, the system will obtain the image data of the 4th frame, which is the previous adjacent frame, and the 6th frame, which is the next adjacent frame, and perform further processing based on this data.

[0120] After acquiring adjacent frame data, the system needs to perform inter-frame analysis to compare the image differences between the previous frame and the next frame. The differences can help determine the changes in the video content between these two time points. For example, motion compensation is performed on the data to estimate the movement of the video content, such as the direction and speed of movement of people and objects, in order to predict the location of the missing frame. For example, the similarity between the previous and next frames is compared to infer the possible content of the lost frame and perform image content similarity analysis.

[0121] Based on the above analysis results, the system will generate a compensation frame, which is inferred from adjacent frames through interpolation, motion estimation or image generation methods. The purpose of the compensation frame is to simulate the content of the lost frame as much as possible to make the video transition smoothly. Once the compensation frame is generated, it will be inserted into the position of the missing frame, that is, the missing frame data is replaced with the data of the compensation frame. In this way, the video will not be stuck due to frame loss during playback, and the video continuity can be maintained.

[0122] Eventually, the system will generate a complete video stream containing compensated frame data. These compensated frames will serve as the first compensation information and continue to be used for subsequent video playback or transmission. In the video decoding process, the embodiment of the present application generates a synthetic frame to replace the lost frame by analyzing the frame content before and after the lost frame through methods such as inter-frame interpolation and motion compensation. Even if the network environment is not good and causes frame loss, the video playback can be kept as smooth as possible, reducing the visual freeze caused by data loss, thereby improving the user experience of cloud phone video calls.

[0123] The operation process of audio data compensation and obtaining the second compensation information in the embodiment of the present application is compared with video compensation in that the data type obtained by decoding is different from the data type obtained by compensation, but the operation steps are similar.

[0124] As one implementation manner, the specific steps of post-processing the first compensation information and the second compensation information to obtain the audio and video information for sending to the receiving device are:

[0125] Acquire time information of the first compensation information and the second compensation information, and perform synchronization processing on the first compensation information and the second compensation information according to the time information to obtain synchronization data;

[0126] The synchronous data is encoded and compressed to obtain audio and video information for sending to a receiving device.

[0127] Specifically, the embodiment of the present application performs compensation post-processing on the video and audio information so that the final data can be smoothly sent to the receiving device. Among them, for each frame of video data and each segment of audio data, there will be a timestamp or time mark, and this time information indicates the actual playback time of the data. Obtaining time information helps to synchronize the video and audio streams so that the playback of the two is not out of step, reducing the phenomenon of asynchrony between the picture and the sound.

[0128] The embodiments of the present application obtain the time information of the video and audio data. Since the video and audio streams may be misaligned in time due to network delays, packet loss and other issues, it is very important to synchronize the information accurately. Specifically, time synchronization algorithms such as clock synchronization and timestamp matching are used to adjust the playback order of audio and video data or perform certain delay compensation so that the video and audio information appear in a coordinated state on the receiving device. For example, if the video is faster than the audio, some frames of the video need to be delayed, or the audio playback needs to be accelerated; if the audio is slower, the audio stream needs to be delayed or appropriate processing needs to be performed on the video end to match it.

[0129] After the synchronization process, the synchronized audio and video data needs to be encoded and compressed. The purpose of encoding and compression is to reduce the bandwidth pressure of data transmission and ensure that the data can be correctly decoded and restored at the receiving end. After completing the synchronization process, encoding and compression, the final result is audio and video information that can be sent directly to the receiving end. This information has been compensated, synchronized and encoded, so that it can be stably transmitted through the network.

[0130] Reference Figure 2 , the embodiment of the present application provides a cloud phone, including:

[0131] An audio and video information acquisition module is used to acquire video information and audio information of a running device;

[0132] A network information acquisition module is used to acquire network information, analyze the network information according to a pre-built compensation prediction model, and obtain a prediction result;

[0133] A first decoding module is used for, when the prediction result indicates that an abnormality exists, decoding the video information with the abnormality according to the prediction result to obtain first decoded information, and performing video data compensation on the first decoded information to obtain first compensation information;

[0134] A second decoding module is used for, when the prediction result indicates that an abnormality exists, decoding the audio information with the abnormality according to the prediction result to obtain second decoded information, and performing audio data compensation on the second decoded information to obtain second compensation information;

[0135] The first information sending module is used to post-process the first compensation information and the second compensation information to obtain audio and video information for sending to a receiving device.

[0136] As one implementation mode, it also includes:

[0137] A third decoding module is used to decode the video information to obtain third decoding information when the prediction result is normal, and adjust the running page information according to the third decoding information;

[0138] The running page image is acquired, and the running page image is encoded and compressed to obtain page image information for feeding back to the running device.

[0139] As one implementation mode, it also includes:

[0140] The second information sending module is used to send the video information and audio information of the running device to the receiving device when the prediction result is normal.

[0141] As one implementation mode, it also includes:

[0142] The model building module is used to obtain historical network information, historical video information, historical audio information and historical compensation information, and build a compensation prediction model based on the historical network information, historical video information, historical audio information and historical compensation information.

[0143] As one implementation manner, the first decoding module performs video data compensation on the first decoded information to obtain the first compensation information in the following specific steps:

[0144] Performing frame anomaly analysis on the first decoded information to obtain missing frame position information;

[0145] Acquire adjacent frame images of missing frame position information, analyze the previous adjacent frame image and the next adjacent frame image, and obtain compensation frame data;

[0146] The missing frame position is compensated according to the compensation frame data to obtain first compensation information.

[0147] As one implementation manner, the first information sending module performs post-processing on the first compensation information and the second compensation information to obtain the audio and video information for sending to the receiving device in the following specific steps:

[0148] Acquire time information of the first compensation information and the second compensation information, and perform synchronization processing on the first compensation information and the second compensation information according to the time information to obtain synchronization data;

[0149] The synchronous data is encoded and compressed to obtain audio and video information for sending to a receiving device.

[0150] Reference Figure 3 , a method for processing running device data is provided for an embodiment of the present application, comprising:

[0151] Get real-time collection data;

[0152] The real-time collected data is encoded and compressed to obtain video information and audio information for feedback to the cloud phone;

[0153] Obtain page image information, decode the page image information, and obtain audio and video information acting on the running device.

[0154] Specifically, the method of the embodiment of the present application is applied to the running device and executed when the prediction result of the cloud phone is normal. The real-time collected data is encoded and compressed to obtain video information and audio information for feedback to the cloud phone. After the audio and video are sent to the cloud phone, the running device obtains the page image information fed back by the cloud phone, and decompresses and decodes the page image information, thereby realizing the function of obtaining the cloud phone running screen on the running device.

[0155] As one implementation mode, it also includes:

[0156] Get abnormal instruction information;

[0157] When abnormal instruction information is obtained, the page image information is subjected to screen recognition and extraction to obtain fixed screen information;

[0158] Obtain partial image information of the cloud phone, and construct audio and video information acting on the running device based on real-time collected data, fixed screen information and partial image information.

[0159] Specifically, in the embodiment of the present application, when a network anomaly occurs in the cloud phone, specifically when the cloud phone predicts that an anomaly exists, abnormal command information is obtained from the cloud phone, and the data composition of the presentation screen of the running device is adjusted according to the abnormal command information.

[0160] By performing screen recognition and extraction on page image information, fixed screen information is obtained. The fixed screen information is an application interface in the video call interface with a low screen change frequency.

[0161] Part of the image information is the coded compressed information sent by the receiving device to the cloud phone, specifically video information and audio information. After receiving the partial image information, the running device performs decompression and decoding processing, and places the decompressed related video at a specific position of the fixed picture information.

[0162] As one implementation manner, the real-time collected data includes real-time video data and real-time audio data.

[0163] The embodiment of the present application processes the real-time acquisition data obtained by the local camera and microphone while obtaining fixed screen information and partial image information to obtain real-time video data and real-time audio data, and constructs audio and video information for screen display on the local running device based on the real-time video data, fixed screen information and partial image information.

[0164] The embodiments of the present application can reduce the network burden and resource usage of a cloud phone when a network anomaly occurs on the cloud phone. When an anomaly occurs on the cloud phone, the operating device takes on a small portion of the data processing, so that the images presented by the operating device remain smooth at all times, and the resources occupied by the cloud phone for data processing and transmission are reduced, so that the cloud phone can predict and compensate for audio and video data more accurately and efficiently.

[0165] As one implementation mode, it also includes:

[0166] Get abnormal instruction information;

[0167] When abnormal instruction information is obtained, the page image information is subjected to screen recognition and extraction to obtain fixed screen information;

[0168] Obtain partial image information of the cloud phone, and construct audio and video information acting on the running device based on real-time collected data, fixed screen information and partial image information.

[0169] As one implementation manner, the real-time collected data includes real-time video data and real-time audio data.

[0170] Reference Figure 4 , the embodiment of the present application provides an operating device, including:

[0171] Real-time data collection module, used to obtain real-time collection data;

[0172] The acquisition data processing module is used to encode and compress the real-time acquisition data to obtain video information and audio information for feedback to the cloud phone;

[0173] The page image processing module is used to obtain page image information, decode the page image information, and obtain audio and video information acting on the running device.

[0174] As one implementation mode, it also includes:

[0175] An abnormal data processing module is used to obtain abnormal instruction information;

[0176] When abnormal instruction information is obtained, the page image information is subjected to screen recognition and extraction to obtain fixed screen information;

[0177] Obtain partial image information of the cloud phone, and construct audio and video information acting on the running device based on real-time collected data, fixed screen information and partial image information.

[0178] As one implementation manner, the real-time collected data of the real-time data collection module includes real-time video data and real-time audio data.

[0179] Reference Figure 5 An embodiment of the present application provides a device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method as described above.

[0180] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method when running.

[0181] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and product can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0182] In several embodiments provided in this application, it should be understood that the disclosed methods, systems, devices and program products may be implemented in other ways.

[0183] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0184] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A cloud phone data processing method, characterized in that: include: Get the video and audio information of the running device; Acquire historical network information, historical video information, historical audio information and historical compensation information, and construct a compensation prediction model based on the historical network information, historical video information, historical audio information and historical compensation information; Obtain network information, analyze the network information according to the pre-built compensation prediction model, and obtain prediction results; When the prediction result indicates that an abnormality exists, the video information with the abnormality is decoded according to the prediction result to obtain first decoded information, and video data compensation is performed on the first decoded information to obtain first compensation information; When the prediction result indicates that an abnormality exists, the audio information with the abnormality is decoded according to the prediction result to obtain second decoded information, and audio data compensation is performed on the second decoded information to obtain second compensated information; Acquire time information of the first compensation information and the second compensation information, and perform synchronization processing on the first compensation information and the second compensation information according to the time information to obtain synchronization data; The synchronous data is encoded and compressed to obtain audio and video information for sending to a receiving device.

2. The cloud phone data processing method according to claim 1, characterized in that: Also includes: When the prediction result is normal, the video information is decoded to obtain third decoded information, and the running page information is adjusted according to the third decoded information; The running page image is acquired, and the running page image is encoded and compressed to obtain page image information for feeding back to the running device.

3. The cloud phone data processing method according to claim 2 is characterized in that: Also includes: When the prediction result is normal, the video information and audio information of the running device are sent to the receiving device.

4. The cloud phone data processing method according to claim 1, characterized in that: The specific steps of performing video data compensation on the first decoded information to obtain the first compensation information are: Performing frame anomaly analysis on the first decoded information to obtain missing frame position information; Acquire adjacent frame images of missing frame position information, analyze the previous adjacent frame image and the next adjacent frame image, and obtain compensation frame data; The missing frame position is compensated according to the compensation frame data to obtain first compensation information.

5. A cloud phone, characterized in that: include: An audio and video information acquisition module is used to acquire video information and audio information of a running device; A model building module is used to obtain historical network information, historical video information, historical audio information and historical compensation information, and to build a compensation prediction model based on the historical network information, historical video information, historical audio information and historical compensation information; A network information acquisition module is used to acquire network information, analyze the network information according to a pre-built compensation prediction model, and obtain a prediction result; A first decoding module is used for, when the prediction result shows that an abnormality exists, decoding the video information with the abnormality according to the prediction result to obtain first decoded information, and performing video data compensation on the first decoded information to obtain first compensation information; A second decoding module is used for, when the prediction result indicates that an abnormality exists, decoding the audio information with the abnormality according to the prediction result to obtain second decoded information, and performing audio data compensation on the second decoded information to obtain second compensation information; A first information sending module is used to obtain time information of the first compensation information and the second compensation information, and synchronize the first compensation information and the second compensation information according to the time information to obtain synchronization data; The synchronous data is encoded and compressed to obtain audio and video information for sending to a receiving device.

6. The cloud phone according to claim 5, characterized in that: Also includes: A third decoding module is used to decode the video information to obtain third decoding information when the prediction result is normal, and adjust the running page information according to the third decoding information; The running page image is acquired, and the running page image is encoded and compressed to obtain page image information for feeding back to the running device.

7. The cloud phone according to claim 6, characterized in that: Also includes: The second information sending module is used to send the video information and audio information of the running device to the receiving device when the prediction result is normal.

8. The cloud phone according to claim 5, characterized in that: The first decoding module performs video data compensation on the first decoding information to obtain the first compensation information in the following specific steps: Performing frame anomaly analysis on the first decoded information to obtain missing frame position information; Acquire adjacent frame images of missing frame position information, analyze the previous adjacent frame image and the next adjacent frame image, and obtain compensation frame data; The missing frame position is compensated according to the compensation frame data to obtain first compensation information.

9. A method for processing data of an operating device that feeds back information to the cloud phone data processing method according to any one of claims 1 to 4, characterized in that: include: Acquire real-time collected data, which includes real-time video data and real-time audio data; The real-time collected data is encoded and compressed to obtain video information and audio information for feedback to the cloud phone; Obtain page image information, decode the page image information, and obtain audio and video information acting on the running device; Get abnormal instruction information; When abnormal instruction information is obtained, the page image information is subjected to screen recognition and extraction to obtain fixed screen information; Obtain partial image information of the cloud phone, and construct audio and video information acting on the running device based on real-time collected data, fixed screen information and partial image information.

10. A device for feeding back information to the cloud phone operation device according to any one of claims 5 to 8, characterized in that: include: A real-time data acquisition module is used to obtain real-time acquisition data. The real-time acquisition data of the real-time data acquisition module includes real-time video data and real-time audio data; The acquisition data processing module is used to encode and compress the real-time acquisition data to obtain video information and audio information for feedback to the cloud phone; The page image processing module is used to obtain page image information, decode the page image information, and obtain audio and video information acting on the running device; An abnormal data processing module is used to obtain abnormal instruction information; When abnormal instruction information is obtained, the page image information is subjected to screen recognition and extraction to obtain fixed screen information; Obtain partial image information of the cloud phone, and construct audio and video information acting on the running device based on real-time collected data, fixed screen information and partial image information.

11. A device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 4 or 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 4 or 9 when executed.

Images