A method and system for controlling multi-screen simultaneous display based on cloud phones

Abstract

This application discloses a multi-screen simultaneous display control method and system based on cloud phones, belonging to the field of Internet technology. The method includes: receiving single-channel media stream data of a cloud phone instance sent by a cloud phone server; wherein the cloud phone server synchronously distributes the single-channel media stream data of the cloud phone instance to at least two terminal devices connected to the cloud phone instance; parsing the single-channel media stream data to obtain video stream data and audio stream data; decoding the video stream data and audio stream data based on a pre-set differential compensation value and a global synchronization identifier to obtain video frame data and audio frame data; and synchronously rendering and displaying the video frame data and audio frame data based on the global synchronization identifier. This technical solution achieves synchronous playback on different display terminals, ensuring the consistency and coordination of content presented by multiple terminal devices, and thereby realizing operation sharing between multiple terminal devices.

Description

Technical Field

[0001] This application belongs to the field of Internet technology, specifically relating to a multi-screen simultaneous display control method and system based on cloud mobile phones. Background Technology

[0002] In today's digital age, multi-screen sharing is a key requirement for improving efficiency and user experience. In the office, team collaboration requires simultaneous viewing of documents, design drafts, etc., to ensure information consistency and avoid decision-making delays; in teaching, synchronized display of courseware and operations helps students keep up with the pace; exhibitions and press conferences use multi-screen synchronized content display to enhance dissemination; and in home entertainment, synchronized movie watching and shared game screens also rely on multi-screen sharing.

[0003] Currently, screen sharing is mainly achieved through remote control software. However, this method is greatly affected by the network environment of the terminal devices. Different terminal devices may be in different network environments, resulting in varying latency in receiving shared content, which greatly affects collaboration and viewing experience. Moreover, this method has obvious interactive limitations. Apart from the main device initiating the sharing, other receiving users can mostly only passively view the screen and cannot operate on the shared content.

[0004] Therefore, how to improve the synchronization of content presented on multiple terminal devices and enable users to operate accordingly is a problem that urgently needs to be solved by those in the field. Summary of the Invention

[0005] This application provides a multi-screen simultaneous display control method and system based on cloud phones, aiming to achieve synchronous playback on different display terminals, effectively eliminate audio and video asynchrony caused by differences in hardware performance and network transmission latency of each terminal, ensure the consistency and coordination of content presented by multiple terminal devices, and thereby realize operation sharing between multiple terminal devices.

[0006] In a first aspect, embodiments of this application provide a multi-screen simultaneous display control method based on a cloud phone, the method comprising:

[0007] The cloud phone server receives single-channel media stream data of a cloud phone instance sent by the cloud phone server; wherein the cloud phone server synchronously distributes the single-channel media stream data of the cloud phone instance to at least two terminal devices connected to the cloud phone instance, and the single-channel media stream data consists of continuous audio and video data blocks, which are identified based on a global synchronization identifier.

[0008] The single-channel media stream data is parsed to obtain video stream data and audio stream data;

[0009] Based on the pre-set differential compensation value and the global synchronization flag, the video stream data and the audio stream data are decoded respectively to obtain video frame data and audio frame data;

[0010] Based on the global synchronization identifier, the video frame data and the audio frame data are rendered and displayed synchronously so that the display content of the current terminal device is synchronized with the display content of other terminal devices connected to the cloud phone instance.

[0011] Optionally, the process of setting the differential compensation value includes:

[0012] Send a multi-screen display request for the cloud phone instance to the cloud phone server; wherein, the multi-screen display request includes the interaction priority of the cloud phone instance;

[0013] The response delay parameter is determined based on the received clock synchronization calibration command, and the response delay parameter is sent to the cloud phone server; wherein, the clock synchronization calibration command is issued by the cloud phone server after receiving the multi-screen simultaneous display request;

[0014] The cloud mobile phone server receives and stores the differentiated compensation value sent by the cloud mobile phone server; wherein the differentiated compensation value is determined by the cloud mobile phone server based on the response latency parameter and the interaction priority.

[0015] Optionally, the process by which the cloud phone server determines the differentiated compensation value based on the response latency parameter and the interaction priority includes:

[0016] Obtain the system base clock of the cloud phone instance;

[0017] The network transmission delay and local processing delay are determined based on the system reference clock and the response delay parameters.

[0018] The total time offset is determined based on the network transmission delay and the local processing delay.

[0019] A first correction coefficient is determined based on the interaction priority, and a differential compensation value is determined based on the first correction coefficient and the total time offset.

[0020] Optionally, the multi-screen display request may also include a preset interaction area for the cloud phone instance and the number of interactive objects within the preset interaction area.

[0021] Accordingly, before determining the differential compensation value based on the first correction coefficient and the total time offset, the method further includes:

[0022] A second correction coefficient is determined based on the preset interactive area and the number of interactive objects;

[0023] Accordingly, determining the differential compensation value based on the first correction coefficient and the total time offset includes:

[0024] The differential compensation value is determined based on the first correction coefficient, the second correction coefficient, and the total time offset.

[0025] Optionally, the step of decoding the video stream data and the audio stream data based on the pre-set differential compensation value and the global synchronization identifier to obtain video frame data and audio frame data includes:

[0026] Based on the pre-set differential compensation value and the global synchronization identifier, the video stream decoding start timestamp is determined;

[0027] Obtain the first specification parameters of the video stream data and the second specification parameters of the audio stream data, and determine the audio and video decoding capability ratio coefficient based on the first specification parameters and the second specification parameters;

[0028] The audio stream decoding start timestamp is determined based on the video stream decoding start timestamp and the audio / video decoding capability ratio coefficient.

[0029] Based on the video stream decoding start timestamp, the video stream data is decoded to obtain video frame data corresponding to the global synchronization identifier; and based on the audio stream decoding start timestamp, the audio stream data is decoded to obtain audio frame data corresponding to the global synchronization identifier.

[0030] Optionally, the first specification parameter of the video stream data includes the frame rate, and the second specification parameter of the audio stream data includes the sampling rate;

[0031] Accordingly, determining the audio / video decoding capability ratio coefficient based on the first specification parameter and the second specification parameter includes:

[0032] The proportion of the original audio and video data is determined based on the frame rate and the sampling rate;

[0033] Obtain the performance parameters of the audio and video decoder, and adjust the ratio of the original audio and video data based on the performance parameters of the audio and video decoder to obtain the audio and video decoding capability ratio coefficient.

[0034] Optionally, determining the audio stream decoding start timestamp based on the video stream decoding start timestamp and the audio / video decoding capability ratio coefficient includes:

[0035] Obtain the media type identifier of the currently running application of the cloud phone instance, and determine the audio and video synchronization accuracy coefficient based on the media type identifier;

[0036] The decoding duration of a single video frame is determined based on the video stream data, and the audio stream decoding start offset is determined based on the decoding duration of the single video frame, the audio and video decoding capability ratio coefficient, and the audio and video synchronization accuracy coefficient.

[0037] The audio stream decoding start timestamp is determined based on the video stream decoding start timestamp and the audio stream decoding start offset.

[0038] Secondly, embodiments of this application provide a multi-screen simultaneous display control system based on a cloud phone, the system comprising:

[0039] A streaming data receiving module is used to receive single-channel media stream data of a cloud phone instance sent by a cloud phone server; wherein, the cloud phone server synchronously distributes the single-channel media stream data of the cloud phone instance to at least two terminal devices connected to the cloud phone instance, and the single-channel media stream data consists of continuous audio and video data blocks, which are identified based on a global synchronization identifier;

[0040] The streaming data parsing module is used to parse the single-channel media stream data to obtain video stream data and audio stream data;

[0041] The streaming data decoding module is used to decode the video streaming data and the audio streaming data based on the pre-set differential compensation value and the global synchronization flag, respectively, to obtain video frame data and audio frame data;

[0042] The frame data display module is used to synchronously render and display the video frame data and the audio frame data based on the global synchronization identifier, so that the display content of the current terminal device is synchronized with the display content of other terminal devices connected to the cloud phone instance.

[0043] Optionally, the system further includes:

[0044] The compensation value setting module is specifically used for:

[0045] Send a multi-screen display request for the cloud phone instance to the cloud phone server; wherein, the multi-screen display request includes the interaction priority of the cloud phone instance;

[0046] The response delay parameter is determined based on the received clock synchronization calibration command, and the response delay parameter is sent to the cloud phone server; wherein, the clock synchronization calibration command is issued by the cloud phone server after receiving the multi-screen simultaneous display request;

[0047] The cloud mobile phone server receives and stores the differentiated compensation value sent by the cloud mobile phone server; wherein the differentiated compensation value is determined by the cloud mobile phone server based on the response latency parameter and the interaction priority.

[0048] Optionally, the compensation value setting module is specifically used for:

[0049] Send a multi-screen display request for the cloud phone instance to the cloud phone server; wherein, the multi-screen display request includes the interaction priority of the cloud phone instance;

[0050] The response delay parameter is determined based on the received clock synchronization calibration command, and the response delay parameter is sent to the cloud phone server; wherein, the clock synchronization calibration command is issued by the cloud phone server after receiving the multi-screen simultaneous display request;

[0051] The cloud phone server receives and stores the differentiated compensation value sent by the cloud phone server; wherein the cloud phone server obtains the system reference clock of the cloud phone instance, determines the network transmission delay and local processing delay based on the system reference clock and the response delay parameter, determines the total time offset based on the network transmission delay and the local processing delay, determines a first correction coefficient based on the interaction priority, determines a second correction coefficient based on the preset interaction area and the number of interactive objects, and determines the differentiated compensation value based on the first correction coefficient, the second correction coefficient, and the total time offset.

[0052] Optionally, the multi-screen display request may also include a preset interaction area for the cloud phone instance and the number of interactive objects within the preset interaction area.

[0053] Accordingly, the compensation value setting module is specifically used for:

[0054] Send a multi-screen display request for the cloud phone instance to the cloud phone server; wherein, the multi-screen display request includes the interaction priority of the cloud phone instance;

[0055] The response delay parameter is determined based on the received clock synchronization calibration command, and the response delay parameter is sent to the cloud phone server; wherein, the clock synchronization calibration command is issued by the cloud phone server after receiving the multi-screen simultaneous display request;

[0056] The cloud phone server receives and stores the differential compensation value sent by the cloud phone server; wherein the cloud phone server obtains the system reference clock of the cloud phone instance, determines the network transmission delay and local processing delay based on the system reference clock and the response delay parameter, determines the total time offset based on the network transmission delay and the local processing delay, determines the first correction coefficient based on the interaction priority, and determines the differential compensation value based on the first correction coefficient and the total time offset.

[0057] Optionally, the streaming data decoding module is specifically used for:

[0058] Based on the pre-set differential compensation value and the global synchronization identifier, the video stream decoding start timestamp is determined;

[0059] Obtain the first specification parameters of the video stream data and the second specification parameters of the audio stream data, and determine the audio and video decoding capability ratio coefficient based on the first specification parameters and the second specification parameters;

[0060] The audio stream decoding start timestamp is determined based on the video stream decoding start timestamp and the audio / video decoding capability ratio coefficient.

[0061] Based on the video stream decoding start timestamp, the video stream data is decoded to obtain video frame data corresponding to the global synchronization identifier; and based on the audio stream decoding start timestamp, the audio stream data is decoded to obtain audio frame data corresponding to the global synchronization identifier.

[0062] Optionally, the streaming data decoding module is specifically used for:

[0063] Based on the pre-set differential compensation value and the global synchronization identifier, the video stream decoding start timestamp is determined;

[0064] Obtain a first specification parameter of the video stream data and a second specification parameter of the audio stream data; wherein, the first specification parameter of the video stream data includes the frame rate, and the second specification parameter of the audio stream data includes the sampling rate;

[0065] The proportion of the original audio and video data is determined based on the frame rate and the sampling rate;

[0066] Obtain the performance parameters of the audio and video decoder, and adjust the ratio of the original audio and video data based on the performance parameters of the audio and video decoder to obtain the audio and video decoding capability ratio coefficient;

[0067] The audio stream decoding start timestamp is determined based on the video stream decoding start timestamp and the audio / video decoding capability ratio coefficient.

[0068] Based on the video stream decoding start timestamp, the video stream data is decoded to obtain video frame data corresponding to the global synchronization identifier; and based on the audio stream decoding start timestamp, the audio stream data is decoded to obtain audio frame data corresponding to the global synchronization identifier.

[0069] Optionally, the streaming data decoding module is specifically used for:

[0070] Based on the pre-set differential compensation value and the global synchronization identifier, the video stream decoding start timestamp is determined;

[0071] Obtain the first specification parameters of the video stream data and the second specification parameters of the audio stream data, and determine the audio and video decoding capability ratio coefficient based on the first specification parameters and the second specification parameters;

[0072] Obtain the media type identifier of the currently running application of the cloud phone instance, and determine the audio and video synchronization accuracy coefficient based on the media type identifier;

[0073] The decoding duration of a single video frame is determined based on the video stream data, and the audio stream decoding start offset is determined based on the decoding duration of the single video frame, the audio and video decoding capability ratio coefficient, and the audio and video synchronization accuracy coefficient.

[0074] The audio stream decoding start timestamp is determined based on the video stream decoding start timestamp and the audio stream decoding start offset;

[0075] Based on the video stream decoding start timestamp, the video stream data is decoded to obtain video frame data corresponding to the global synchronization identifier; and based on the audio stream decoding start timestamp, the audio stream data is decoded to obtain audio frame data corresponding to the global synchronization identifier.

[0076] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0077] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0078] In this embodiment, a single-channel media stream data of a cloud phone instance is received from a cloud phone server. The cloud phone server synchronously distributes the single-channel media stream data of the cloud phone instance to at least two terminal devices connected to the cloud phone instance. The single-channel media stream data consists of continuous audio and video data blocks, which are identified based on a global synchronization identifier. The single-channel media stream data is parsed to obtain video stream data and audio stream data. Based on a pre-set differential compensation value and the global synchronization identifier, the video stream data and the audio stream data are decoded to obtain video frame data and audio frame data, respectively. Based on the global synchronization identifier, the video frame data and the audio frame data are synchronously rendered and displayed to synchronize the display content of the current terminal device with the display content of other terminal devices connected to the cloud phone instance. This multi-screen simultaneous display control method based on a cloud phone achieves synchronous playback on different display terminals, effectively eliminating audio and video asynchrony problems caused by differences in hardware performance and network transmission latency among terminals. It ensures the consistency and coordination of content presented by multiple terminal devices and thereby achieves operation sharing among multiple terminal devices. Attached Figure Description

[0079] Figure 1 This is a flowchart illustrating a multi-screen simultaneous display control method based on a cloud phone, as provided in an embodiment of this application.

[0080] Figure 2 This is an example diagram of the processing architecture of a multi-screen simultaneous display control system based on a cloud phone, provided in an embodiment of this application.

[0081] Figure 3 This is a flowchart illustrating the process of setting a differential compensation value according to an embodiment of this application;

[0082] Figure 4 This is a flowchart illustrating another multi-screen simultaneous display control method based on a cloud phone provided in this application embodiment;

[0083] Figure 5 This is a schematic diagram of a multi-screen simultaneous display control system based on a cloud phone, provided in an embodiment of this application.

[0084] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0085] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0086] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0087] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0088] The multi-screen simultaneous display control method and system based on cloud mobile phones provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0089] First, this application applies to scenarios where multiple terminal devices simultaneously access the same cloud phone instance and real-time synchronization of displayed content is required. Based on this usage scenario, it is understood that the executing entity of this application can be the terminal device accessing the cloud phone instance. Specifically, the receiving of single-channel media stream data, the parsing and decoding of video and audio stream data, and the synchronous rendering and display of video and audio frame data can be performed by the terminal device. Each terminal device accessing the cloud phone instance simultaneously renders and displays the corresponding video and audio frame data, achieving synchronization of the displayed content across all terminal devices, enabling users to share operation and control based on the synchronized displayed content.

[0090] Cloud phones can refer to virtual mobile devices that run on cloud servers using cloud computing technology. They have operating systems, application environments, and functions similar to physical mobile phones, and users can remotely access and control them through terminal devices.

[0091] Figure 1 This is a flowchart illustrating a multi-screen simultaneous display control method based on a cloud phone, as provided in an embodiment of this application. Figure 1 As shown, the specific steps include the following:

[0092] S101, receiving single-channel media stream data of a cloud phone instance sent by a cloud phone server; wherein, the cloud phone server synchronously distributes the single-channel media stream data of the cloud phone instance to at least two terminal devices accessing the cloud phone instance, the single-channel media stream data is composed of continuous audio and video data blocks, and the audio and video data blocks are identified based on a global synchronization identifier.

[0093] Among them, a cloud phone server can refer to the collection of core hardware devices and software systems for deploying and running cloud phone instances. It is responsible for providing basic resource support such as computing, storage, and network for cloud phone instances. The cloud phone server has the core capabilities of managing the life cycle of cloud phone instances, processing terminal device interaction commands, and synchronizing terminal device display content. It is the core hub connecting cloud phone instances and terminal devices.

[0094] A cloud phone instance can refer to an independent cloud phone instance created on a cloud phone server. Each instance has an independent system environment, application data, and configuration information, and can be accessed and used by one or more terminal devices simultaneously.

[0095] In this context, "terminal device" refers to a hardware device that possesses network access capabilities, audio and video decoding and rendering capabilities, and human-computer interaction functions, and can access a cloud phone instance through specific software or a browser. Terminal devices can include, but are not limited to, mobile phones, tablets, desktop computers, laptops, smart TVs, set-top boxes, and head-mounted displays. Furthermore, a terminal device accessing a cloud phone instance can refer to a terminal device that has established a stable network connection with the cloud phone server and can interact with the accessed cloud phone instance.

[0096] In this context, a single media stream refers to a media stream uniformly encapsulated by the cloud phone server after real-time acquisition, encoding, and compression of the screen output (video signal) and system audio output (audio signal) of the cloud phone instance. A single media stream consists of continuous blocks of audio and video data.

[0097] Specifically, a continuous audio and video data block can refer to the smallest media data unit generated sequentially in the time dimension and carrying a global synchronization identifier. Each data block corresponds to a very short segment of audio and video content. The global synchronization identifier can be a unique identifier uniformly generated by the cloud phone server and embedded in each audio and video data block, and may include a timestamp.

[0098] In one embodiment, receiving single-channel media stream data of a cloud phone instance from a cloud phone server can be achieved by having the current terminal device connect to the ScreenCopyServer within the cloud phone server to receive the single-channel media stream data. Similarly, the cloud phone server can synchronously distribute the single-channel media stream data of the cloud phone instance to terminal devices connected to the cloud phone instance via the ScreenCopyServer.

[0099] S102, the single-channel media stream data is parsed to obtain video stream data and audio stream data.

[0100] Among them, video stream data can refer to the encoded data stream that carries the screen image information of the cloud mobile phone instance in a single media stream; audio stream data can refer to the encoded data stream that carries the audio information of the cloud mobile phone instance system in a single media stream.

[0101] In one embodiment, the method for parsing a single media stream to obtain video stream data and audio stream data can be achieved by transmitting the single media stream data to a BaseStreamReader. The VideoStreamReader subclass of BaseStreamReader can read the video stream data, and the AudioStreamReader subclass of BaseStreamReader can read the audio stream data.

[0102] S103, based on the pre-set differential compensation value and the global synchronization flag, decode the video stream data and the audio stream data respectively to obtain video frame data and audio frame data.

[0103] The differential compensation value can refer to the time compensation parameter preset for different terminal devices, which is used to correct the decoding time of each terminal device and ensure that the audio and video frames rendered and displayed by each terminal device are time-aligned, that is, the display content of the terminal devices is synchronized.

[0104] In one embodiment, the method of pre-setting the differential compensation value can be achieved by the current terminal device sending a multi-screen display request for the cloud phone instance to the cloud phone server, and receiving and storing the differential compensation value determined by the cloud phone server based on the hardware performance information and network environment information of the current terminal device.

[0105] Among them, video frame data can refer to the original image data generated after decoding video stream data, which is the smallest image unit that can be directly used for rendering and display; audio frame data can refer to the original audio sampling data generated after decoding audio stream data, which is the smallest audio unit that can be directly used for audio playback.

[0106] In one embodiment, the video stream data and audio stream data are decoded based on a pre-set differential compensation value and a global synchronization flag to obtain video frame data and audio frame data, respectively. This can be achieved by determining the decoding start timestamp based on the pre-set differential compensation value and the global synchronization flag, and then decoding the video stream data and audio stream data based on the decoding start timestamp to obtain video frame data and audio frame data corresponding to the global synchronization flag.

[0107] Specifically, in one embodiment, video stream data can be decoded using a VideoDecoder, and audio stream data can be decoded using an AudioDecoder.

[0108] S104, based on the global synchronization identifier, the video frame data and the audio frame data are rendered and displayed synchronously so that the display content of the current terminal device is synchronized with the display content of other terminal devices connected to the cloud phone instance.

[0109] In one embodiment, the method of synchronously rendering and displaying video frame data and audio frame data based on a global synchronization identifier can be achieved by associating video frame data and audio frame data based on a global synchronization identifier, and rendering and displaying video frame data and audio frame data when the local time of the current terminal device is the same as the timestamp corresponding to the global synchronization identifier, so that the display content of the current terminal device is synchronized with the display content of other terminal devices connected to the cloud phone instance.

[0110] Specifically, in one embodiment, video frame data can be rendered and displayed using a SurfaceView; audio frame data can be rendered and displayed using a PcmPlayer.

[0111] Figure 2 This is an example diagram of the processing architecture of a multi-screen simultaneous display control system based on a cloud phone, provided in an embodiment of this application. Figure 2 As shown on the left, BaseStreamReader receives transmission data (i.e., single-channel media stream data) from ScreenCopyServer. The subclass VideoStreamReader of BaseStreamReader reads the video stream data, and the subclass AudioStreamReader of BaseStreamReader reads the audio stream data. The subclass VideoDecoder of BaseDecoder decodes the video stream data output by VideoStreamReader, and the subclass AudioDecoder of BaseDecoder decodes the audio stream data output by AudioStreamReader. SurfaceView renders and displays the video frame data output by VideoDecoder, and PcmPlayer renders and displays the audio stream data output by AudioDecoder. Furthermore, as... Figure 2 As shown on the right, video is the video stream data reading process, audio is the audio stream data reading process, videodecode is the video stream data decoding process, audiodecode is the audio stream data decoding process, video is the video frame data rendering and display process, and audioshow is the audio stream data rendering and display process. That is, each processing process does not interfere with the others.

[0112] In this embodiment, a single-channel media stream data of a cloud phone instance is received from a cloud phone server. The cloud phone server synchronously distributes the single-channel media stream data of the cloud phone instance to at least two terminal devices connected to the cloud phone instance. The single-channel media stream data consists of continuous audio and video data blocks, which are identified based on a global synchronization identifier. The single-channel media stream data is parsed to obtain video stream data and audio stream data. Based on a pre-set differential compensation value and the global synchronization identifier, the video stream data and the audio stream data are decoded to obtain video frame data and audio frame data, respectively. Based on the global synchronization identifier, the video frame data and the audio frame data are synchronously rendered and displayed to synchronize the display content of the current terminal device with the display content of other terminal devices connected to the cloud phone instance. This multi-screen simultaneous display control method based on a cloud phone achieves synchronous playback on different display terminals, effectively eliminating audio and video asynchrony problems caused by differences in hardware performance and network transmission latency among terminals. It ensures the consistency and coordination of content presented by multiple terminal devices and thereby achieves operation sharing among multiple terminal devices.

[0113] Figure 3 This is a flowchart illustrating the process of setting a differential compensation value according to an embodiment of this application. Figure 3 As shown, the specific steps include the following:

[0114] S301, Send a multi-screen display request for the cloud phone instance to the cloud phone server; wherein, the multi-screen display request includes the interaction priority of the cloud phone instance.

[0115] Among them, the multi-screen display request of a cloud phone instance can refer to a request initiated by a terminal device to a cloud phone server to request the screen content of the cloud phone instance to be synchronously distributed to the terminal device so that multiple terminal devices can display the screen content synchronously.

[0116] The interaction priority of a cloud phone instance can refer to a parameter used to define the response order of operation control commands to different terminal devices when multiple terminal devices access the same cloud phone instance at the same time.

[0117] In one embodiment, the method of sending a multi-screen display request for a cloud phone instance to the cloud phone server can be achieved by using the cloud phone server's ServerSocket to listen for the multi-screen display request of the cloud phone instance on the terminal device.

[0118] S302, determine the response delay parameter based on the received clock synchronization calibration command, and send the response delay parameter to the cloud phone server; wherein, the clock synchronization calibration command is issued by the cloud phone server after receiving the multi-screen simultaneous display request.

[0119] Among them, the clock synchronization calibration command can refer to the command issued by the cloud phone server to the terminal device that initiated the multi-screen display request after receiving the multi-screen display request, which is used to calibrate the local time of the terminal device and the global reference time of the cloud phone server.

[0120] In one embodiment, the clock synchronization calibration command can be received by the terminal device through the command channel between the terminal device and the cloud phone server.

[0121] The response latency parameter refers to the parameters determined by the terminal device after receiving the clock synchronization calibration command from the cloud phone server, used to calculate network transmission latency and local processing latency. The response latency parameter can include the server's command sending time, the local time of command reception, and the local time of response transmission. The server's command sending time can refer to the system base clock of the cloud phone instance when the cloud phone server sends the clock synchronization calibration command.

[0122] In one embodiment, the method of determining the response delay parameter based on the clock synchronization calibration command can be as follows: when the terminal device receives the clock synchronization calibration command, it immediately records the local time of command reception and parses the time when the server sends the command in the clock synchronization calibration command. The terminal device then generates a calibration response and records the local time when the response is sent, thereby obtaining the response delay parameter.

[0123] In one embodiment, the response latency parameter can be sent to the cloud phone server by the terminal device through the command channel between the terminal device and the cloud phone server.

[0124] S303, Receive and store the differentiated compensation value sent by the cloud mobile phone server; wherein the differentiated compensation value is determined by the cloud mobile phone server based on the response delay parameter and the interaction priority.

[0125] In one embodiment, the method of receiving and storing the differential compensation value sent by the cloud mobile phone server can be that the terminal device receives the differential compensation value sent by the cloud mobile phone server through the instruction channel between the terminal device and the cloud mobile phone server, and stores the differential compensation value locally.

[0126] In one embodiment, the method by which the cloud phone server determines the differential compensation value based on response latency parameters and interaction priority can be as follows: obtain the system reference clock of the cloud phone instance, determine the network transmission latency and local processing latency based on the system reference clock and response latency parameters, determine the total time offset based on the network transmission latency and local processing latency, determine the first correction coefficient based on the interaction priority, and determine the differential compensation value based on the first correction coefficient and the total time offset.

[0127] Optionally, the process by which the cloud phone server determines the differentiated compensation value based on the response latency parameter and the interaction priority includes:

[0128] Obtain the system base clock of the cloud phone instance;

[0129] The network transmission delay and local processing delay are determined based on the system reference clock and the response delay parameters.

[0130] The total time offset is determined based on the network transmission delay and the local processing delay.

[0131] A first correction coefficient is determined based on the interaction priority, and a differential compensation value is determined based on the first correction coefficient and the total time offset.

[0132] The system reference clock of a cloud phone instance can refer to the global reference time that the cloud phone instance relies on during runtime and has undergone high-precision time synchronization.

[0133] In one embodiment, the system base clock of a cloud phone instance can be obtained by having the cloud phone server read the system base clock of the cloud phone instance when it receives the response latency parameter.

[0134] Among them, network transmission latency can refer to the time it takes for data to travel from the sending end to the receiving end when data is transmitted between the cloud mobile phone server and the terminal device.

[0135] Local processing latency can refer to the time consumed by the terminal device to parse, calculate, and process data or instructions after receiving them.

[0136] In one embodiment, the method for determining network transmission delay and local processing delay based on system reference clock and response delay parameters can be as follows: calculate the difference between the local time of response sending and the local time of instruction receiving as the local processing delay; calculate the difference between the system reference clock and the time the server sends the instruction; subtract twice the local processing delay from this difference; and finally divide the result of the subtraction by 2 to obtain the network transmission delay.

[0137] The total time offset can refer to the time difference between the time when the terminal device displays the actual usable audio and video data and the time when the cloud mobile phone server sends a single media stream data.

[0138] In one embodiment, the total time offset can be determined by summing the network transmission delay and the local processing delay.

[0139] The first correction factor can refer to the weighting factor of the total time offset that is dynamically adjusted based on the interaction priority of the terminal device with the cloud phone instance.

[0140] In one embodiment, the method of determining the first correction coefficient based on the interaction priority can be to pre-set corresponding first correction coefficients for different interaction priorities. For example, the first correction coefficient corresponding to high priority is 1.0, the first correction coefficient corresponding to medium priority is 0.85, and the first correction coefficient corresponding to low priority is 0.65.

[0141] In one embodiment, the differential compensation value can be determined by multiplying the first correction coefficient by the total time offset to obtain the differential compensation value.

[0142] Optionally, the multi-screen display request may also include a preset interaction area for the cloud phone instance and the number of interactive objects within the preset interaction area.

[0143] Accordingly, before determining the differential compensation value based on the first correction coefficient and the total time offset, the method further includes:

[0144] A second correction coefficient is determined based on the preset interactive area and the number of interactive objects;

[0145] Accordingly, determining the differential compensation value based on the first correction coefficient and the total time offset includes:

[0146] The differential compensation value is determined based on the first correction coefficient, the second correction coefficient, and the total time offset.

[0147] The preset interaction area for a cloud phone instance can refer to the specific area on the cloud phone instance screen that needs to be interacted with when the terminal device initiates a multi-screen display request.

[0148] The number of interactive objects within the preset interactive area can refer to the total number of UI elements (such as buttons, input boxes, and controls) that have user interaction capabilities within the preset interactive area.

[0149] The second correction coefficient can refer to the weighting coefficient of the total time offset that is dynamically adjusted based on the importance of the preset interaction area and the density of interactive objects within the preset interaction area.

[0150] In one embodiment, the method of determining the second correction coefficient based on the preset interaction area and the number of interactive objects can be achieved by calculating the number of pixels in the preset interaction area, dividing the number of pixels by the number of interactive objects to obtain the estimated interaction density, and pre-setting a corresponding second correction coefficient for different estimated interaction densities. For example, if the estimated interaction density is greater than 5000, the corresponding second correction coefficient is 1.2; if the estimated interaction density is less than 5000 but greater than 1000, the corresponding second correction coefficient is 1.0; and if the estimated interaction density is less than 1000, the corresponding second correction coefficient is 0.8.

[0151] In one embodiment, the differential compensation value can be determined by multiplying the first correction coefficient, the second correction coefficient, and the total time offset.

[0152] The advantage of this scheme is that by determining the second correction coefficient based on the preset interaction area and the number of interactive objects, and by determining the differentiated compensation value based on the first correction coefficient, the second correction coefficient, and the total time offset, the compensation intensity can be increased for the core area with dense interaction, and the compensation intensity can be appropriately reduced for the non-core area with sparse interaction, thus laying the foundation for the orderly execution of different user operation controls.

[0153] The advantage of this scheme is that by determining the first correction coefficient based on the interaction priority, and then determining the differentiated compensation value based on the first correction coefficient and the total time offset, high-priority terminal devices can receive more adequate compensation, ensuring optimal operation response speed and screen synchronization accuracy, while low-priority terminal devices use lightweight compensation, thereby ensuring the interactive experience of key users.

[0154] The advantage of this solution is that by sending a multi-screen display request for the cloud phone instance to the cloud phone server, the cloud phone server can determine a differentiated compensation value based on the response latency parameter and interaction priority. This enables synchronized display of multiple terminal devices in a globally collaborative manner, eliminating synchronization deviations caused by differences in terminal hardware and network environment, ensuring that the display content of all terminal devices remains highly consistent, and improving the smoothness and reliability of multi-screen collaboration.

[0155] Figure 4 This is a flowchart illustrating another multi-screen simultaneous display control method based on a cloud phone, provided in an embodiment of this application. Figure 4 As shown, the specific steps include the following:

[0156] S401, Receive single-channel media stream data of a cloud phone instance sent by a cloud phone server; wherein, the cloud phone server synchronously distributes the single-channel media stream data of the cloud phone instance to at least two terminal devices accessing the cloud phone instance, the single-channel media stream data consists of continuous audio and video data blocks, and the audio and video data blocks are identified based on a global synchronization identifier.

[0157] S402, the single-channel media stream data is parsed to obtain video stream data and audio stream data.

[0158] S403, based on the pre-set differential compensation value and the global synchronization identifier, determine the video stream decoding start timestamp.

[0159] The video stream decoding start timestamp can refer to the reference time point at which the terminal device decodes each video data block in the video stream data.

[0160] In one embodiment, the method for determining the video stream decoding start timestamp based on a pre-set differential compensation value and a global synchronization identifier can be to subtract the pre-set differential compensation value from the timestamp corresponding to the global synchronization identifier to obtain the video stream decoding start timestamp.

[0161] S404, obtain the first specification parameter of the video stream data and the second specification parameter of the audio stream data, and determine the audio and video decoding capability ratio coefficient based on the first specification parameter and the second specification parameter.

[0162] The first specification parameter of the video stream data can be a parameter used to describe the amount of video stream data, and may include the frame rate.

[0163] The second specification parameter of the audio stream data can be a parameter used to describe the amount of audio stream data, and may include the sampling rate.

[0164] In one embodiment, the first specification parameters of the video stream data and the second specification parameters of the audio stream data can be obtained by having the first specification parameters of the video stream data and the second specification parameters of the audio stream data determined by the cloud phone server and stored in a global synchronization identifier, so that the terminal device can obtain the first specification parameters of the video stream data and the second specification parameters of the audio stream data.

[0165] The audio and video decoding capability ratio coefficient can refer to the ratio of the decoding time of the audio data in a single audio and video data block of a single media stream to the decoding time of its video data.

[0166] In one embodiment, the method of determining the audio and video decoding capability ratio coefficient based on the first specification parameter and the second specification parameter can be achieved by determining the ratio of the original audio and video data according to the frame rate and the sampling rate, obtaining the audio and video decoder performance parameters, and correcting the ratio of the original audio and video data based on the audio and video decoder performance parameters to obtain the audio and video decoding capability ratio coefficient.

[0167] Optionally, the first specification parameter of the video stream data includes the frame rate, and the second specification parameter of the audio stream data includes the sampling rate;

[0168] Accordingly, determining the audio / video decoding capability ratio coefficient based on the first specification parameter and the second specification parameter includes:

[0169] The proportion of the original audio and video data is determined based on the frame rate and the sampling rate;

[0170] Obtain the performance parameters of the audio and video decoder, and adjust the ratio of the original audio and video data based on the performance parameters of the audio and video decoder to obtain the audio and video decoding capability ratio coefficient.

[0171] Frame rate can refer to the number of video frames contained per unit of time.

[0172] The sampling rate can refer to the number of audio samples contained per unit of time.

[0173] The ratio of raw audio and video data can refer to the ratio of raw audio data volume to raw video data volume per unit time.

[0174] In one embodiment, the method for determining the proportion of the original audio and video data based on the frame rate and the sampling rate can be by dividing the sampling rate by the frame rate to obtain the proportion of the original audio and video data.

[0175] Among them, the audio and video decoder performance parameters can be parameters that reflect the decoding efficiency of the video decoder and the audio decoder, and can include video decoding speed and audio decoding speed.

[0176] In one embodiment, the audio and video decoder performance parameters can be obtained by pre-storing the audio and video decoder performance parameters within the terminal device, thus allowing the audio and video decoder performance parameters to be read directly.

[0177] In one embodiment, the method of correcting the ratio of the original audio and video data based on the performance parameters of the audio and video decoder to obtain the audio and video decoding capability ratio coefficient can be achieved by dividing the video decoding speed by the audio decoding speed and multiplying the result of the division by the ratio of the original audio and video data.

[0178] The advantage of this approach is that by determining the ratio of the original audio and video data based on the frame rate and sampling rate, and then correcting the ratio based on the performance parameters of the audio and video decoder, an audio and video decoding capability ratio coefficient is obtained. This allows the approach to adapt to the hardware decoding characteristics of different terminal devices, ensuring consistent audio and video decoding progress and providing stable support for subsequent synchronous rendering.

[0179] S405, determine the audio stream decoding start timestamp based on the video stream decoding start timestamp and the audio / video decoding capability ratio coefficient.

[0180] The audio stream decoding start timestamp can refer to the reference time point at which the terminal device decodes each audio data block in the audio stream data.

[0181] In one embodiment, the method of determining the audio stream decoding start timestamp based on the video stream decoding start timestamp and the audio-video decoding capability ratio coefficient can be as follows: determine the single-frame video decoding duration based on the video stream data, determine the audio stream decoding start offset based on the single-frame video decoding duration and the audio-video decoding capability ratio coefficient, and determine the audio stream decoding start timestamp based on the video stream decoding start timestamp and the audio stream decoding start offset.

[0182] Optionally, determining the audio stream decoding start timestamp based on the video stream decoding start timestamp and the audio / video decoding capability ratio coefficient includes:

[0183] Obtain the media type identifier of the currently running application of the cloud phone instance, and determine the audio and video synchronization accuracy coefficient based on the media type identifier;

[0184] The decoding duration of a single video frame is determined based on the video stream data, and the audio stream decoding start offset is determined based on the decoding duration of the single video frame, the audio and video decoding capability ratio coefficient, and the audio and video synchronization accuracy coefficient.

[0185] The audio stream decoding start timestamp is determined based on the video stream decoding start timestamp and the audio stream decoding start offset.

[0186] The media type identifier of the application currently running on the cloud phone instance can be a tag or parameter used to distinguish the media category to which the application running on the cloud phone instance belongs, and can include games, meetings, video playback, and audio playback, etc.

[0187] In one embodiment, the media type identifier of the currently running application of the cloud phone instance can be obtained by having the cloud phone server obtain the media type identifier of the currently running application of the cloud phone instance and store it in a global synchronization identifier, so that the terminal device can obtain the media type identifier of the currently running application of the cloud phone instance.

[0188] Among them, the audio and video synchronization accuracy coefficient can be a synchronization strictness parameter that is dynamically adjusted based on the application media type, used to quantify the time alignment requirements of audio and video decoding in different scenarios.

[0189] In one embodiment, the method of determining the audio-video synchronization accuracy coefficient based on the media type identifier can be achieved by pre-setting corresponding audio-video synchronization accuracy coefficients for different media type identifiers. For example, the audio-video synchronization accuracy coefficient for games is 1.0, for meetings it is 0.5, for video playback it is 0.9, and for audio playback it is 0.7.

[0190] Among them, the single-frame video decoding time can refer to the average time required for the terminal device to decode a single video frame data.

[0191] In one embodiment, the method for determining the decoding duration of a single video frame based on video stream data can be achieved by having the terminal device first decode a preset number of video frame data and record the total decoding duration, and then dividing the total decoding duration by the preset number to obtain the decoding duration of a single video frame.

[0192] The audio stream decoding start offset can refer to the time difference between the video stream decoding start timestamp and the audio stream decoding start timestamp.

[0193] In one embodiment, the method for determining the audio stream decoding start offset based on the single-frame video decoding duration, the audio-video decoding capability ratio coefficient, and the audio-video synchronization accuracy coefficient can be achieved by multiplying the single-frame video decoding duration by the audio-video decoding capability ratio coefficient, subtracting the multiplication result from the single-frame video decoding duration, and finally dividing the subtraction result by the audio-video synchronization accuracy coefficient to obtain the audio stream decoding start offset.

[0194] In one embodiment, the method for determining the audio stream decoding start timestamp based on the video stream decoding start timestamp and the audio stream decoding start offset can be to add the audio stream decoding start offset to the video stream decoding start timestamp to obtain the audio stream decoding start timestamp.

[0195] The advantage of this solution is that it determines the audio and video synchronization accuracy coefficient based on the media type identifier of the currently running application on the cloud phone instance, and determines the audio stream decoding start offset based on the single-frame video decoding duration, the audio and video decoding capability ratio coefficient, and the audio and video synchronization accuracy coefficient. This allows for dynamic adjustment of the audio and video synchronization strictness for different media types, ensuring high-precision synchronization in critical scenarios while appropriately relaxing the requirements for non-critical scenarios to reduce the resource consumption of terminal devices.

[0196] S406, the video stream data is decoded based on the video stream decoding start timestamp to obtain video frame data corresponding to the global synchronization identifier, and the audio stream data is decoded based on the audio stream decoding start timestamp to obtain audio frame data corresponding to the global synchronization identifier.

[0197] In one embodiment, the method of decoding video stream data based on the video stream decoding start timestamp to obtain video frame data corresponding to the global synchronization identifier can be achieved by decoding the video data block corresponding to the global synchronization identifier in the video stream data when the local time of the current terminal device is the same as the video stream decoding start timestamp corresponding to the global synchronization identifier, thereby obtaining the video frame data corresponding to the global synchronization identifier.

[0198] In one embodiment, the method of decoding audio stream data based on the audio stream decoding start timestamp to obtain audio frame data corresponding to the global synchronization identifier can be achieved by decoding the audio data block corresponding to the global synchronization identifier in the audio stream data when the local time of the current terminal device is the same as the audio stream decoding start timestamp corresponding to the global synchronization identifier, thereby obtaining the audio frame data corresponding to the global synchronization identifier.

[0199] S407, Based on the global synchronization identifier, the video frame data and the audio frame data are rendered and displayed synchronously so that the display content of the current terminal device is synchronized with the display content of other terminal devices connected to the cloud phone instance.

[0200] The advantage of this scheme is that by determining the video stream decoding start timestamp based on the pre-set differential compensation value and the global synchronization identifier, determining the audio and video decoding capability ratio coefficient based on the specification parameters of the video stream data and the audio and video decoding capability ratio coefficient, and determining the audio stream decoding start timestamp based on the video stream decoding start timestamp and the audio and video decoding capability ratio coefficient, it can ensure that the audio stream data and the video stream data keep matched in terms of decoding progress, and finally achieve synchronized audio and video rendering at the timestamp corresponding to the global synchronization identifier.

[0201] Figure 5 This is a schematic diagram of a multi-screen simultaneous display control system based on a cloud phone, provided in an embodiment of this application. Figure 5 As shown, the system includes:

[0202] The streaming data receiving module 510 is used to receive single-channel media stream data of a cloud phone instance sent by a cloud phone server; wherein, the cloud phone server synchronously distributes the single-channel media stream data of the cloud phone instance to at least two terminal devices connected to the cloud phone instance, and the single-channel media stream data consists of continuous audio and video data blocks, and the audio and video data blocks are identified based on a global synchronization identifier.

[0203] The streaming data parsing module 520 is used to parse the single-channel media stream data to obtain video stream data and audio stream data;

[0204] The streaming data decoding module 530 is used to decode the video streaming data and the audio streaming data based on the preset differential compensation value and the global synchronization flag, respectively, to obtain video frame data and audio frame data;

[0205] The frame data display module 540 is used to synchronously render and display the video frame data and the audio frame data based on the global synchronization identifier, so that the display content of the current terminal device is synchronized with the display content of other terminal devices connected to the cloud phone instance.

[0206] Optionally, the system further includes:

[0207] The compensation value setting module 550 is specifically used for:

[0208] Send a multi-screen display request for the cloud phone instance to the cloud phone server; wherein, the multi-screen display request includes the interaction priority of the cloud phone instance;

[0209] The response delay parameter is determined based on the received clock synchronization calibration command, and the response delay parameter is sent to the cloud phone server; wherein, the clock synchronization calibration command is issued by the cloud phone server after receiving the multi-screen simultaneous display request;

[0210] The cloud mobile phone server receives and stores the differentiated compensation value sent by the cloud mobile phone server; wherein the differentiated compensation value is determined by the cloud mobile phone server based on the response latency parameter and the interaction priority.

[0211] Optionally, the compensation value setting module 550 is specifically used for:

[0212] Send a multi-screen display request for the cloud phone instance to the cloud phone server; wherein, the multi-screen display request includes the interaction priority of the cloud phone instance;

[0213] The response delay parameter is determined based on the received clock synchronization calibration command, and the response delay parameter is sent to the cloud phone server; wherein, the clock synchronization calibration command is issued by the cloud phone server after receiving the multi-screen simultaneous display request;

[0214] The cloud phone server receives and stores the differentiated compensation value sent by the cloud phone server; wherein the cloud phone server obtains the system reference clock of the cloud phone instance, determines the network transmission delay and local processing delay based on the system reference clock and the response delay parameter, determines the total time offset based on the network transmission delay and the local processing delay, determines a first correction coefficient based on the interaction priority, determines a second correction coefficient based on the preset interaction area and the number of interactive objects, and determines the differentiated compensation value based on the first correction coefficient, the second correction coefficient, and the total time offset.

[0215] Optionally, the multi-screen display request may also include a preset interaction area for the cloud phone instance and the number of interactive objects within the preset interaction area.

[0216] Accordingly, the compensation value setting module 550 is specifically used for:

[0217] Send a multi-screen display request for the cloud phone instance to the cloud phone server; wherein, the multi-screen display request includes the interaction priority of the cloud phone instance;

[0218] The response delay parameter is determined based on the received clock synchronization calibration command, and the response delay parameter is sent to the cloud phone server; wherein, the clock synchronization calibration command is issued by the cloud phone server after receiving the multi-screen simultaneous display request;

[0219] The cloud phone server receives and stores the differential compensation value sent by the cloud phone server; wherein the cloud phone server obtains the system reference clock of the cloud phone instance, determines the network transmission delay and local processing delay based on the system reference clock and the response delay parameter, determines the total time offset based on the network transmission delay and the local processing delay, determines the first correction coefficient based on the interaction priority, and determines the differential compensation value based on the first correction coefficient and the total time offset.

[0220] Optionally, the streaming data decoding module 530 is specifically used for:

[0221] Based on the pre-set differential compensation value and the global synchronization identifier, the video stream decoding start timestamp is determined;

[0222] Obtain the first specification parameters of the video stream data and the second specification parameters of the audio stream data, and determine the audio and video decoding capability ratio coefficient based on the first specification parameters and the second specification parameters;

[0223] The audio stream decoding start timestamp is determined based on the video stream decoding start timestamp and the audio / video decoding capability ratio coefficient.

[0224] Based on the video stream decoding start timestamp, the video stream data is decoded to obtain video frame data corresponding to the global synchronization identifier; and based on the audio stream decoding start timestamp, the audio stream data is decoded to obtain audio frame data corresponding to the global synchronization identifier.

[0225] Optionally, the streaming data decoding module 530 is specifically used for:

[0226] Based on the pre-set differential compensation value and the global synchronization identifier, the video stream decoding start timestamp is determined;

[0227] Obtain a first specification parameter of the video stream data and a second specification parameter of the audio stream data; wherein, the first specification parameter of the video stream data includes the frame rate, and the second specification parameter of the audio stream data includes the sampling rate;

[0228] The proportion of the original audio and video data is determined based on the frame rate and the sampling rate;

[0229] Obtain the performance parameters of the audio and video decoder, and adjust the ratio of the original audio and video data based on the performance parameters of the audio and video decoder to obtain the audio and video decoding capability ratio coefficient;

[0230] The audio stream decoding start timestamp is determined based on the video stream decoding start timestamp and the audio / video decoding capability ratio coefficient.

[0231] Based on the video stream decoding start timestamp, the video stream data is decoded to obtain video frame data corresponding to the global synchronization identifier; and based on the audio stream decoding start timestamp, the audio stream data is decoded to obtain audio frame data corresponding to the global synchronization identifier.

[0232] Optionally, the streaming data decoding module 530 is specifically used for:

[0233] Based on the pre-set differential compensation value and the global synchronization identifier, the video stream decoding start timestamp is determined;

[0234] Obtain the first specification parameters of the video stream data and the second specification parameters of the audio stream data, and determine the audio and video decoding capability ratio coefficient based on the first specification parameters and the second specification parameters;

[0235] Obtain the media type identifier of the currently running application of the cloud phone instance, and determine the audio and video synchronization accuracy coefficient based on the media type identifier;

[0236] The decoding duration of a single video frame is determined based on the video stream data, and the audio stream decoding start offset is determined based on the decoding duration of the single video frame, the audio and video decoding capability ratio coefficient, and the audio and video synchronization accuracy coefficient.

[0237] The audio stream decoding start timestamp is determined based on the video stream decoding start timestamp and the audio stream decoding start offset;

[0238] Based on the video stream decoding start timestamp, the video stream data is decoded to obtain video frame data corresponding to the global synchronization identifier; and based on the audio stream decoding start timestamp, the audio stream data is decoded to obtain audio frame data corresponding to the global synchronization identifier.

[0239] In this embodiment, a streaming data receiving module is used to receive single-channel media stream data of a cloud phone instance sent by a cloud phone server; wherein, the cloud phone server synchronously distributes the single-channel media stream data of the cloud phone instance to at least two terminal devices connected to the cloud phone instance, and the single-channel media stream data consists of continuous audio and video data blocks, which are identified based on a global synchronization identifier; a streaming data parsing module is used to parse the single-channel media stream data to obtain video stream data and audio stream data; a streaming data decoding module is used to decode the video stream data and the audio stream data respectively based on a pre-set differential compensation value and the global synchronization identifier to obtain video frame data and audio frame data; a frame data display module is used to synchronously render and display the video frame data and the audio frame data based on the global synchronization identifier, so that the display content of the current terminal device is synchronized with the display content of other terminal devices connected to the cloud phone instance. The aforementioned multi-screen simultaneous display control system based on cloud phones enables synchronized playback on different display terminals, effectively eliminating audio and video asynchrony caused by differences in hardware performance and network transmission latency among terminals. This ensures the consistency and coordination of content presented by multiple terminal devices, thereby enabling operation sharing among multiple terminal devices.

[0240] The multi-screen simultaneous display control system based on cloud phones in this application embodiment can be a system, or a component, integrated circuit, or chip in a terminal. The system can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0241] The multi-screen simultaneous display control system based on cloud phones in this application embodiment can be a system with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0242] The multi-screen simultaneous display control system based on cloud phones provided in this application can realize the various processes implemented in the above embodiments. To avoid repetition, it will not be described again here.

[0243] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 6 As shown, this application embodiment also provides an electronic device 600, including a processor 601, a memory 602, and a program or instructions stored in the memory 602 and executable on the processor 601. When the program or instructions are executed by the processor 601, they implement the various processes of the above-described multi-screen simultaneous display control method embodiment based on cloud mobile phones and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0244] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0245] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described multi-screen simultaneous display control method embodiment based on cloud phones and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0246] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0247] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element. Furthermore, it should be noted that the scope of the methods and systems in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0248] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0249] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0250] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. A multi-screen simultaneous display control method based on cloud phones, applied to terminal devices, characterized in that, The method includes: The cloud phone server receives single-channel media stream data of a cloud phone instance sent by the cloud phone server; wherein the cloud phone server synchronously distributes the single-channel media stream data of the cloud phone instance to at least two terminal devices connected to the cloud phone instance, and the single-channel media stream data consists of continuous audio and video data blocks, which are identified based on a global synchronization identifier. The single-channel media stream data is parsed to obtain video stream data and audio stream data; Based on the pre-set differential compensation value and the global synchronization flag, the video stream data and the audio stream data are decoded respectively to obtain video frame data and audio frame data; The process of setting the differential compensation value includes: sending a multi-screen display request for the cloud phone instance to the cloud phone server; wherein the multi-screen display request includes the interaction priority of the cloud phone instance; determining a response latency parameter based on a received clock synchronization calibration command, and sending the response latency parameter to the cloud phone server; wherein the clock synchronization calibration command is issued by the cloud phone server after receiving the multi-screen display request; receiving and storing the differential compensation value sent by the cloud phone server; wherein the differential compensation value is determined by the cloud phone server based on the response latency parameter and the interaction priority. The step of decoding the video stream data and the audio stream data based on a pre-set differential compensation value and the global synchronization identifier to obtain video frame data and audio frame data includes: determining the video stream decoding start timestamp based on the pre-set differential compensation value and the global synchronization identifier; obtaining a first specification parameter of the video stream data and a second specification parameter of the audio stream data, and determining an audio-video decoding capability ratio coefficient based on the first specification parameter and the second specification parameter; determining the audio stream decoding start timestamp based on the video stream decoding start timestamp and the audio-video decoding capability ratio coefficient; decoding the video stream data based on the video stream decoding start timestamp to obtain video frame data corresponding to the global synchronization identifier; and decoding the audio stream data based on the audio stream decoding start timestamp to obtain audio frame data corresponding to the global synchronization identifier. Based on the global synchronization identifier, the video frame data and the audio frame data are rendered and displayed synchronously so that the display content of the current terminal device is synchronized with the display content of other terminal devices connected to the cloud phone instance.

2. The multi-screen simultaneous display control method based on cloud mobile phone according to claim 1, characterized in that, The process by which the cloud phone server determines the differential compensation value based on the response latency parameter and the interaction priority includes: Obtain the system base clock of the cloud phone instance; The network transmission delay and local processing delay are determined based on the system reference clock and the response delay parameters. The total time offset is determined based on the network transmission delay and the local processing delay. A first correction coefficient is determined based on the interaction priority, and a differential compensation value is determined based on the first correction coefficient and the total time offset.

3. The multi-screen simultaneous display control method based on cloud mobile phone according to claim 2, characterized in that, The multi-screen simultaneous display request also includes a preset interactive area for the cloud phone instance and the number of interactive objects within the preset interactive area. Accordingly, before determining the differential compensation value based on the first correction coefficient and the total time offset, the method further includes: A second correction coefficient is determined based on the preset interactive area and the number of interactive objects; Accordingly, determining the differential compensation value based on the first correction coefficient and the total time offset includes: The differential compensation value is determined based on the first correction coefficient, the second correction coefficient, and the total time offset.

4. The multi-screen simultaneous display control method based on cloud mobile phone according to claim 1, characterized in that, The first specification parameter of the video stream data includes the frame rate, and the second specification parameter of the audio stream data includes the sampling rate. Accordingly, determining the audio / video decoding capability ratio coefficient based on the first specification parameter and the second specification parameter includes: The proportion of the original audio and video data is determined based on the frame rate and the sampling rate; Obtain the performance parameters of the audio and video decoder, and adjust the ratio of the original audio and video data based on the performance parameters of the audio and video decoder to obtain the audio and video decoding capability ratio coefficient.

5. The multi-screen simultaneous display control method based on cloud mobile phone according to claim 1, characterized in that, Determining the audio stream decoding start timestamp based on the video stream decoding start timestamp and the audio / video decoding capability ratio coefficient includes: Obtain the media type identifier of the currently running application of the cloud phone instance, and determine the audio and video synchronization accuracy coefficient based on the media type identifier; The decoding duration of a single video frame is determined based on the video stream data, and the audio stream decoding start offset is determined based on the decoding duration of the single video frame, the audio and video decoding capability ratio coefficient, and the audio and video synchronization accuracy coefficient. The audio stream decoding start timestamp is determined based on the video stream decoding start timestamp and the audio stream decoding start offset.

6. A multi-screen simultaneous display control system based on cloud phones, characterized in that, The system includes: A streaming data receiving module is used to receive single-channel media stream data of a cloud phone instance sent by a cloud phone server; wherein, the cloud phone server synchronously distributes the single-channel media stream data of the cloud phone instance to at least two terminal devices connected to the cloud phone instance, and the single-channel media stream data consists of continuous audio and video data blocks, which are identified based on a global synchronization identifier; The streaming data parsing module is used to parse the single-channel media stream data to obtain video stream data and audio stream data; The streaming data decoding module is used to decode the video streaming data and the audio streaming data based on the pre-set differential compensation value and the global synchronization flag, respectively, to obtain video frame data and audio frame data; The system is further configured to: send a multi-screen simultaneous display request for the cloud phone instance to the cloud phone server; wherein the multi-screen simultaneous display request includes an interaction priority for the cloud phone instance; determine a response latency parameter based on a received clock synchronization calibration instruction, and send the response latency parameter to the cloud phone server; wherein the clock synchronization calibration instruction is issued by the cloud phone server after receiving the multi-screen simultaneous display request; receive and store a differential compensation value sent by the cloud phone server; wherein the differential compensation value is determined by the cloud phone server based on the response latency parameter and the interaction priority; The streaming data decoding module is specifically configured to: determine the video stream decoding start timestamp based on a pre-set differential compensation value and the global synchronization identifier; obtain a first specification parameter of the video stream data and a second specification parameter of the audio stream data, and determine an audio / video decoding capability ratio coefficient based on the first specification parameter and the second specification parameter; determine an audio stream decoding start timestamp based on the video stream decoding start timestamp and the audio / video decoding capability ratio coefficient; decode the video stream data based on the video stream decoding start timestamp to obtain video frame data corresponding to the global synchronization identifier; and decode the audio stream data based on the audio stream decoding start timestamp to obtain audio frame data corresponding to the global synchronization identifier. The frame data display module is used to synchronously render and display the video frame data and the audio frame data based on the global synchronization identifier, so that the display content of the current terminal device is synchronized with the display content of other terminal devices connected to the cloud phone instance.

7. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of the multi-screen simultaneous display control method based on a cloud phone as described in any one of claims 1-5.

8. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of the multi-screen simultaneous display control method based on a cloud phone as described in any one of claims 1-7.

Images