Screen projection sharing method and device

Abstract

A projection screen sharing method and device are used for realizing projection screen sharing. The method comprises the following steps: under the condition that a call connection is established between a first device and a second device, the first device obtains first instruction information, and the first instruction information is used for indicating shared screen projection information; further, the first device receives screen projection information from the third device based on the first instruction information, and extracts an elementary stream ES of the screen projection information; and finally, the first device plays the ES and sends the ES to the second device. Based on the method, the first device can directly extract the ES of the screen projection information and transmit the ES to the opposite terminal, the time of decoding, screen recording and coding is reduced, and the time delay of screen projection sharing end-to-end is greatly reduced. Moreover, the method can improve the picture precision, and improves the picture experience effect. Meanwhile, non-user expected content can be prevented from being transmitted to an opposite end, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of communication technologies, and in particular, to a screen mirroring sharing method and apparatus. Background Art

[0002] During video conferencing or calls, screen sharing / remote collaboration guidance has become increasingly common. With the hardware upgrade of home smart screens / set-top boxes (equipped with cameras), video call applications have been launched, allowing users to make video calls between large and small screens. During the call, if the user sees an interesting video or content on the small-screen mobile phone / personal digital assistant (PDA) and wants to share it with the other party on the call, or share the homework content on the mobile phone / PAD with the parent on the other party of the call for homework guidance, it is considered that screen mirroring sharing can be used to share with the other party. Based on this, how to implement screen mirroring sharing is an urgent problem to be solved. Summary of the Invention

[0003] Embodiments of this application provide a screen mirroring sharing method and apparatus to implement screen mirroring sharing.

[0004] In a first aspect, this application provides a screen mirroring sharing method, which can be applied to a first device, such as a processor, a chip, or a functional module in the first device. Taking the application of this method to the first device as an example, the method may include: when a call connection is established between the first device and the second device, the first device obtains first instruction information for instructing to share screen mirroring information; further, the first device receives the screen mirroring information from the third device based on the first instruction information, and extracts the elementary stream (ES) of the screen mirroring information; finally, the first device plays the ES and sends the ES to the second device.

[0005] Based on the above method, the first device can directly extract the ES of the screen mirroring information and transmit it to the other party, subtracting the time for decoding, recording the screen, and then encoding, greatly reducing the end-to-end delay of screen mirroring sharing. And this method can improve the picture accuracy and the picture experience effect; at the same time, it can avoid transmitting content that is not expected by the user to the other party, improving the user experience.

[0006] In a possible design, first parameter information may also be negotiated, which is used for sharing screen mirroring information during the call between the first device and the second device. The negotiation process of the first parameter information may be initiated by the first device or the second device, and this application does not make a limitation. In this way, based on the first parameter information, the first device and the second device can implement screen mirroring sharing during the call.

[0007] In a possible design, the first parameter information may include the audio channel and / or video channel used by the first device and the second device to establish shared screen mirroring information. In this way, the first device and the second device can transmit screen mirroring information through a channel other than the call connection channel, so that the first device and the second device can simultaneously maintain a call and share screen mirroring.

[0008] In a possible design, second parameter information may also be negotiated, and the second parameter information is used for the third device to screen mirror to the first device. Among them, the negotiation process of the second parameter information may be initiated by the first device or the third device, which is not limited in this application. In this way, the third device can accurately transmit screen mirroring information to the first device.

[0009] In a possible design, the screen mirroring information may include audio information and / or video information; correspondingly, for the first device to extract the elementary stream (ES) of the screen mirroring information, it may include: the first device extracts the ES of the audio information and / or the ES of the video information. In this way, the first device can accurately extract the ES of the screen mirroring information.

[0010] In a possible design, the first device may send the ES to the second device through the following method: the first device may send the ES of the audio information to the second device through the first audio channel, and / or send the ES of the video information to the second device through the first video channel; the first audio channel is the audio channel corresponding to the call connection between the first device and the second device, and the first video channel is the video channel corresponding to the call connection between the first device and the second device; or, the first device may send the ES of the audio information to the second device through the second audio channel, and / or send the ES of the video information to the second device through the second video channel; the second audio channel is the audio channel used by the first device and the second device to establish shared screen mirroring information, and the second video channel is the video channel used by the first device and the second device to establish shared screen mirroring information. In this way, the first device can flexibly select a suitable method to send the ES to the second device according to the actual situation.

[0011] In a possible design, the first device extracts the presentation time stamp (PTS) of the original packet corresponding to the ES of the screen mirroring information; the first device records the PTS and sends the PTS to the second device. In this way, the screen mirroring information transmitted between the first device and the second device can be kept synchronized.

[0012] In a possible design, when the first device and the second device establish a call connection, the first device and the second device interact to determine whether they have a first capability. The first capability is to support the channel corresponding to the call connection and channels other than the channel corresponding to the call connection. This way, subsequently, based on the capabilities of the first device and the second device, it can be determined whether it is necessary to negotiate channels other than the channel corresponding to the call connection to transmit screen mirroring information.

[0013] In a possible design, the first device and the second device interacting to determine whether they have a first capability may include: the first device sending a first identifier to the second device, where the first identifier is used to indicate whether the first device has the first capability; the first device receiving a second identifier from the second device, where the second identifier is used to indicate whether the second device has the first capability. This way, it can be accurately determined whether the first device and the second device have the first capability.

[0014] In a possible design, the first identifier is an indication of the number of supported channels, or the first identifier is a preset code corresponding to the first device; the second identifier is an indication of the number of supported channels, or the second identifier is a preset code corresponding to the second device. This way, it can flexibly indicate whether the first device or the second device has the first capability.

[0015] In a possible design, before the first device receives the screen mirroring information from the third device, the first device determines whether a first parameter information and a second parameter information are consistent; the first parameter information is used for sharing screen mirroring information during the call between the first device and the second device; the second parameter information is used for the third device to perform screen mirroring to the first device; if so, the first device sends a screen mirroring instruction to the third device; otherwise, the first parameter information and the second parameter information are renegotiated, and after determining that the renegotiated first parameter information and the second information are consistent, the first device sends a screen mirroring instruction to the third device. This way, the third device can accurately perform screen mirroring to the first device to accurately achieve screen mirroring sharing between the first device and the second device.

[0016] In a possible design, when the first device determines that the first parameter and the second parameter information are inconsistent, the first device records the parameter information supported by the third device. This way, subsequently, the first device can renegotiate the first parameter information, etc., based on the parameter information of the third device directly.

[0017] In a possible design, when negotiating the first parameter information, the first parameter information can be negotiated according to the second parameter information, where the second parameter information is used for the third device to perform screen mirroring to the first device. This way, the consistency between the negotiated first parameter information and the second parameter information can be improved.

[0018] In a second aspect, the present application provides a screen mirroring sharing method, which can be applied to a second device, such as a processor, a chip, or a functional module in the second device. Taking the application of this method to the second device as an example, the method may include: when a call connection is established between a first device and the second device, the second device receives an elementary stream (ES) from the first device, and the ES is extracted based on the screen mirroring information of a third device; the second device plays the ES.

[0019] Based on the above method, the first device can directly extract the ES of the screen mirroring information and transmit it to the peer end, subtracting the time for decoding, recording the screen, and then encoding, greatly reducing the end-to-end latency of screen mirroring sharing. And this method can improve the picture accuracy and the picture experience effect; at the same time, it can avoid the transmission of content that is not expected by the user to the peer end, improving the user experience.

[0020] In a possible design, a first parameter information is negotiated, and the first parameter information is used for sharing screen mirroring information during the call between the first device and the second device. Among them, the negotiation process of the first parameter information can be initiated by the first device or the second device, and the present application does not make a limitation. In this way, based on the first parameter information, the first device and the second device can achieve screen mirroring sharing during the call.

[0021] In a possible design, the first parameter information may include the audio channel and / or video channel used by the first device and the second device to establish the shared screen mirroring information. In this way, the first device and the second device can transmit the screen mirroring information through a channel other than the call connection channel, so that the first device and the second device can simultaneously maintain the call and screen mirroring sharing.

[0022] In a possible design, the screen mirroring information may include audio information and / or video information; the ES includes the ES of the audio information and / or the ES of the video information.

[0023] In a possible design, the second device may receive the ES from the first device through the following methods: The second device receives the ES of the audio information from the first device through a first audio channel, and / or receives the ES of the video information from the first device through a first video channel; the first audio channel is the audio channel corresponding to the call connection between the first device and the second device, and the first video channel is the video channel corresponding to the call connection between the first device and the second device; or, the second device receives the ES of the audio information from the first device through a second audio channel, and / or receives the ES of the video information from the first device through a second video channel; the second audio channel is the audio channel used for sharing screen projection information established between the first device and the second device, and the second video channel is the video channel used for sharing screen projection information established between the first device and the second device. In this way, the second device can flexibly select a suitable method to receive the ES from the first device according to the actual situation.

[0024] In a possible design, the second device receives the original packet time PTS corresponding to the ES from the first device. This can keep the screen projection information transmitted between the first device and the second device synchronized.

[0025] In a possible design, when the first device and the second device establish a call connection, the second device and the first device interact to determine whether they have a first capability, where the first capability is to support the channel corresponding to the call connection and channels other than the channel corresponding to the call connection. In this way, it can be determined later whether it is necessary to negotiate channels other than the channel corresponding to the call connection to transmit the screen projection information based on the capabilities of the first device and the second device.

[0026] In a possible design, the interaction between the second device and the first device to determine whether they have the first capability may include: The second device receives a first identifier from the first device, where the first identifier is used to indicate whether the first device has the first capability; the second device sends a second identifier to the first device, where the second identifier is used to indicate whether the second device has the first capability. In this way, it can be accurately determined whether the first device and the second device have the first capability.

[0027] In a possible design, the first identifier is an indication of the number of supported channels, or the first identifier is a preset code corresponding to the first device; the second identifier is an indication of the number of supported channels, or the second identifier is a preset code corresponding to the second device. In this way, it can be flexibly indicated whether the first device or the second device has the first capability.

[0028] In a possible design, negotiating the first parameter information may include: the second device receiving second parameter information from the first device, where the second parameter information is used for the third device to project a screen to the first device; and negotiating the first parameter information according to the second parameter information. This can improve the consistency between the negotiated first parameter information and the second parameter information.

[0029] In a third aspect, the present application further provides a communication device, which can be applied to the first device, and this communication device has the function of implementing the method in the above first aspect or each possible design example of the first aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0030] In a possible design, the structure of the communication device may include a communication unit and a processing unit, and these units can execute the functions of the method in the above first aspect or each possible design example of the first aspect, which will not be elaborated here.

[0031] In a possible design, the structure of the communication device includes a processor, and optionally further includes a memory and / or a communication interface. The communication interface is used to send and receive data, messages, or information, etc., and to communicate and interact with other devices in the system. The processor is configured to support the communication device to execute the corresponding functions in the above first aspect or each possible design example of the first aspect. The memory is coupled to the processor and stores the necessary program instructions and data of the communication device.

[0032] In a fourth aspect, the present application further provides a communication device, which can be applied to the second device, and this communication device has the function of implementing the method in the above second aspect or each possible design example of the second aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0033] In a possible design, the structure of the communication device may include a communication unit and a processing unit, and these units can execute the functions of the method in the above second aspect or each possible design example of the second aspect, which will not be elaborated here.

[0034] In a possible design, the structure of the communication device includes a processor, and optionally further includes a memory and / or a communication interface. The communication interface is used to transmit and receive data, messages, information, etc., and to communicate and interact with other devices in the system. The processor is configured to support the communication device to execute the corresponding functions in the above-mentioned second aspect or each possible design example of the second aspect. The memory is coupled to the processor and stores the necessary program instructions and data of the communication device.

[0035] In a fifth aspect, an embodiment of the present application provides a system, which may include a first device, a second device, a third device, etc. Among them, the first device is used to implement the method in the above-mentioned first aspect or each possible design example of the first aspect. The second device is used to implement the method in the above-mentioned second aspect or each possible design example of the second aspect.

[0036] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores program instructions. When the program instructions run on a computer, the computer is enabled to execute the methods described in the first aspect and any possible design thereof, or the second aspect and any possible design thereof in the embodiments of the present application. Exemplarily, the computer-readable storage medium may be any available medium that can be accessed by a computer. Taking this as an example but not limited to: the computer-readable medium may include a non-transitory computer-readable medium, a random-access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a CD-ROM or other optical disc storage, a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer.

[0037] In a seventh aspect, an embodiment of the present application provides a computer program product, including instructions. When the instructions run on a computer, the methods described in the above-mentioned first aspect or any possible design of the first aspect, or the above-mentioned second aspect or any possible design of the second aspect are executed.

[0038] In an eighth aspect, the present application further provides a chip, including a processor. The processor is coupled to a memory and is used to read and execute the program instructions stored in the memory, so that the chip implements the methods described in the above-mentioned first aspect or any possible design of the first aspect, or the above-mentioned second aspect or any possible design of the second aspect.

[0039] For the technical effects that can be achieved by each of the above aspects from the third aspect to the eighth aspect, please refer to the technical effects that can be achieved by the above-mentioned first aspect or various possible solutions in the first aspect, or the second aspect or various possible solutions in the second aspect. They will not be repeated here. Description of the Drawings

[0040] Figure 1 It is a schematic diagram of the architecture of a system provided by this application;

[0041] Figure 2 It is a flowchart of a screen mirroring sharing method provided by this application;

[0042] Figure 3 It is a schematic diagram of a screen mirroring sharing scenario of a set-top box supporting multiple channels provided by this application;

[0043] Figure 4 It is a schematic diagram of the effect of screen mirroring sharing provided by this application;

[0044] Figure 5 It is a flowchart of an example of a screen mirroring sharing method provided by this application;

[0045] Figure 6 It is a schematic diagram of maintaining synchronization in a screen mirroring sharing method provided by this application;

[0046] Figure 7 It is a schematic diagram of a screen mirroring sharing scenario of a set-top box not supporting multiple channels provided by this application;

[0047] Figure 8 It is a flowchart of an example of another screen mirroring sharing method provided by this application;

[0048] Figure 9 It is a schematic diagram of reminding the user during the screen mirroring process when the screen mirroring parameters and the shared screen mirroring parameters are inconsistent in a scenario supporting multiple channels provided by this application;

[0049] Figure 10 It is a schematic diagram of reminding the user during the screen mirroring process when the screen mirroring parameters and the shared screen mirroring parameters are inconsistent in a scenario supporting a single channel provided by this application;

[0050] Figure 11 It is a flowchart of an example of a re-negotiation process when the screen mirroring parameters and the shared screen mirroring parameters are inconsistent provided by this application;

[0051] Figure 12 It is a flowchart of an example of using other screen mirroring methods for screen mirroring when the screen mirroring parameters and the shared screen mirroring parameters are inconsistent provided by this application;

[0052] Figure 13A schematic diagram of a scenario where a mobile phone casting screen to a first set-top box can trigger call media re-negotiation provided by this application;

[0053] Figure 14 An example of a process with consistent parameters when a mobile phone casting screen to a first set-top box can trigger call media re-negotiation provided by this application;

[0054] Figure 15 An example of a process with inconsistent parameters when a mobile phone casting screen to a first set-top box can trigger call media re-negotiation provided by this application;

[0055] Figure 16 A schematic structural diagram of a communication device provided by this application;

[0056] Figure 17 A structural diagram of a communication device provided by this application. Detailed implementation manners

[0057] The following will further describe this application in detail with reference to the accompanying drawings.

[0058] The embodiments of this application provide a screen mirroring sharing method and device to achieve screen mirroring sharing. Among them, the method and device described in this application are based on the same technical concept. Since the principles of the method and device for solving problems are similar, the implementation of the device and the method can be referred to each other, and the repeated parts will not be described again.

[0059] In the description of this application, terms such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0060] In the description of this application, "at least one (kind)" means one (kind) or more than one (kind), and more than one (kind) means two (kinds) or more than two (kinds). "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or, a and b and c, where a, b, c can be single or multiple.

[0061] In the description of this application, "and / or", describing the association relationship of associated objects, indicates that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. " / " represents "or", for example, a / b represents a or b.

[0062] To more clearly describe the technical solutions of the embodiments of this application, the screen mirroring sharing method and device provided by the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0063] Figure 1 Exemplarily shown is the architecture of a possible system applicable to the screen mirroring sharing method of the present application. The architecture of this system may include a first device, a second device, and a third device. Among them, the first device and the second device can establish a call connection. When a call connection is established between the first device and the second device, the screen mirroring information projected by the third device to the first device can be shared to the second device. That is to say, the third device is the screen mirroring source, and the first device is the screen mirroring receiver.

[0064] Among them, the first device, the second device, and the third device can be terminal devices. The first device and the second device can be terminal devices with call functions. The first device can be a terminal device with screen mirroring function.

[0065] Exemplarily, a terminal device, which can also be called a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc., is a device that provides voice and / or data connectivity to users. For example, a terminal device may include a handheld device with wireless connection function, a vehicle-mounted device, etc. Currently, a terminal device can be: a mobile phone, a tablet computer, a laptop computer, a palm computer, a PAD, a mobile internet device (MID), a wearable device, a smart home screen, a set-top box, a virtual reality (VR) device, an augmented reality (AR) device, an extended reality (XR) device, a mixed reality (MR) device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, or a wireless terminal in smart home, etc.

[0066] The terminal device may also be a device-to-device (D2D) terminal device, a vehicle to everything (V2X) communication terminal device, an intelligent vehicle, a telematics box (or vehicle networking system), a machine-to-machine / machine-type communications (M2M / MTC) terminal device, or an internet of things (IoT) terminal device.

[0067] It should be understood that, in each of the foregoing examples of the terminal device, only some of the examples may be used as the first device, the second device, or the third device. In other words, the first device, the second device, or the third device may be a part of the terminal devices in the foregoing examples. For example, the first device may be a set-top box, the second device may be a mobile phone, and the third device may be a mobile phone; or the first device may be a set-top box, the second device may be a set-top box, and the third device may be a PAD.

[0068] Among them, the set-top box may be a device integrated with the display screen or an independent device from the display screen. When the set-top box is independent of the display screen, the set-top box is connected to the display screen to jointly implement calls with other devices or screen mirroring information sharing.

[0069] Currently, the screen mirroring sharing uses the method of screen capture recording. This method requires decoding, rendering, and screen recording encoding of the screen mirroring data, resulting in a relatively long end-to-end data processing chain and a large delay; re-encoding the rendered image will cause loss of video quality; if an unexpected pop-up window appears in the screen recording area, the unexpected content will be transmitted to the other end, resulting in a poor experience. Based on this, the embodiments of the application provide a screen mirroring sharing method. The first device can directly extract the elementary stream (ES) of the screen mirroring information and transmit it to the other end, subtracting the time of decoding, screen recording, and re-encoding, and greatly reducing the end-to-end delay of screen mirroring sharing. And this method can improve the picture accuracy and the picture experience effect; at the same time, it can avoid the transmission of unexpected content to the other end, thereby improving the user experience.

[0070] Based on the above description, the embodiments of the application provide a screen mirroring sharing method. Refer to Figure 2 as shown, the method may include the following steps:

[0071] Step 201: When a call connection is established between the first device and the second device, the first device obtains first instruction information, and the first instruction information is used to indicate sharing of screen mirroring information.

[0072] In some embodiments, the first device may be a set-top box, the second device is the call counterpart of the first device, the second device may be a set-top box or a mobile phone, etc., and the third device may be a mobile phone or a PAD that projects a screen to the first device. Optionally, the first device and the third device may correspond to the same user.

[0073] Of course, the above is only an example and is not a limitation to this application.

[0074] Among them, the call connection can be understood to include connections for video calls and voice calls.

[0075] The first device or the second device may call the other party to establish a call connection between the first device and the second device. In some embodiments, when the first device and the second device establish a call connection, the first device and the second device may interact to determine whether they have a first capability. The first capability is to support the channel corresponding to the call connection and channels other than the channel corresponding to the call connection.

[0076] Optionally, whether the first device and the second device have the first capability can also be understood as whether the first device and the second device support multi-channels. Among them, supporting multi-channels means supporting more than 1 video channel and 1 audio channel. That is to say, the channel corresponding to the call connection includes 1 video channel and 1 audio channel. If the first device and the second device support multi-channels (that is, have the first capability), the first device and the second device can support at least 1 video channel and / or at least 1 audio channel other than the call connection.

[0077] If the first device and the second device do not support multi-channels (that is, do not have the first capability), it can be understood that the first device and the second device support single-channel, that is, the first device and the second device only support 1 video channel and 1 audio channel.

[0078] In an alternative embodiment, the first device and the second device may interact to determine whether they have the first capability by the following method: the first device sends a first identifier to the second device, and the first identifier is used to indicate whether the first device has the first capability; and, the first device receives a second identifier from the second device, and the second identifier is used to indicate whether the second device has the first capability. Among them, the first identifier or the second identifier can also be understood as an identifier indicating whether multi-channels are supported.

[0079] Exemplarily, the first identifier may be an indication of the number of supported channels, or the first identifier may be a preset code corresponding to the first device; the second identifier is an indication of the number of supported channels, or the second identifier is a preset code corresponding to the second device.

[0080] The preset code can be used to determine whether the device has the first capability (i.e., whether it supports multi-channel). Optionally, the preset code corresponding to the first device can be the number corresponding to the first device for being called, etc., such as the communication number assigned by the operator. For example, the communication number of the first device can be the area code of the region where the first device is located plus a fixed number with A digits, A is a positive integer, for example, A can be 8, etc., which is not limited in this application. The preset code corresponding to the second device can be the number corresponding to the second device for being called, etc., such as the communication number assigned by the operator. For example, the communication number of the second device can be the area code of the region where the second device is located plus a fixed number with B digits, B is a positive integer, for example, B can be 8, etc., which is not limited in this application.

[0081] In a possible manner, the first device may also directly determine whether the second device has the first capability by the form of the number it calls (here, that is, the form of the number of the second device). The second device may also directly determine whether the first device has the first capability by the form of the number it calls (here, that is, the form of the number of the first device).

[0082] The interaction between the first device and the second device when establishing a call connection can be understood as the first device and the second device performing call media negotiation. It should be understood that during the call media negotiation, the first device and the second device can negotiate other call parameters in addition to the interaction of whether they have the first capability, and this application does not limit this.

[0083] Furthermore, after the call between the first device or the second device is successful, a call connection is established between the first device and the second device, and a call video stream and / or a call audio stream can be transmitted between the first device and the second device.

[0084] In some embodiments, the first device may obtain the first instruction information in the following two ways:

[0085] Mode a1: The user of the first device can trigger the first instruction information through the first device, so that the first device obtains the first instruction information.

[0086] For example, a user of the first device can trigger the screen projection sharing function through the user interface corresponding to the first device, that is, trigger the first instruction information.

[0087] The user interface corresponding to the first device may be displayed on a display screen arranged in the first device, or may be displayed on a display screen connected to the first device.

[0088] Method a2: The third device initiates screen projection to the first device, triggering the first command information so that the first device obtains the first command information.

[0089] In this method a2, when the user of the third device sees something that they want to share with the user of the second device, the user can initiate screen mirroring from the third device to the first device, thereby triggering the first instruction information.

[0090] Step 202: The first device receives the screen mirroring information from the third device based on the first instruction information. Correspondingly, the third device sends the screen mirroring information to the first device.

[0091] In some embodiments, before the first device receives the screen mirroring information from the third device, the first device and the second device may negotiate first parameter information, which is used to share the screen mirroring information during the call between the first device and the second device. Among them, the negotiation process of the first parameter information can be initiated by the first device or the second device, which is not limited in this application.

[0092] It should be understood that the negotiation described in this application is a process of two devices interacting with each other, and the specific negotiation process is not limited in this application. Among them, during the negotiation process of the two devices, this application does not limit the initiating device of the negotiation process, that is, the negotiation process can be initiated by either of the two devices.

[0093] The above process of negotiating the first parameter information can also be understood as shared screen mirroring media negotiation, or it can also be understood as call media re-negotiation.

[0094] In a possible implementation, when the first device and the second device support multiple channels, the first parameter information may include the audio channel and / or video channel used for the shared screen mirroring information to be established between the first device and the second device. That is, when the first device and the second device support multiple channels, the first device and the second device can negotiate channels other than those included in the call connection to use different channels to transmit the content of the screen mirroring and the content of the current call.

[0095] Optionally, the media parameters of the audio channel and / or video channel used for the shared screen mirroring information can adopt preset parameters or common screen mirroring parameters.

[0096] In some possible ways, when the first device and the second device do not support multiple channels, the first device and the second device can directly reuse the channels included in the call connection as the audio channel and / or video channel used for the shared screen mirroring information. Therefore, the first device and the second device may not negotiate the first parameter information.

[0097] Optionally, when the first device and the second device do not support multiple channels, if the first device or the second device anticipates that the screen mirroring parameters are inconsistent with the current call parameters, the first parameter information can be negotiated. The first parameter information may include common parameters or preset parameters for screen mirroring.

[0098] In some embodiments, before the first device receives the screen mirroring information from the third device, the first device and the third device may negotiate second parameter information, which is used for the third device to screen mirror to the first device. Among them, the negotiation process of the second parameter information may be initiated by the first device or the third device, which is not limited in this application.

[0099] The above process of negotiating the second parameter information can be understood as screen mirroring media negotiation.

[0100] Optionally, the second parameter information may include an encoding format, a resolution, etc.

[0101] In an alternative embodiment, before the first device receives the screen mirroring information from the third device, the first device determines whether the first parameter information and the second parameter information are consistent; if so, the first device sends a screen mirroring instruction to the third device; otherwise, the first device renegotiates the first parameter information with the second device and the first device renegotiates the second parameter information with the third device. After determining that the renegotiated first parameter information and the second parameter information are consistent, the first device sends a screen mirroring instruction to the third device.

[0102] Optionally, when the first device determines that the first parameter and the second parameter information are inconsistent, the first device may record the parameter information supported by the third device.

[0103] In an example, if the first device records the parameter information (which can be understood as the second parameter information) supported by the third device, the first device may negotiate or renegotiate the first parameter information according to the second parameter information.

[0104] In some embodiments, the screen mirroring information includes audio information and / or video information. Correspondingly, when the third device sends the screen mirroring information to the first device, it may be that the third device sends audio information (or an audio stream) and / or video information (or a video stream) to the first device.

[0105] Step 203: The first device extracts the ES of the screen mirroring information.

[0106] Since the screen mirroring information includes audio information and / or video information, when the first device extracts the ES of the screen mirroring information, it may be that the first device extracts the ES of the audio information and / or the ES of the video information.

[0107] For example, the ES may be encoded data such as video H264 / H265 or audio Advanced Audio Coding (AAC).

[0108] Step 204: The first device plays the ES and sends the ES to the second device.

[0109] In some embodiments, the first device playing the ES may be the first device playing the ES on the first device. In some other embodiments, the first device playing the ES may be downloading the ES to the first device through an application (APP) and then playing it on the first device. In still some other embodiments, the first device playing the ES may also be the first device playing the ES through the APP on the first device. Of course, there may be other playback situations, which are not limited in this application. The same applies to other devices playing in this application, and the following embodiments of this application will not be elaborated one by one.

[0110] In some embodiments, the first device sending the ES to the second device may include the following two situations:

[0111] Situation b1: The first device sends the ES of the audio information to the second device through a first audio channel, and / or sends the ES of the video information to the second device through a first video channel.

[0112] Among them, the first audio channel is the audio channel corresponding to the call connection between the first device and the second device, and the first video channel is the video channel corresponding to the call connection between the first device and the second device.

[0113] In this situation b1, the first device and the second device do not have the first capability (that is, they do not support multi-channels and only support single-channel), and the first device and the second device multiplex the channels of the call connection to transmit the screen projection information.

[0114] Situation b2: The first device sends the ES of the audio information to the second device through a second audio channel, and / or sends the ES of the video information to the second device through a second video channel.

[0115] Among them, the second audio channel is the audio channel used by the first device and the second device to establish shared screen projection information, and the second video channel is the video channel used by the first device and the second device to establish shared screen projection information.

[0116] In this situation b2, the first device and the second device have the first capability (that is, they support multi-channels), and the first device and the second device can use channels other than the channels of the call connection and used for sharing screen projection information to transmit the screen projection information.

[0117] In an alternative embodiment, the first device may also extract the original packet time (PTS) corresponding to the ES of the screen mirroring information; the first device records the PTS and sends the PTS to the second device. In this way, the PTS can be used as the timestamp corresponding to the screen mirroring information to keep the screen mirroring information synchronized between the first device and the second device.

[0118] Exemplarily, the first device may extract the PTS corresponding to the ES of the audio information and / or the PTS corresponding to the ES of the video information, and send the PTS corresponding to the ES of the audio information and / or the PTS corresponding to the ES of the video information to the second device.

[0119] In some embodiments, the first device may encapsulate and send the ES of the audio information and the PTS corresponding to the ES of the audio information together, or send them separately. The same applies to the video information, which is not limited in this application.

[0120] Step 205: The second device plays the ES.

[0121] Based on the above screen mirroring sharing method, the first device can directly extract the ES of the screen mirroring information and transmit it to the peer end, subtracting the time of decoding, recording the screen, and then encoding, which greatly reduces the end-to-end delay of screen mirroring sharing. And this method can improve the picture accuracy and the picture experience effect; at the same time, it can avoid transmitting content that is not expected by the user to the peer end, improving the user experience.

[0122] In some embodiments, if the first parameter information and the second information are inconsistent, in order to ensure successful screen mirroring sharing, the first device may also start a common screen recording or transcoding process. The situation where the first parameter information and the second information are inconsistent may be inconsistent in negotiation or inconsistent in re-negotiation, which is not limited in this application.

[0123] Based on the above embodiments, the screen mirroring sharing method provided by the embodiments of the present application will be described in detail below through specific scenarios and examples. It should be understood that the following examples are only for illustrative purposes and do not limit the present application. In the following scenarios and examples, the first device is a set-top box (hereinafter exemplified by the first set-top box), the second device is a set-top box (hereinafter exemplified by the second set-top box), and the third device is a mobile phone as an example.

[0124] Exemplarily, Figure 3Shows a screen mirroring and sharing scenario of a set-top box supporting multiple channels. When a user triggers screen mirroring and sharing during a call connection between the first set-top box and the second set-top box, the first set-top box and the second set-top box perform shared screen mirroring media negotiation, and can negotiate multiple channels for sharing screen mirroring information, audio and video media parameters (such as parameters corresponding to the highest resolution on the call side), etc. The mobile phone initiates screen mirroring to the first set-top box, and the first set-top box and the mobile phone perform screen mirroring media negotiation, and the screen mirroring parameters use the parameters after the above-mentioned shared screen mirroring media negotiation. After that, the mobile phone can send screen mirroring information to the first set-top box. The first set-top box extracts the ES stream of the screen mirroring audio and video information, plays it simultaneously, and transmits it to the second set-top box at the opposite end through the corresponding channels. This can eliminate the screen stream decoding, rendering, and then screen recording and encoding links of screen mirroring, reduce the end-to-end (E2E) latency, and through multiple channels, the screen of the first set-top box and the audio and video streams of the call can be transmitted to the second set-top box, as Figure 4 shown. This can achieve the following effects: Through ES passthrough and 4-channel transmission (screen mirroring information and call audio and video), the desktop shared by the mobile phone, the image captured by the camera of the first set-top box, and the image captured by the camera of the second set-top box can be displayed simultaneously. The second set-top box on the call opposite end can hear the screen mirroring audio stream and the call audio stream simultaneously, and the picture quality is clear and the latency is low. Among them, Figure 4 in, A and B represent the users of the first set-top box and the second set-top box.

[0125] Based on Figure 3 the scenario shown, Figure 5 shows an example of a screen mirroring and sharing method. The process of this example can include the following steps:

[0126] S501: The user of the first set-top box or the second set-top box makes a call, and the first set-top box and the second set-top box perform call media negotiation.

[0127] Among them, as Figure 6 shown, in the call media negotiation message, the first set-top box and the second set-top box can respectively carry identifiers indicating whether they support multiple channels, so as to judge whether they support multiple channels (supporting more than 1 video and 1 audio) through the identifiers; or, judge whether they support multiple channels through the form of the called number.

[0128] Exemplarily, the identifier indicating whether it supports multiple channels can refer to the relevant descriptions of the first identifier and the second identifier described in the foregoing embodiments, and will not be described in detail here.

[0129] In this example, the first set-top box and the second set-top box support multiple channels.

[0130] S502: After the call is successful, the call video stream is transmitted between the first set-top box and the second set-top box.

[0131] S503: After the call is successful, the call audio stream is transmitted between the first set-top box and the second set-top box.

[0132] It should be understood that the sequence order of S502 and S503 is only an example, and this application does not make any limitation.

[0133] S504: The user of the first set-top box triggers the screen mirroring sharing function on the user interface corresponding to the first set-top box, and the first set-top box and the second set-top box perform shared screen mirroring media negotiation (or called call media renegotiation).

[0134] During the shared screen mirroring media negotiation process, the first set-top box and the second set-top box can negotiate the audio channel and / or video channel required for the newly added shared screen mirroring information. The media parameters of this audio channel and / or video channel can use preset parameters or the commonly used parameters for screen mirroring.

[0135] After the shared screen mirroring media negotiation is completed, the user interface of the first set-top box can be adjusted to the screen to be mirrored, reminding the user that they can initiate screen mirroring.

[0136] S505: The mobile phone user initiates screen mirroring through the mobile phone, and the mobile phone and the first set-top box can perform screen mirroring media negotiation.

[0137] Exemplarily, the user can initiate mirror screen mirroring through the mobile phone (such as wireless screen mirroring (miracast) or Cast+ (Cast+ is a screen mirroring protocol), etc.).

[0138] Optionally, the main media parameters used during the screen mirroring media negotiation can be the parameters negotiated by the first set-top box and the second set-top box during the shared screen mirroring media negotiation process (such as encoding format, resolution, etc.).

[0139] S506: When the screen mirroring media negotiation is successful, the first set-top box can send a screen mirroring instruction to the mobile phone to notify the mobile phone that it can transmit screen mirroring information (such as audio information and video information).

[0140] S507: The mobile phone transmits screen mirroring video information to the first set-top box.

[0141] For example, the screen mirroring video information can be the screen video stream of the mobile phone.

[0142] S508: The mobile phone transmits screen mirroring audio information to the first set-top box.

[0143] For example, the screen mirroring audio information can be the screen audio stream of the mobile phone.

[0144] It should be understood that the sequence order of S507 and S508 is only an example, and this application does not make any limitation.

[0145] S509: The first set-top box performs ES extraction on the screen mirroring video information, plays the ES of the screen mirroring video information, extracts the PTS corresponding to the ES of the screen mirroring video information, and packs the ES of the screen mirroring video information and the corresponding PTS in the transmission format of the call channel.

[0146] Optionally, the first set-top box may also extract other original data containing the ES of the screen mirroring video information, such as packetized elementary stream (PES), transport stream (TS), etc.

[0147] S510: The first set-top box sends the ES of the screen mirroring video information and the corresponding PTS packed in the transmission format of the call channel to the second set-top box.

[0148] Among them, the first set-top box sends the ES of the screen mirroring video information and the corresponding PTS through the negotiated video channel for sharing screen mirroring information.

[0149] S511: The first set-top box performs ES extraction on the screen mirroring audio information, plays the ES of the screen mirroring audio information, extracts the PTS corresponding to the ES of the screen mirroring audio information, and packs the ES of the screen mirroring audio information and the corresponding PTS in the transmission format of the call channel.

[0150] Optionally, the first set-top box may also extract other original data containing the ES of the screen mirroring audio information, such as PES, TS, etc.

[0151] S512: The first set-top box sends the ES of the screen mirroring audio information and the corresponding PTS packed in the transmission format of the call channel to the second set-top box.

[0152] Among them, the first set-top box sends the ES of the screen mirroring audio information and the corresponding PTS through the negotiated audio channel for sharing screen mirroring information.

[0153] Among them, the execution of S509 and S511, that is, the operations of the first set-top box as shown in Figure 6 , while extracting the ES of the screen mirroring audio information and the screen mirroring video information, record the corresponding PTS, and pack the corresponding PTS into the transmission packet of the channel for sharing screen mirroring information as the timestamp of the source data, which can keep the screen mirroring information synchronized between the first set-top box and the second set-top box.

[0154] It should be understood that the above sequential order of S509 - S510 and S511 - S512 is only an example, and this application is not limited.

[0155] S513: The second set-top box plays the ES of the screen mirroring audio information and the ES of the screen mirroring video information.

[0156] Exemplarily, Figure 7 A screen mirroring sharing scenario of a set-top box that does not support multi-channels is shown. When a call connection is established between a first set-top box and a second set-top box, the first set-top box and the second set-top box can directly reuse or renegotiate (anticipate that the screen mirroring parameters and call parameters are inconsistent and initiate negotiation in advance) the channels of the call connection. The first set-top box can send screen mirroring audio and video streams, or screen mirroring video streams and call audio streams to the second set-top box.

[0157] Based on Figure 7 the scenario shown, Figure 8 an example of a screen mirroring sharing method is shown. The process of this example may include the following steps:

[0158] S801 is similar to S501 and can be referred to each other. Details are not repeated here. Different from S501, in S801, the determined result may be that the first set-top box supports multi-channels and the second set-top box does not support multi-channels, that is, it supports single-channel. Or neither the first set-top box nor the second set-top box supports multi-channels.

[0159] S802 - S803 are similar to S502 - S503 and can be referred to each other. Details are not repeated here.

[0160] S804: The user of the first set-top box triggers the screen mirroring sharing function on the user interface corresponding to the first set-top box, and the first set-top box and the second set-top box determine that the channels of the call connection can be directly reused. Optionally, as shown in Figure 8 S804, the first set-top box and the second set-top box perform shared screen mirroring media negotiation (or called call media renegotiation).

[0161] Optionally, when the first set-top box or the second set-top box anticipates that the screen mirroring parameters and call parameters are inconsistent, the shared screen mirroring media negotiation process can be executed. For example, the first set-top box and the second set-top box can initiate the shared screen mirroring media negotiation using the common parameters of screen mirroring.

[0162] After the shared screen mirroring media negotiation is completed, or when it is determined to use the direct reuse method, the user interface of the first set-top box can be adjusted to the screen to be mirrored, reminding the user that screen mirroring can be initiated.

[0163] S805 - S808 are similar to S505 - S508 and can be referred to each other. Details are not repeated here.

[0164] S809: The first set-top box performs ES extraction on the screen mirroring video information, plays the ES of the screen mirroring video information, extracts the PTS corresponding to the ES of the screen mirroring video information, and packs the ES of the screen mirroring video information and the corresponding PTS in the transmission form of the call channel.

[0165] Optionally, the first set-top box can also extract other raw data of the ES containing the screen mirroring video information, such as PES, TS, etc.

[0166] S810: The first set-top box sends the ES of the screen mirroring video information and the corresponding PTS, which are packed in the transmission form of the call channel, to the second set-top box.

[0167] Among them, the first set-top box sends the ES of the screen mirroring video information and the corresponding PTS through the video channel of the multiplexed call connection.

[0168] S811: The first set-top box extracts the ES of the screen mirroring audio information, extracts the PTS corresponding to the ES of the screen mirroring audio information, and packs the ES of the screen mirroring audio information and the corresponding PTS in the transmission form of the call channel.

[0169] Optionally, the first set-top box can also extract other raw data of the ES containing the screen mirroring audio information, such as PES, TS, etc.

[0170] S812: The first set-top box plays the audio stream and sends the audio stream to the second set-top box. Among them, the audio stream can include the ES of the screen mirroring audio information and the corresponding PTS packed in the transmission form of the call channel, or the audio stream can include the audio data of the call connection, or the audio stream can include the mixed audio data of the ES of the screen mirroring audio information and the audio data of the call connection.

[0171] Optionally, the user interface of the first set-top box can display prompt information for the user to select. The prompt information can include the audio data of the call connection, the screen mirroring audio information, or the mixed audio data of the two above. The first set-top box can determine the specific content of the audio stream based on the user's selection instruction.

[0172] Among them, the first set-top box sends the audio stream through the audio channel of the multiplexed call connection.

[0173] S813: The second set-top box plays the ES of the screen mirroring video information and the audio stream.

[0174] In some embodiments, the parameters (such as the encoding format, resolution information, etc.) used by the mobile phone and the set-top box can be negotiated to be consistent. However, there may be an inconsistent situation between the call media parameters (i.e., the shared screen mirroring parameters) (i.e., the first parameter information described above) for sharing the screen mirroring information negotiated between the first set-top box and the second set-top box and the screen mirroring parameters (i.e., the second parameter information described above) negotiated between the first set-top box and the mobile phone. For example Figure 9It shows a schematic diagram of reminding the user during the screen mirroring process when the screen mirroring parameters and the shared screen mirroring parameters are inconsistent in the scenario where the first set-top box and the second set-top box support multiple channels. In the scenario of supporting multiple channels, the negotiation of shared screen mirroring media is completed first. Based on the negotiated shared screen mirroring media parameters, the first set-top box conducts screen mirroring negotiation with the mobile phone. If the parameters are inconsistent, the negotiation fails, and the user needs to be reminded that the parameters are inconsistent and the screen mirroring needs to be restarted. The first set-top box can record the parameters supported by the mobile phone side. Based on the parameters supported by the mobile phone side, the shared screen mirroring media is renegotiated. After the negotiation is completed, the user is reminded that the status is ready. The user conducts screen mirroring again to complete the negotiation of the screen mirroring media. As Figure 10 It shows a schematic diagram of reminding the user during the screen mirroring process when the screen mirroring parameters and the shared screen mirroring parameters are inconsistent in the scenario where the first set-top box and the second set-top box support a single channel. In the scenario of supporting a single channel, compared with the scenario of supporting multiple channels, the channel for sharing screen mirroring information may reuse the channel of the call connection. Therefore, it is optional to conduct the negotiation of shared screen mirroring media before screen mirroring (it can be predicted that the screen mirroring parameters and the call parameters will be inconsistent, and the negotiation can be initiated in advance). Other processes are the same as those in the multiple-channel scenario.

[0175] Next, the possible situations after the parameters are inconsistent will be described through different examples.

[0176] Figure 11 It shows an example of the renegotiation process when the screen mirroring parameters and the shared screen mirroring parameters are inconsistent. The process of this example may include the following steps:

[0177] S1101: The user of the first set-top box triggers the screen mirroring sharing function on the user interface corresponding to the first set-top box. The first set-top box and the second set-top box conduct negotiation of shared screen mirroring media (or called renegotiation of call media). Or, in the single-channel scenario, the first set-top box and the second set-top box reuse the channel of the call.

[0178] Figure 11 In, S1101 is exemplarily shown with the first set-top box and the second set-top box conducting negotiation of shared screen mirroring media.

[0179] Among them, the specific content of the negotiation of shared screen mirroring media can refer to the relevant descriptions involved in the foregoing embodiments, and will not be elaborated here.

[0180] S1102: The mobile phone user initiates screen mirroring through the mobile phone. The mobile phone and the first set-top box can conduct negotiation of screen mirroring media.

[0181] Exemplarily, the user can initiate mirror screen mirroring through the mobile phone (such as miracast or Cast+).

[0182] Optionally, the main media parameters used in the screen mirroring media negotiation may be the parameters negotiated by the first set-top box and the second set-top box during the shared screen mirroring media negotiation (such as encoding format, resolution, etc.).

[0183] S1103: The screen mirroring media negotiation between the mobile phone and the first set-top box fails, that is, the parameter negotiation fails. The first set-top box records the parameters supported by the mobile phone.

[0184] Among them, the user interface corresponding to the first set-top box can remind the user that the screen mirroring parameters are inconsistent with the shared screen mirroring parameters, and the screen mirroring needs to be performed again.

[0185] S1104: The first set-top box and the second set-top box can perform a shared media re-negotiation (that is, a call media re-negotiation).

[0186] Among them, when the first set-top box and the second set-top box re-negotiate, they can negotiate based on the parameters supported by the mobile phone.

[0187] After the shared media re-negotiation is successful, the user interface of the first set-top box can remind the user that screen mirroring can be performed.

[0188] S1105: The first set-top box and the mobile phone perform screen mirroring media negotiation again, and the parameters of the re-screen mirroring media negotiation are the same as those of the shared screen mirroring media re-negotiation (such as the encoding format, resolution, etc. are the same).

[0189] S1106: The screen mirroring media negotiation between the first set-top box and the mobile phone is successful again. The first set-top box can send a screen mirroring instruction to the mobile phone to notify the mobile phone that it can transmit screen mirroring audio and video information.

[0190] Figure 12 An example of screen mirroring with inconsistent screen mirroring parameters and shared screen mirroring parameters using other screen mirroring methods (such as screen recording or transcoding methods) is shown. The process of this example may include the following steps:

[0191] S1201 - S1203 are similar to S1101 - S1103 and can be referred to each other, and will not be elaborated here.

[0192] S1204: The first set-top box determines that the parameters supported by the mobile phone are incompatible with the parameters supported by the call media (that is, the shared screen mirroring parameters), starts the screen recording or transcoding function, and enters the state of waiting for screen mirroring. And the user interface of the first set-top box can remind the user that screen mirroring is available.

[0193] S1205: The mobile phone user initiates screen mirroring through the mobile phone, and the mobile phone and the first set-top box can perform screen mirroring media negotiation again.

[0194] Exemplarily, the user can initiate mirror screen mirroring (such as miracast or Cast+) through the mobile phone.

[0195] It should be understood that the screen mirroring media negotiation in S1205 can refer to the negotiation process of the screen mirroring scenario in the current screen recording or transcoding function, which will not be elaborated here.

[0196] S1206: The screen mirroring media negotiation between the mobile phone and the first set-top box is successful. The first set-top box can send a screen mirroring instruction to the mobile phone to notify the mobile phone to transmit screen mirroring information (such as audio information and video information).

[0197] S1207: The mobile phone transmits screen mirroring video information to the first set-top box.

[0198] For example, the screen mirroring video information can be the screen video stream of the mobile phone.

[0199] S1208: The mobile phone transmits screen mirroring audio information to the first set-top box.

[0200] For example, the screen mirroring audio information can be the screen audio stream of the mobile phone.

[0201] It should be understood that the sequence of S1207 and S1208 is only an example, and this application does not make any limitations.

[0202] S1209: The first set-top box performs screen recording or transcoding on the screen mirroring video information and the screen mirroring audio information.

[0203] S1210: The first set-top box packs the screen recorded (or transcoded) screen mirroring video information in the transmission format of the call channel, and sends the packed screen recorded (or transcoded) screen mirroring video information to the second set-top box.

[0204] S1211: The first set-top box packs the screen recorded (or transcoded) screen mirroring audio information in the transmission format of the call channel, and sends the packed screen recorded (or transcoded) screen mirroring audio information to the second set-top box.

[0205] In some embodiments, the screen mirroring from the mobile phone to the first set-top box can trigger call media re-negotiation (i.e., shared screen mirroring media negotiation). As Figure 13 shown in the scenario schematic, the user initiates screen mirroring through the mobile phone. The mobile phone and the first set-top box perform screen mirroring media negotiation, carrying media parameters (such as encoding format, resolution, etc.). The first set-top box does not respond to the supported media parameters for the time being, and instructs the mobile phone to wait for screen mirroring media negotiation. The first set-top box performs call media re-negotiation (i.e., shared screen mirroring media negotiation) based on the media parameters supported by the mobile phone. After the negotiation is completed, the first set-top box replies to the mobile phone with the supported media parameters, and the remaining process of the screen mirroring media negotiation is completed, such as the process in stage 2 in the figure.

[0206] Based on Figure 13 the scenario shown, Figure 14Shows an example of the process with consistent parameters when a mobile phone casts the screen to the first set-top box, which can trigger call media re-negotiation. The process of this example may include the following steps:

[0207] S1401 - S1403 are similar to the aforementioned S501 - S503, and can be referred to each other. Details are not repeated here.

[0208] S1404: The mobile phone user initiates screen mirroring through the mobile phone. The mobile phone and the first set-top box conduct screen mirroring media negotiation. At this time, the screen mirroring media negotiation between the mobile phone and the first set-top box is recorded as stage 1 of the screen mirroring media negotiation process.

[0209] Exemplarily, the user can initiate mirror screen casting (such as miracast or Cast+) through the mobile phone.

[0210] Optionally, the mobile phone and the first set-top box can conduct screen mirroring media negotiation based on media parameters such as encoding format and resolution.

[0211] In stage 1 of the screen mirroring media negotiation process, the first set-top box does not respond to the supported media parameters for the time being and waits for negotiation.

[0212] S1405: The first set-top box records the media parameters supported by the mobile phone.

[0213] S1406: The first set-top box sends a screen mirroring media negotiation timeout waiting message to the mobile phone to make the mobile phone wait for the reply from the first set-top box.

[0214] S1406 is an optional step.

[0215] S1407: The first set-top box and the second set-top box conduct call media re-negotiation (i.e., shared screen mirroring media negotiation) based on the media parameters supported by the mobile phone.

[0216] Optionally, in the case where the first set-top box and the second set-top box support multiple channels, a new channel needs to be negotiated; in the case where the first set-top box and the second set-top box support a single channel, if the screen mirroring parameters and call parameters are consistent, they can be directly reused, otherwise, the screen mirroring parameters and call parameters need to be re-negotiated.

[0217] S1408: The first set-top box replies to the mobile phone with the supported media parameters. The mobile phone and the first set-top box complete the remaining process of the screen mirroring media negotiation. At this time, the screen mirroring media negotiation between the mobile phone and the first set-top box is recorded as stage 2 of the screen mirroring media negotiation process.

[0218] S1409: The screen mirroring media negotiation between the mobile phone and the first set-top box is successful. The first set-top box can send a screen mirroring instruction to the mobile phone to notify the mobile phone that it can transmit screen mirroring information (such as audio information and video information).

[0219] Based on Figure 13The scene shown Figure 15 shows an example of the process of inconsistent parameters when the mobile phone casts the screen to the first set-top box and triggers call media re-negotiation. The process of this example may include the following steps:

[0220] S1501 - S1506 are similar to S1401 - S1406 and can be referred to each other. Details are not repeated here.

[0221] S1507: The first set-top box and the second set-top box perform call media re-negotiation (i.e., shared screen mirroring media negotiation) based on the media parameters supported by the mobile phone. The re-negotiation fails, that is, the screen mirroring parameters and the shared screen mirroring parameters are inconsistent.

[0222] S1508: The first set-top box and the second set-top box perform call media re-negotiation again based on the media parameters supported by the second set-top box, and the negotiation is successful.

[0223] S1509: The first set-top box starts the screen recording or transcoding function and enters the state waiting for screen mirroring.

[0224] S1510: The first set-top box sends the supported media parameters to the mobile phone, and the mobile phone and the first set-top box complete the remaining process of screen mirroring media negotiation. At this time, the screen mirroring media negotiation between the mobile phone and the first set-top box is recorded as stage 2 of the screen mirroring media negotiation process.

[0225] S1511: The screen mirroring media negotiation is successful, and the first set-top box can send a screen mirroring instruction to the mobile phone to notify the mobile phone that it can transmit screen mirroring information (such as audio information and video information).

[0226] S1512: The mobile phone transmits screen mirroring video information to the first set-top box.

[0227] For example, the screen mirroring video information may be the screen video stream of the mobile phone.

[0228] S1513: The mobile phone transmits screen mirroring audio information to the first set-top box.

[0229] For example, the screen mirroring audio information may be the screen audio stream of the mobile phone.

[0230] It should be understood that the sequence of S1512 and S1513 is only an example and is not limited in this application.

[0231] S1514: The first set-top box performs screen recording or transcoding on the screen mirroring video information and the screen mirroring audio information.

[0232] S1515: The first set-top box packs the screen recorded (or transcoded) screen mirroring video information in the transmission form of the call channel, and sends the packed screen recorded (or transcoded) screen mirroring video information to the second set-top box.

[0233] S1516: The first set-top box packs the screen-captured (or transcoded) screen mirroring audio information in the transmission format of the call channel, and sends the packed screen-captured (or transcoded) screen mirroring audio information to the second set-top box.

[0234] Based on the above embodiments, the embodiments of the present application further provide a communication device. Refer to Figure 16 As shown, the communication device 1600 may include a communication unit 1601 and a processing unit 1602. Among them, the communication unit 1601 is used for the communication device 1600 to communicate, such as receiving information (messages or data) or sending information (messages or data), and the processing unit 1602 is used to control and manage the actions of the communication device 1600. The processing unit 1602 may also control the steps executed by the communication unit 1601.

[0235] Exemplarily, the communication device 1600 may specifically be the first device, the processor of the first device, or a chip, or a chip system, or a functional module in the above embodiments. Or, the communication device 1600 may specifically be the second device, the processor in the second device, or a chip, or a chip system, or a functional module in the above embodiments.

[0236] In one embodiment, when the communication device 1600 is used to implement the functions of the first device in the above embodiments, the communication unit 1601 may be used to obtain a first instruction message when a call connection is established between the first device and the second device, where the first instruction message is used to instruct to share screen mirroring information; and receive the screen mirroring information from the third device based on the first instruction message; the processing unit 1602 may be used to extract the elementary stream ES of the screen mirroring information and play the ES; the communication unit 1602 may also be used to send the ES to the second device.

[0237] In an optional implementation manner, the processing unit 1602 may also be used to negotiate first parameter information, where the first parameter information is used for sharing screen mirroring information during the call between the first device and the second device.

[0238] Exemplarily, the first parameter information includes the audio channel and / or video channel used for the first device and the second device to establish shared screen mirroring information.

[0239] In an optional implementation manner, the processing unit 1602 may also be used to negotiate second parameter information, where the second parameter information is used for the third device to project the screen to the first device.

[0240] In some embodiments, the screen mirroring information includes audio information and / or video information; further, when the processing unit 1602 extracts the ES of the screen mirroring information, it can be used to: extract the ES of the audio information and / or the ES of the video information.

[0241] In a possible manner, when the communication unit 1601 sends the ES to the second device, it can be used to: send the ES of the audio information to the second device through a first audio channel, and / or send the ES of the video information to the second device through a first video channel; the first audio channel is the audio channel corresponding to the call connection between the first device and the second device, and the first video channel is the video channel corresponding to the call connection between the first device and the second device; or

[0242] send the ES of the audio information to the second device through a second audio channel, and / or send the ES of the video information to the second device through a second video channel; the second audio channel is the audio channel used for sharing the screen mirroring information established between the first device and the second device, and the second video channel is the video channel used for sharing the screen mirroring information established between the first device and the second device.

[0243] Optionally, the processing unit 1602 can also be used to extract the original packet time PTS corresponding to the ES of the screen mirroring information and record the PTS; the communication unit 1601 can also be used to send the PTS to the second device.

[0244] In some embodiments, the communication unit 1601 can also be used to, when establishing a call connection between the first device and the second device, interact with the second device to determine whether it has a first capability, where the first capability is to support the channel corresponding to the call connection and channels other than the channel corresponding to the call connection.

[0245] In one example, the communication unit 1601 can also be used to send a first identifier to the second device, where the first identifier is used to indicate whether the first device has the first capability; and receive a second identifier from the second device, where the second identifier is used to indicate whether the second device has the first capability.

[0246] Optionally, the first identifier is an indication of the number of supported channels, or the first identifier is a preset code corresponding to the first device; the second identifier is an indication of the number of supported channels, or the second identifier is a preset code corresponding to the second device.

[0247] In an alternative embodiment, the processing unit 1602 may further be configured to: before the communication unit 1601 receives the screen mirroring information from the third device, determine whether first parameter information and second parameter information are consistent; the first parameter information is used for sharing screen mirroring information during a call between the first device and the second device; the second parameter information is used for the third device to screen mirror to the first device.

[0248] If so, control the communication unit 1601 to send a screen mirroring instruction to the third device.

[0249] Otherwise, renegotiate the first parameter information and renegotiate the second parameter information. After determining that the renegotiated first parameter information and the second information are consistent, control the communication unit 1601 to send a screen mirroring instruction to the third device.

[0250] In some embodiments, the processing unit 1602 may further be configured to record the parameter information supported by the third device when determining that the first parameter and the second parameter information are inconsistent.

[0251] Optionally, when negotiating the first parameter information, the processing unit 1602 may be configured to: negotiate the first parameter information according to the second parameter information, and the second parameter information is used for the third device to screen mirror to the first device.

[0252] In yet another embodiment, when the communication device 1600 is used to implement the function of the second device in the above embodiment, the communication unit 1601 may be configured to receive an elementary stream ES from the first device when a call connection is established between the first device and the second device, and the ES is extracted based on the screen mirroring information of the third device; the processing unit 1602 may be configured to play the ES.

[0253] In an alternative embodiment, the processing unit 1602 may further be configured to negotiate first parameter information, and the first parameter information is used for sharing screen mirroring information during a call between the first device and the second device.

[0254] Exemplarily, the first parameter information includes an audio channel and / or a video channel used for the first device and the second device to establish shared screen mirroring information.

[0255] Optionally, the screen mirroring information includes audio information and / or video information; the ES includes the ES of the audio information and / or the ES of the video information.

[0256] In some embodiments, when the communication unit 1601 receives the ES from the first device, it can be used to: receive the ES of the audio information from the first device through a first audio channel, and / or receive the ES of the video information from the first device through a first video channel; the first audio channel is the audio channel corresponding to the call connection between the first device and the second device, and the first video channel is the video channel corresponding to the call connection between the first device and the second device; or

[0257] receive the ES of the audio information from the first device through a second audio channel, and / or receive the ES of the video information from the first device through a second video channel; the second audio channel is the audio channel used for sharing the screen projection information established between the first device and the second device, and the second video channel is the video channel used for sharing the screen projection information established between the first device and the second device.

[0258] In a possible manner, the communication unit 1601 can also be used to receive the original packet time PTS corresponding to the ES from the first device.

[0259] In some embodiments, the communication unit 1601 can also be used to interact with the first device to determine whether it has a first capability when the first device and the second device establish a call connection, and the first capability is to support the channel corresponding to the call connection and support the channels other than the channel corresponding to the call connection.

[0260] In an example, the communication unit 1601 can also be used to receive a first identifier from the first device, and the first identifier is used to indicate whether the first device has the first capability; and send a second identifier to the first device, and the second identifier is used to indicate whether the second device has the first capability.

[0261] Exemplarily, the first identifier is an indication of the number of supported channels, or the first identifier is a preset code corresponding to the first device; the second identifier is an indication of the number of supported channels, or the second identifier is a preset code corresponding to the second device.

[0262] Optionally, the communication unit 1601 can also be used to receive second parameter information from the first device, and the second parameter information is used for the third device to project the screen to the first device; when negotiating the first parameter information, the processing unit 1602 can be used to: negotiate the first parameter information according to the second parameter information.

[0263] It should be noted that the division of units in the embodiments of this application is illustrative, merely a logical function division, and there may be other division methods in actual implementation. In the embodiments of this application, each functional unit can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0264] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0265] Based on the above embodiments, the embodiments of this application also provide a communication device. Refer to Figure 17 As shown, the communication device 1700 may include a processor 1702. Optionally, the communication device 1700 may further include a communication interface 1701. Optionally, the communication device 1700 may further include a memory 1703. Among them, the memory 1703 may be disposed inside the communication device 1700 or may also be disposed outside the communication device 1700. Among them, the processor 1702 may control the communication interface 1701 to receive and send information, messages, or data, etc.

[0266] Specifically, the processor 1702 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. The processor 1702 may further include a hardware chip. The above-mentioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0267] Among them, the communication interface 1701, the processor 1702, and the memory 1703 are interconnected with each other. Optionally, the communication interface 1701, the processor 1702, and the memory 1703 are interconnected through a bus 1704; the bus 1704 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 17 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0268] In an optional implementation manner, the memory 1703 is used to store programs, etc. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory 1703 may include a RAM, and may also include a non-volatile memory, such as one or more disk memories. The processor 1702 executes the application program stored in the memory 1703 to implement the above functions, thereby implementing the functions of the communication device 1700.

[0269] Exemplarily, the communication device 1700 may be the first device in the above embodiment; it may also be the second device in the above embodiment.

[0270] In one embodiment, when the communication device 1700 implements the functions of the first device in the foregoing method embodiments, the communication interface 1701 may implement the transceiver operations performed by the first device in the foregoing method embodiments; the processor 1702 may implement other operations performed by the first device in the foregoing method embodiments except for the transceiver operations. For specific relevant descriptions, reference may be made to the relevant descriptions in the foregoing method embodiments, and details are not described herein again.

[0271] In another embodiment, when the communication device 1700 implements the functions of the second device in the foregoing method embodiments, the communication interface 1701 may implement the transceiver operations performed by the second device in the foregoing method embodiments; the processor 1702 may implement other operations performed by the second device in the foregoing method embodiments except for the transceiver operations. For specific relevant descriptions, reference may be made to the relevant descriptions in the foregoing method embodiments, and details are not described herein again.

[0272] Based on the above embodiments, the embodiments of the present application provide a system, which may include the first device, the first device, and the third device involved in the above embodiments, etc.

[0273] The embodiments of the present application also provide a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a computer, the computer may implement the screen mirroring and sharing method provided by the foregoing method embodiments.

[0274] The embodiments of the present application also provide a computer program product, which is used to store a computer program. When the computer program is executed by a computer, the computer may implement the screen mirroring and sharing method provided by the foregoing method embodiments.

[0275] The embodiments of the present application also provide a chip, including a processor, the processor is coupled to a memory, and is used to call a program in the memory to enable the chip to implement the screen mirroring and sharing method provided by the foregoing method embodiments.

[0276] The embodiments of the present application also provide a chip, the chip is coupled to a memory, and the chip is used to implement the screen mirroring and sharing method provided by the foregoing method embodiments.

[0277] Those skilled in the art should understand that the embodiments of the present application may be provided as a method, a system, or a computer program product. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0278] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce means for implementing the functions specified in one or more flows and / or blocks. Figure 1 one or more flows and / or blocks Figure 1 in one or more blocks.

[0279] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means that implement the functions specified in one or more flows and / or blocks. Figure 1 one or more flows and / or blocks Figure 1 in one or more blocks.

[0280] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows and / or blocks. Figure 1 one or more flows and / or blocks Figure 1 in one or more blocks.

[0281] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

Claims

1. A screen projection sharing method, characterized in that, Including: When a call connection is established between a first device and a second device, the first device obtains first instruction information for instructing to share screen mirroring information. Based on the first instruction information, the first device receives the screen mirroring information from a third device and extracts the elementary stream (ES) of the screen mirroring information. The first device plays the ES and sends the ES to the second device.

2. The method according to claim 1, wherein The method further includes: Negotiating first parameter information for sharing screen mirroring information during the call between the first device and the second device.

3. The method according to claim 2, wherein The first parameter information includes the audio channel and / or video channel used for establishing the screen mirroring information to be shared between the first device and the second device.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: Negotiating second parameter information for the third device to perform screen mirroring to the first device.

5. The method according to any one of claims 1 to 4, characterized in that, The screen mirroring information includes audio information and / or video information. The first device extracts the ES of the screen mirroring information, including: The first device extracts the ES of the audio information and / or the ES of the video information.

6. The method according to claim 5, wherein The first device sends the ES to the second device, including: The first device sends the ES of the audio information to the second device through a first audio channel, and / or sends the ES of the video information to the second device through a first video channel; the first audio channel is the audio channel corresponding to the call connection between the first device and the second device, and the first video channel is the video channel corresponding to the call connection between the first device and the second device; or The first device sends the ES of the audio information to the second device through a second audio channel, and / or sends the ES of the video information to the second device through a second video channel; the second audio channel is the audio channel used for establishing the screen mirroring information to be shared between the first device and the second device, and the second video channel is the video channel used for establishing the screen mirroring information to be shared between the first device and the second device.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: The first device extracts the presentation time stamp (PTS) corresponding to the ES of the screen mirroring information. The first device records the PTS and sends the PTS to the second device.

8. The method according to any one of claims 1 to 7, characterized in that The method further includes: When the first device and the second device establish a call connection, the first device and the second device interact to determine whether they have a first capability, where the first capability is to support the channel corresponding to the call connection and also support channels other than the channel corresponding to the call connection.

9. The method according to claim 8, wherein The first device and the second device interact to determine whether they have the first capability, including: The first device sends a first identifier to the second device, where the first identifier is used to indicate whether the first device has the first capability. The first device receives a second identifier from the second device, where the second identifier is used to indicate whether the second device has the first capability.

10. The method according to claim 9, wherein The first identifier is an indication of the number of supported channels, or the first identifier is a preset code corresponding to the first device. The second identifier is an indication of the number of supported channels, or the second identifier is a preset code corresponding to the second device.

11. The method according to any one of claims 1 to 10, characterized in that, Before the first device receives the screen mirroring information from the third device, the method further includes: The first device determines whether the first parameter information and the second parameter information are consistent; the first parameter information is used for sharing screen mirroring information during a call between the first device and the second device; the second parameter information is used for the third device to screen mirror to the first device. If so, the first device sends a screen mirroring instruction to the third device. Otherwise, re-negotiate the first parameter information and re-negotiate the second parameter information. After determining that the re-negotiated first parameter information and the second information are consistent, the first device sends a screen mirroring instruction to the third device.

12. The method according to claim 11, wherein The method further includes: When the first device determines that the first parameter and the second parameter information are inconsistent, the first device records the parameter information supported by the third device.

13. The method according to any one of claims 2-4, 11-12, characterized in that, Negotiating the first parameter information includes: Negotiating the first parameter information according to the second parameter information, where the second parameter information is used for the third device to screen mirror to the first device.

14. A screen mirroring and sharing method, characterized in that, Includes: When a call connection is established between the first device and the second device, the second device receives an elementary stream (ES) from the first device, and the ES is extracted based on the screen mirroring information of the third device. The second device plays the ES.

15. The method according to claim 14, characterized in that, The method further includes: Negotiating the first parameter information, where the first parameter information is used for sharing screen mirroring information during a call between the first device and the second device.

16. The method according to claim 15, wherein The first parameter information includes the audio channel and / or video channel used by the first device and the second device to establish shared screen mirroring information.

17. The method according to any one of claims 14 to 16, characterized in that The screen mirroring information includes audio information and / or video information; the ES includes the ES of the audio information and / or the ES of the video information.

18. The method according to claim 17, wherein The second device receiving the ES from the first device includes: The second device receives the ES of the audio information from the first device through a first audio channel, and / or receives the ES of the video information from the first device through a first video channel; the first audio channel is the audio channel corresponding to the call connection between the first device and the second device, and the first video channel is the video channel corresponding to the call connection between the first device and the second device; or The second device receives the ES of the audio information from the first device through a second audio channel, and / or receives the ES of the video information from the first device through a second video channel; the second audio channel is the audio channel used by the first device and the second device to establish shared screen mirroring information, and the second video channel is the video channel used by the first device and the second device to establish shared screen mirroring information.

19. The method according to any one of claims 14-18, characterized in that, The method further includes: The second device receives the presentation time stamp (PTS) of the original packet corresponding to the ES from the first device.

20. The method according to any one of claims 14-19, characterized in that, The method further includes: When the first device and the second device establish a call connection, the second device and the first device interact to determine whether the first capability is available. The first capability supports the channel corresponding to the call connection and also supports channels other than the channel corresponding to the call connection.

21. The method according to claim 20, wherein The interaction between the second device and the first device to determine whether the first capability is available includes: The second device receives a first identifier from the first device, and the first identifier is used to indicate whether the first device has the first capability; The second device sends a second identifier to the first device, and the second identifier is used to indicate whether the second device has the first capability.

22. The method according to claim 21, wherein The first identifier is an indication of the number of supported channels, or the first identifier is a preset code corresponding to the first device; The second identifier is an indication of the number of supported channels, or the second identifier is a preset code corresponding to the second device.

23. The method according to claim 15 or 16, characterized in that, The second device negotiates first parameter information with the first device, including: The second device receives second parameter information from the first device, and the second parameter information is used for a third device to cast a screen to the first device; Negotiate the first parameter information according to the second parameter information.

24. A communication device, characterized in that, It includes a module or unit for executing the method according to any one of claims 1-13, or includes a module or unit for executing the method according to any one of claims 14-23.

25. A communication device, characterized in that, It includes a processor, and the processor is coupled to a memory, wherein: The processor is used to call computer instructions in the memory so that the communication device executes the method according to any one of claims 1-13, or executes the method according to any one of claims 14-23.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called by the computer, they execute the method according to any one of claims 1-13, or execute the method according to any one of claims 14-23.

27. A computer program product, characterized in that, It contains instructions that, when running on a computer, cause the method according to any one of claims 1-13, or the method according to any one of claims 14-23 to be executed.

28. A chip, characterized in that, The chip is coupled to the memory and is used to read and execute program instructions stored in the memory to implement the method according to any one of claims 1-13, or to implement the method according to any one of claims 14-23.

Images