Screen mirroring sharing method and apparatus

Abstract

A screen mirroring sharing method and apparatus, used for realizing screen mirroring sharing. The method comprises: when a call connection has been established between a first device and a second device, the first device acquires first instruction information, the first instruction information being used for instructing to share screen mirroring information; then the first device receives the screen mirroring information from a third device on the basis of the first instruction information, and extracts an elementary stream (ES) of the screen mirroring information; and finally, the first device plays the ES and transmits the ES to the second device. According to the method, a first device can directly extract an ES of screen mirroring information and transmit the ES to a peer device, so that the time for decoding, screen recording, and re-encoding is omitted, and thus the end-to-end time delay of screen mirroring sharing is greatly reduced. Moreover, the method can improve the picture precision and improve the picture experience effect. In addition, the content not desired by a user can be prevented from being transmitted to the peer device, thereby improving the user experience.

Description

Screen projection sharing method and device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of the People's Republic of China on January 16, 2024, with application number 202410068375.5 and application name "A Screen Projection Sharing Method and Device", the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of communication technology, and in particular to a screen projection sharing method and device. Background Art

[0004] Screen sharing and remote collaborative instruction during video conferences or phone calls are becoming increasingly common. With the hardware upgrades of home smart screens and set-top boxes (with cameras), video calling applications have been introduced, allowing users to conduct video calls on both large and small screens. During a call, if a user sees an interesting video or content on a small-screen phone or personal digital assistant (PDA) and wants to share it with the other party, or wants to share homework content on their phone or PDA with a parent on the other end for homework guidance, they might consider using screen mirroring to share it with the other party. Therefore, implementing screen mirroring and sharing has become a pressing issue. Summary of the Invention

[0005] The embodiments of the present application provide a screen projection sharing method and device for realizing screen projection sharing.

[0006] In the first aspect, the present application provides a screen projection sharing method, which can be applied to a first device, a processor, a chip or a functional module in the first device, etc. Taking the application of this method to the first device as an example, the method may include: when a call connection is established between the first device and the second device, the first device obtains first instruction information, and the first instruction information is used to indicate the sharing of screen projection information; then, the first device receives the screen projection information from the third device based on the first instruction information, and extracts the basic stream ES of the screen projection information; finally, the first device plays the ES and sends the ES to the second device.

[0007] Based on this method, the first device can directly extract the ES of the screen projection information and transmit it to the other end, subtracting the time required to decode the recorded screen and then re-encode it, significantly reducing the end-to-end latency of screen projection sharing. This method can also improve image accuracy and enhance the visual experience; it can also prevent the transmission of unexpected content to the other end, improving the user experience.

[0008] In one possible design, first parameter information can also be negotiated, and the first parameter information is used to share screen projection information during a call between the first device and the second device. The negotiation process of the first parameter information can be initiated by the first device or the second device, and this application does not limit it. In this way, based on the first parameter information, the first device and the second device can subsequently realize screen projection sharing during the call.

[0009] In one possible design, the first parameter information may include an audio channel and / or a video channel for the first device and the second device to establish shared screen projection information. In this way, the first device and the second device can transmit screen projection information through a channel other than the call connection channel, so that the first device and the second device can simultaneously maintain a call and share screen projection.

[0010] In one possible design, second parameter information can also be negotiated, and the second parameter information is used by the third device to project the screen to the first device. The negotiation process of the second parameter information can be initiated by the first device or the third device, and this application does not limit it. This allows the third device to accurately transmit the projection information to the first device.

[0011] In one possible design, the projection information may include audio information and / or video information; accordingly, the first device extracting the ES of the projection information may include: the first device extracting the ES of the audio information and / or the ES of the video information. In this way, the first device can accurately extract the ES of the projection information.

[0012] In one possible design, the first device may send the ES to the second device in the following manner: the first device may send the ES of the audio information to the second device through the first audio channel, and / or send the ES of the video information to the second device through the first video channel; the first audio channel is the audio channel corresponding to the call connection between the first device and the second device, and the first video channel is the video channel corresponding to the call connection between the first device and the second device; or, the first device may send the ES of the audio information to the second device through the second audio channel, and / or send the ES of the video information to the second device through the second video channel; the second audio channel is the audio channel used for the shared screen projection information established by the first device and the second device, and the second video channel is the video channel used for the shared screen projection information established by the first device and the second device. In this way, the first device can flexibly select an appropriate method to send the ES to the second device according to actual conditions.

[0013] In one possible design, the first device extracts the original packet time PTS corresponding to the ES of the projection information; the first device records the PTS and sends the PTS to the second device. This allows the projection information transmitted between the first device and the second device to remain synchronized.

[0014] In one possible design, when the first device and the second device establish a call connection, the first device and the second device interact to determine whether they have a first capability, where the first capability is to support the channel corresponding to the call connection and to support channels other than the channel corresponding to the call connection. In this way, based on the capabilities of the first and second devices, it can be determined whether both devices need to negotiate channels other than the channel corresponding to the call connection to transmit screen projection information.

[0015] In one possible design, determining whether the first device and the second device have the first capability for interaction may include: the first device sending a first identifier to the second device, the first identifier being used to indicate whether the first device has the first capability; and the first device receiving a second identifier from the second device, the second identifier being used to indicate whether the second device has the first capability. In this way, whether the first device and the second device have the first capability can be accurately determined.

[0016] In one possible design, the first identifier indicates the number of supported channels, or the first identifier is a preset code corresponding to the first device; the second identifier indicates the number of supported channels, or the second identifier is a preset code corresponding to the second device. This allows for flexible indication of whether the first device or the second device has the first capability.

[0017] In one possible design, before the first device receives the screen projection information from the third device, the first device determines whether the first parameter information and the second parameter information are consistent; the first parameter information is used to share the screen projection information during a call between the first device and the second device; the second parameter information is used for the third device to project the screen to the first device; if so, the first device sends a screen projection instruction to the third device; otherwise, the first parameter information and the second parameter information are renegotiated, and after determining that the renegotiated first parameter information and the second information are consistent, the first device sends a screen projection instruction to the third device. This allows the third device to accurately project the screen to the first device, thereby accurately realizing screen projection sharing between the first and second devices.

[0018] In one possible design, when the first device determines that the first parameter and the second parameter information are inconsistent, the first device records the parameter information supported by the third device. In this way, the first device can subsequently renegotiate the first parameter information based on the parameter information directly provided by the third device.

[0019] In one possible design, when negotiating the first parameter information, the first parameter information can be negotiated based on the second parameter information, and the second parameter information is used for the third device to project the screen to the first device. This can improve the consistency of the negotiated first parameter information and the second parameter information.

[0020] In a second aspect, the present application provides a screen projection sharing method, which can be applied to a second device, a processor, a chip, or a functional module in the second device, etc. Taking the method applied to the second device as an example, the method may include: when a call connection is established between the first device and the second device, the second device receives an elementary stream ES from the first device, the ES being extracted based on the projection information of the third device; and the second device plays the ES.

[0021] Based on this method, the first device can directly extract the ES of the screen projection information and transmit it to the other end, subtracting the time required to decode the recorded screen and then re-encode it, significantly reducing the end-to-end latency of screen projection sharing. This method can also improve image accuracy and enhance the visual experience; it can also prevent the transmission of unexpected content to the other end, improving the user experience.

[0022] In one possible design, first parameter information is negotiated, and the first parameter information is used to share screen projection information during a call between the first device and the second device. The negotiation process of the first parameter information can be initiated by the first device or the second device, which is not limited in this application. In this way, based on the first parameter information, the first device and the second device can subsequently share screen projection during the call.

[0023] In one possible design, the first parameter information may include an audio channel and / or a video channel for the first device and the second device to establish shared screen projection information. In this way, the first device and the second device can transmit screen projection information through a channel other than the call connection channel, so that the first device and the second device can simultaneously maintain a call and share screen projection.

[0024] In one possible design, the screen projection information may include audio information and / or video information; the ES includes the ES of the audio information and / or the ES of the video information.

[0025] In one possible design, the second device may receive the ES from the first device in the following manner: the second device receives the ES of the audio information from the first device through the first audio channel, and / or receives the ES of the video information from the first device through the first video channel; the first audio channel is the audio channel corresponding to the call connection between the first device and the second device, and the first video channel is the video channel corresponding to the call connection between the first device and the second device; or, the second device receives the ES of the audio information from the first device through the second audio channel, and / or receives the ES of the video information from the first device through the second video channel; the second audio channel is the audio channel used for the shared screen projection information established by the first device and the second device, and the second video channel is the video channel used for the shared screen projection information established by the first device and the second device. In this way, the second device can flexibly select an appropriate method to receive the ES from the first device according to actual conditions.

[0026] In one possible design, the second device receives the original packet time PTS corresponding to the ES from the first device. This allows the screen projection information transmitted between the first device and the second device to remain synchronized.

[0027] In one possible design, when the first device and the second device establish a call connection, the second device and the first device interact to determine whether they have a first capability, where the first capability is to support the channel corresponding to the call connection and to support channels other than the channel corresponding to the call connection. In this way, based on the capabilities of the first and second devices, it can be determined whether both devices need to negotiate channels other than the channel corresponding to the call connection to transmit screen projection information.

[0028] In one possible design, determining whether the second device and the first device have the first capability for interaction may include: the second device receiving a first identifier from the first device, the first identifier being used to indicate whether the first device has the first capability; and the second device sending a second identifier to the first device, the second identifier being used to indicate whether the second device has the first capability. In this way, whether the first and second devices have the first capability can be accurately determined.

[0029] In one possible design, the first identifier indicates the number of supported channels, or the first identifier is a preset code corresponding to the first device; the second identifier indicates the number of supported channels, or the second identifier is a preset code corresponding to the second device. This allows for flexible indication of whether the first device or the second device has the first capability.

[0030] In one possible design, negotiating the first parameter information may include: the second device receiving second parameter information from the first device, where the second parameter information is used by the third device to project a screen onto the first device; and negotiating the first parameter information based on the second parameter information. This can improve consistency between the negotiated first and second parameter information.

[0031] In a third aspect, the present application further provides a communication device, which can be applied to a first device and has the function of implementing the method in the first aspect or various possible design examples of the first aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0032] In one possible design, the structure of the communication device may include a communication unit and a processing unit, which can perform the functions of the method in the above-mentioned first aspect or various possible design examples of the first aspect, which are not repeated here.

[0033] In one possible design, the communication device includes a processor and, optionally, a memory and / or a communication interface. The communication interface is used to send and receive data, messages, or information, and to communicate and interact with other devices in the system. The processor is configured to support the communication device in performing the corresponding functions described in the first aspect or various possible design examples of the first aspect. The memory is coupled to the processor and stores program instructions and data necessary for the communication device.

[0034] In a fourth aspect, the present application further provides a communication device, which can be applied to a second device and has the function of implementing the method in the second aspect or various possible design examples of the second aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0035] In one possible design, the structure of the communication device may include a communication unit and a processing unit, which can perform the functions of the method in the above-mentioned second aspect or various possible design examples of the second aspect, which are not repeated here.

[0036] In one possible design, the communication device includes a processor and, optionally, a memory and / or a communication interface. The communication interface is used to send and receive data, messages, or information, and to communicate and interact with other devices in the system. The processor is configured to support the communication device in performing the corresponding functions described in the second aspect or various possible design examples of the second aspect. The memory is coupled to the processor and stores program instructions and data necessary for the communication device.

[0037] In a fifth aspect, embodiments of the present application provide a system that may include a first device, a second device, and a third device. The first device is configured to implement the method described in the first aspect or various possible design examples of the first aspect. The second device is configured to implement the method described in the second aspect or various possible design examples of the second aspect.

[0038] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores program instructions. When the program instructions are run on a computer, the computer executes the method described in the first aspect of the embodiment of the present application and any possible design thereof, or the second aspect and any possible design thereof. Exemplarily, the computer-readable storage medium can be any available medium that can be accessed by a computer. Taking this as an example but not limited to: the computer-readable medium may include a non-transitory computer-readable medium, a random-access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a CD-ROM or other optical disk storage, a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer.

[0039] In the seventh aspect, an embodiment of the present application provides a computer program product, comprising instructions, which, when executed on a computer, causes the method described in the above-mentioned first aspect or any possible design of the first aspect, or the above-mentioned second aspect or any possible design of the second aspect to be executed.

[0040] In an eighth aspect, the present application also provides a chip comprising a processor, wherein the processor is coupled to a memory and is used to read and execute program instructions stored in the memory so that the chip implements the method described in the above-mentioned first aspect or any possible design of the first aspect, or the above-mentioned second aspect or any possible design of the second aspect.

[0041] For each of the above-mentioned aspects from the third to the eighth aspect and the technical effects that may be achieved by each of the aspects, please refer to the above-mentioned description of the technical effects that can be achieved by the first aspect or the various possible solutions in the first aspect, or the above-mentioned second aspect or the various possible solutions in the second aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] FIG1 is a schematic diagram of the architecture of a system provided by this application;

[0043] FIG2 is a flow chart of a screen projection sharing method provided by the present application;

[0044] FIG3 is a schematic diagram of a screen sharing scenario of a set-top box supporting multiple channels provided by the present application;

[0045] FIG4 is a schematic diagram of a screen projection sharing effect provided by this application;

[0046] FIG5 is a flowchart of an example of a screen projection sharing method provided by the present application;

[0047] FIG6 is a schematic diagram of maintaining synchronization in a screen projection and sharing method provided by the present application;

[0048] FIG7 is a schematic diagram of a screen sharing scenario of a set-top box that does not support multi-channel provided by the present application;

[0049] FIG8 is a flowchart of an example of another screen projection sharing method provided by the present application;

[0050] FIG9 is a schematic diagram of a process for reminding a user to re-project the screen when projection parameters and shared projection parameters are inconsistent in a multi-channel scenario provided by the present application;

[0051] FIG10 is a schematic diagram of a process for reminding a user to re-project the screen when the projection parameters and the shared projection parameters are inconsistent in a single-channel scenario provided by the present application;

[0052] FIG11 is a flowchart of an example of a renegotiation process in which projection parameters and shared projection parameters are inconsistent, provided by the present application;

[0053] FIG12 is a flowchart of an example of using other screen projection methods when screen projection parameters and shared screen projection parameters are inconsistent with each other provided by the present application;

[0054] FIG13 is a schematic diagram of a scenario in which a mobile phone casting a screen to a first set-top box can trigger call media renegotiation, as provided in the present application;

[0055] FIG14 is an example of a process for consistent parameters when screen projection from a mobile phone to a first set-top box can trigger call media renegotiation provided by the present application;

[0056] FIG15 is an example of a parameter inconsistency process provided by the present application when screen projection from a mobile phone to a first set-top box can trigger call media renegotiation;

[0057] FIG16 is a schematic structural diagram of a communication device provided by the present application;

[0058] FIG17 is a structural diagram of a communication device provided in this application. DETAILED DESCRIPTION

[0059] The present application will be described in further detail below with reference to the accompanying drawings.

[0060] The embodiments of the present application provide a screen projection sharing method and device for implementing screen projection sharing. The method and device described in this application are based on the same technical concept. Since the method and device solve the problem in a similar way, the implementation of the device and method can refer to each other, and the repeated parts will not be repeated.

[0061] In the description of this application, words such as "first" and "second" are only used for the purpose of distinguishing the description and cannot be understood as indicating or implying relative importance or order.

[0062] In the description of this application, "at least one" means one or more, and "more" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c can be single or plural.

[0063] In the description of this application, "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. " / " means "or", for example, a / b means a or b.

[0064] In order to more clearly describe the technical solution of the embodiment of the present application, the screen projection sharing method and device provided in the embodiment of the present application are described in detail below with reference to the accompanying drawings.

[0065] Figure 1 illustrates an exemplary architecture of a possible system applicable to the screen projection sharing method of the present application. The architecture of the system may include a first device, a second device, and a third device. Among them, the first device and the second device can establish a call connection. When a call connection is established between the first device and the second device, the screen projection information of the third device projecting the screen to the first device can be shared with the second device. That is, the third device is the screen projection source (source) and the first device is the screen projection receiving end (sink).

[0066] The first device, the second device, and the third device may be terminal devices. The first device and the second device may be terminal devices with a call function. The first device may be a terminal device with a screen projection function.

[0067] For example, a terminal device, which may also be referred to as user equipment (UE), mobile station (MS), or mobile terminal (MT), is a device that provides voice and / or data connectivity to a user. For example, a terminal device may include a handheld device or vehicle-mounted device with wireless connectivity. Currently, terminal devices can be: mobile phones, tablets, laptops, PDAs, PADs, mobile internet devices (MIDs), wearable devices, home smart screens, set-top boxes, virtual reality (VR) devices, augmented reality (AR) devices, extended reality (XR) devices, mixed reality (MR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, or wireless terminals in smart homes, etc.

[0068] The terminal device can also be a device-to-device communication (D2D) terminal device, a vehicle to everything (V2X) communication terminal device, an intelligent vehicle, a vehicle-to-vehicle system (or a vehicle network system) (telematics box, TBOX), a machine-to-machine / machine-type communication (M2M / MTC) terminal device, or an Internet of Things (IoT) terminal device.

[0069] It should be understood that among the various examples of terminal devices described above, only some may be used as the first device, the second device, or the third device. In other words, the first device, the second device, or the third device may be part of the terminal devices in the aforementioned examples. For example, the first device may be a set-top box, the second device may be a mobile phone, and the third device may be a mobile phone; or the first device may be a set-top box, the second device may be a set-top box, and the third device may be a PAD.

[0070] Among them, the set-top box can be a device that is combined with the display screen, or it can be a device independent of the display screen. When the set-top box is independent of the display screen, the set-top box is connected to the display screen to jointly realize calls with other devices or share screen projection information.

[0071] Currently, screen sharing uses a screen capture and recording method, which requires decoding, rendering, and encoding the screen data, resulting in a long end-to-end data processing chain and a large delay; re-encoding the rendered image will result in a loss of video quality; if a pop-up window that the user does not expect appears in the screen recording area, the content that the user does not expect will be transmitted to the other end, resulting in a poor experience. Based on this, the embodiment of the application provides a screen sharing method, in which the first device can directly extract the elementary stream (elementary stream, ES) of the screen information and transmit it to the other end, subtracting the time for decoding, recording, and re-encoding, greatly reducing the end-to-end delay of screen sharing. In addition, this method can improve the picture accuracy and the picture experience effect; at the same time, it can avoid the transmission of content that the user does not expect to the other end, thereby improving the user experience.

[0072] Based on the above description, an embodiment of the present application provides a screen projection sharing method. Referring to FIG2 , the method may include the following steps:

[0073] Step 201: When a call connection is established between the first device and the second device, the first device obtains first instruction information, which is used to indicate shared screen projection information.

[0074] In some embodiments, the first device may be a set-top box, the second device is the call partner of the first device, and the third device may be a mobile phone or tablet that projects its screen onto the first device. Optionally, the first device and the third device may correspond to the same user.

[0075] Of course, the above is only an example and is not intended to limit the present application.

[0076] Here, the call connection can be understood as a connection including a video call and a voice call.

[0077] The first device or the second device may call the other device to establish a call connection between the first device and the second device. In some embodiments, when the first device and the second device establish a call connection, the first device and the second device may exchange whether they have a first capability, where the first capability is supporting a channel corresponding to the call connection and supporting a channel other than the channel corresponding to the call connection.

[0078] Optionally, whether the first device and the second device have the first capability can also be understood as whether the first device and the second device support multi-channel. Among them, supporting multi-channel means supporting more than one video channel and one audio channel. In other words, the channels corresponding to the call connection include one video channel and one audio channel. If the first device and the second device support multi-channel (that is, have the first capability), the first device and the second device can support at least one video channel and / or at least one audio channel in addition to the call connection.

[0079] If the first device and the second device do not support multi-channel (ie, do not have the first capability), it can be understood that the first device and the second device support single channel, that is, the first device and the second device only support one video channel and one audio channel.

[0080] In an optional embodiment, the first device and the second device may communicate whether they have the first capability by: the first device sends a first identifier to the second device, the first identifier being used to indicate whether the first device has the first capability; and the first device receives a second identifier from the second device, the second identifier being used to indicate whether the second device has the first capability. The first identifier or the second identifier may also be understood as an identifier of whether multi-channel support is supported.

[0081] Exemplarily, the first identifier may be an indication of the number of supported channels, or the first identifier may be a preset code corresponding to the first device; the second identifier may be an indication of the number of supported channels, or the second identifier may be a preset code corresponding to the second device.

[0082] The preset code can be used to determine whether the device has the first capability (i.e., whether it supports multi-channel). Optionally, the preset code corresponding to the first device can be the number corresponding to the first device for being called, etc., such as the communication number assigned by the operator. For example, the communication number of the first device can be the area code of the region where the first device is located plus a fixed number with A digits, A is a positive integer, for example, A can be 8, etc., which is not limited in this application. The preset code corresponding to the second device can be the number corresponding to the second device for being called, etc., such as the communication number assigned by the operator. For example, the communication number of the second device can be the area code of the region where the second device is located plus a fixed number with B digits, B is a positive integer, for example, B can be 8, etc., which is not limited in this application.

[0083] In one possible approach, the first device may also directly determine whether the second device has the first capability based on the number format of the number it calls (here, also the number format of the second device). The second device may also directly determine whether the first device has the first capability based on the number format of the number it calls (here, also the number format of the first device).

[0084] The interaction between the first device and the second device when establishing a call connection can be understood as the first device and the second device performing call media negotiation. It should be understood that during the call media negotiation, the first device and the second device can negotiate other call parameters in addition to whether the interaction has the first capability, and this application does not limit this.

[0085] Furthermore, after the call between the first device or the second device is successful, a call connection is established between the first device and the second device, and a call video stream and / or a call audio stream can be transmitted between the first device and the second device.

[0086] In some embodiments, the first device may obtain the first instruction information in the following two ways:

[0087] Mode a1: The user of the first device may trigger the first instruction information through the first device, so that the first device obtains the first instruction information.

[0088] For example, the user of the first device can trigger the screen projection sharing function through the user interface corresponding to the first device, that is, trigger the first instruction information.

[0089] The user interface corresponding to the first device may be displayed on a display screen provided in the first device, or may be displayed on a display screen connected to the first device.

[0090] Method a2: The third device initiates screen projection to the first device, triggering the first instruction information so that the first device obtains the first instruction information.

[0091] In method a2, when the user of the third device sees something that he wants to share with the user of the second device, the user can initiate screen projection to the first device through the third device, thereby triggering the first instruction information.

[0092] Step 202: The first device receives screen projection information from the third device based on the first instruction information, and accordingly, the third device sends the screen projection information to the first device.

[0093] In some embodiments, before the first device receives the projection information from the third device, the first device and the second device may negotiate first parameter information, where the first parameter information is used to share the projection information during a call between the first and second devices. The negotiation of the first parameter information may be initiated by either the first device or the second device, and this application does not limit this.

[0094] It should be understood that the negotiation described in this application is the process of exchanging information between two devices, and the specific negotiation process is not limited in this application. Specifically, during the negotiation process between two devices, this application does not limit the initiating device of the negotiation process, that is, either device can initiate the negotiation process.

[0095] The above-mentioned process of negotiating the first parameter information can also be understood as shared screen projection media negotiation, or can also be understood as call media re-negotiation.

[0096] In one possible implementation, when the first device and the second device support multiple channels, the first parameter information may include the audio channel and / or video channel used by the first device and the second device to establish shared screen projection information. That is, when the first device and the second device support multiple channels, the first device and the second device can negotiate channels other than those included in the call connection to use different channels to transmit the projected content and the content of the current call.

[0097] Optionally, the media parameters of the audio channel and / or video channel used for sharing the screen projection information can adopt preset parameters or commonly used screen projection parameters.

[0098] In some possible ways, when the first device and the second device do not support multiple channels, the first device and the second device can directly reuse the channels included in the call connection as audio channels and / or video channels for sharing screen projection information. Therefore, the first device and the second device may not negotiate the first parameter information.

[0099] Optionally, when the first device and the second device do not support multi-channel, if the first device or the second device predicts that the projection parameters are inconsistent with the current call parameters, first parameter information can be negotiated. The first parameter information may include commonly used parameters for projection or preset parameters.

[0100] In some embodiments, before the first device receives the projection information from the third device, the first device and the third device may negotiate second parameter information, where the second parameter information is used for the third device to project the screen to the first device. The negotiation process for the second parameter information may be initiated by either the first device or the third device, and this application does not limit this.

[0101] The above process of negotiating the second parameter information can be understood as screen projection media negotiation.

[0102] Optionally, the second parameter information may include encoding format, resolution, etc.

[0103] In an optional embodiment, before the first device receives the screen projection information from the third device, the first device determines whether the first parameter information and the second parameter information are consistent; if so, the first device sends a screen projection instruction to the third device; otherwise, the first device renegotiates the first parameter information with the second device and the first device renegotiates the second parameter information with the third device, and after determining that the renegotiated first parameter information and the second information are consistent, the first device sends a screen projection instruction to the third device.

[0104] Optionally, when the first device determines that the first parameter information is inconsistent with the second parameter information, the first device may record parameter information supported by the third device.

[0105] In an example, if the first device records parameter information supported by the third device (which can be understood as second parameter information), the first device can negotiate or renegotiate the first parameter information with the second device based on the second parameter information.

[0106] In some embodiments, the screen projection information includes audio information and / or video information. Accordingly, the third device sends the screen projection information to the first device, which may be the third device sending audio information (or audio stream) and / or video information (or video stream) to the first device.

[0107] Step 203: The first device extracts the ES of the screen projection information.

[0108] Since the screen projection information includes audio information and / or video information, the ES of the screen projection information extracted by the first device may be the ES of the audio information and / or the ES of the video information extracted by the first device.

[0109] For example, ES may be data encoded using H264 / H265 video or advanced audio coding (AAC) audio.

[0110] Step 204: The first device plays the ES and sends the ES to the second device.

[0111] In some embodiments, the first device playing the ES may be the first device playing the ES on the first device. In some other embodiments, the first device playing the ES may be downloading the ES to the first device in some applications (APP) for playback. In some other embodiments, the first device playing the ES may also be the first device playing the ES through the APP of the first device. Of course, there may be other playback situations, which are not limited in this application. The same applies to the playback of other devices in this application, and the following embodiments of this application will not be described one by one.

[0112] In some embodiments, the first device sending the ES to the second device may include the following two situations:

[0113] Case b1: the first device sends the ES of the audio information to the second device through the first audio channel, and / or sends the ES of the video information to the second device through the first video channel.

[0114] The first audio channel is an audio channel corresponding to the call connection between the first device and the second device, and the first video channel is a video channel corresponding to the call connection between the first device and the second device.

[0115] In this situation b1, the first device and the second device do not have the first capability (that is, they do not support multiple channels and only support single channels), and the first device and the second device reuse the channel of the call connection to transmit the projection information.

[0116] Case b2: The first device sends the ES of the audio information to the second device through the second audio channel, and / or sends the ES of the video information to the second device through the second video channel.

[0117] Among them, the second audio channel is the audio channel used for the shared screen projection information established by the first device and the second device, and the second video channel is the video channel used for the shared screen projection information established by the first device and the second device.

[0118] In this case b2, the first device and the second device have the first capability (i.e., support multiple channels), and the first device and the second device can use channels other than the call connection channel and used to share the projection information to transmit the projection information.

[0119] In an optional embodiment, the first device may further extract the original package time (PTS) corresponding to the ES of the projection information; the first device records the PTS and sends the PTS to the second device. In this way, the PTS can be used as the timestamp of the corresponding projection information to keep the projection information synchronized between the first device and the second device.

[0120] Exemplarily, the first device may extract the PTS corresponding to the ES of the audio information and / or the PTS corresponding to the ES of the video information, and send the PTS corresponding to the ES of the audio information and / or the PTS corresponding to the ES of the video information to the second device.

[0121] In some embodiments, the first device may encapsulate the ES of the audio information and the PTS corresponding to the ES of the audio information and send them together, or may send them separately. The same is true for video information, which is not limited in this application.

[0122] Step 205: The second device plays the ES.

[0123] Based on the above screen sharing method, the first device can directly extract the ES of the screen projection information and transmit it to the other end, subtracting the time for decoding, recording and re-encoding the screen, greatly reducing the end-to-end latency of screen sharing. This method can also improve image accuracy and enhance the visual experience; at the same time, it can prevent the transmission of unexpected content to the other end, thereby improving the user experience.

[0124] In some embodiments, if the first parameter information and the second information are inconsistent, in order to ensure successful screen sharing, the first device may also initiate a common screen recording or transcoding process. The inconsistency between the first parameter information and the second information may be a negotiation inconsistency or a re-negotiation inconsistency, which is not limited in this application.

[0125] Based on the above embodiments, the screen projection sharing method provided by the embodiments of the present application is described in detail below through specific scenarios and examples. It should be understood that the following examples are only exemplary descriptions and are not intended to limit the present application. In the following scenarios and examples, the first device is a set-top box (hereinafter referred to as the first set-top box as an example), the second device is a set-top box (hereinafter referred to as the second set-top box as an example), and the third device is a mobile phone as an example.

[0126] Exemplarily, FIG3 shows a screen sharing scenario of a set-top box that supports multiple channels. When a call connection is established between the first set-top box and the second set-top box, when the user triggers a screen sharing call, the first set-top box and the second set-top box perform a shared screen media negotiation, and can negotiate to obtain the multiple channels used for the shared screen information, audio and video media parameters (such as the parameters corresponding to the highest resolution on the call side), etc. The mobile phone initiates screen casting to the first set-top box, and the first set-top box and the mobile phone perform screen casting media negotiation, and the screen casting parameters use the parameters negotiated by the above-mentioned shared screen casting media. Afterwards, the mobile phone can send screen casting information to the first set-top box, and the first set-top box extracts the ES stream of the screen casting audio and video information, plays it at the same time, and transmits it to the second set-top box on the other end through the corresponding channels. In this way, the screen stream decoding and rendering of the screen casting and then the screen recording encoding link can be omitted, reducing the end-to-end (E2E) delay. The screen of the first set-top box and the audio and video stream of the call can be transmitted to the second set-top box through multiple channels, as shown in FIG4. This achieves the following: Through ES transparent transmission and four-way channel transmission (screen projection information and call audio and video), the shared desktop of the mobile phone, the image captured by the camera of the first set-top box, and the image captured by the camera of the second set-top box can be displayed simultaneously. The second set-top box on the other end of the call can hear the screen projection audio stream and the call audio stream at the same time, with clear image quality and low latency. In Figure 4, A and B represent the users of the first and second set-top boxes.

[0127] Based on the scenario shown in FIG3 , FIG5 shows an example of a screen projection sharing method. The process of this example may include the following steps:

[0128] S501: A user of a first set-top box or a second set-top box makes a call, and the first set-top box and the second set-top box perform call media negotiation.

[0129] As shown in FIG6 , in the call media negotiation message, the first set-top box and the second set-top box may respectively carry an identification of whether they support multiple channels, so as to determine whether they support multiple channels (supporting more than one video channel and one audio channel) through the identification; or, determine whether they support multiple channels through the call number format.

[0130] For example, whether the identifier of supporting multiple channels can refer to the related description of the first identifier and the second identifier described in the above embodiment, which will not be described in detail here.

[0131] In this example, the first set-top box and the second set-top box support multi-channel.

[0132] S502: After the call is successful, the call video stream is transmitted between the first set-top box and the second set-top box.

[0133] S503: After the call is successful, the call audio stream is transmitted between the first set-top box and the second set-top box.

[0134] It should be understood that the order of S502 and S503 is only an example and is not limited in this application.

[0135] S504: The user of the first set-top box triggers the screen sharing function on the user interface corresponding to the first set-top box, and the first set-top box and the second set-top box perform shared screen sharing media negotiation (or call media renegotiation).

[0136] During the shared screen projection media negotiation process, the first set-top box and the second set-top box can negotiate the audio channel and / or video channel required for the newly added shared screen projection information. The media parameters of this audio channel and / or video channel can use preset parameters or commonly used parameters for screen projection.

[0137] After the shared screen projection media negotiation is completed, the user interface of the first set-top box can be adjusted to the screen to be projected, reminding the user that the screen projection can be initiated.

[0138] S505: The mobile phone user initiates screen projection through the mobile phone, and the mobile phone and the first set-top box can negotiate the screen projection media.

[0139] For example, a user can initiate a mirror screen projection (such as wireless screen projection (miracast) or Cast+ (Cast+ is a screen projection protocol)) through a mobile phone.

[0140] Optionally, the main media parameters used in the screen projection media negotiation process may be parameters (such as encoding format, resolution, etc.) negotiated by the first set-top box and the second set-top box in the shared screen projection media negotiation process.

[0141] S506: The screen projection media negotiation is successful, and the first set-top box can send a screen projection instruction to the mobile phone to notify the mobile phone that the screen projection information (such as audio information and video information) can be transmitted.

[0142] S507: The mobile phone transmits the projection video information to the first set-top box.

[0143] For example, the screen projection video information can be the screen video stream of a mobile phone.

[0144] S508: The mobile phone transmits the screen projection audio information to the first set-top box.

[0145] For example, the screen projection audio information can be the screen audio stream of a mobile phone.

[0146] It should be understood that the order of S507 and S508 is only an example and is not limited in this application.

[0147] S509: The first set-top box extracts the ES of the projection video information, plays the ES of the projection video information, extracts the PTS corresponding to the ES of the projection video information, and packages the ES and corresponding PTS of the projection video information according to the transmission format of the call channel.

[0148] Optionally, the first set-top box may also extract other original data of the ES containing the projection video information, such as a packetized elementary stream (PES), a transport stream (TS), etc.

[0149] S510: The first set-top box sends the ES and corresponding PTS of the projection video information packaged according to the transmission format of the call channel to the second set-top box.

[0150] Among them, the first set-top box sends the ES and corresponding PTS of the projection video information through the negotiated video channel for sharing the projection information.

[0151] S511: The first set-top box extracts the ES of the projection audio information, plays the ES of the projection audio information, extracts the PTS corresponding to the ES of the projection audio information, and packages the ES and corresponding PTS of the projection audio information according to the transmission format of the call channel.

[0152] Optionally, the first set-top box can also extract other original data of ES containing the projection audio information, such as PES, TS, etc.

[0153] S512: The first set-top box sends the ES and corresponding PTS of the projection audio information packaged according to the transmission format of the call channel to the second set-top box.

[0154] Among them, the first set-top box sends the ES and corresponding PTS of the projection audio information through the negotiated audio channel for sharing the projection information.

[0155] Among them, the execution of S509 and S511, that is, the operation of the first set-top box as shown in Figure 6, records the corresponding PTS while extracting the ES of the projection audio information and the projection video information, and packages the corresponding PTS into the transmission packet of the channel sharing the projection information as the timestamp of the source data, which can keep the projection information synchronized between the first set-top box and the second set-top box.

[0156] It should be understood that the aforementioned order of S509-S510 and S511-S512 is only an example and is not limited in this application.

[0157] S513: The second set-top box plays the ES of the screen-casting audio information and the ES of the screen-casting video information.

[0158] For example, Figure 7 shows a screen sharing scenario for a set-top box that does not support multiple channels. When a call connection is established between a first set-top box and a second set-top box, the first set-top box and the second set-top box can directly reuse or renegotiate (pre-determine that the screen projection parameters and call parameters are inconsistent and initiate negotiation in advance) the channel for the call connection. The first set-top box can send the screen projection audio and video stream to the second set-top box, or the screen projection video stream and the call audio stream.

[0159] Based on the scenario shown in FIG7 , FIG8 shows an example of a screen projection sharing method. The process of this example may include the following steps:

[0160] S801 is similar to S501 and can be referred to each other, so they will not be described in detail here. Unlike S501, in S801, the result of the judgment can be that the first set-top box can support multi-channel, while the second set-top box does not support multi-channel, that is, supports single channel. Alternatively, both the first set-top box and the second set-top box do not support multi-channel.

[0161] S802-S803 are similar to S502-S503, and can be referred to each other, so they will not be repeated here.

[0162] S804: The user of the first set-top box triggers the screen sharing function in the user interface corresponding to the first set-top box, and the first set-top box and the second set-top box determine that a channel for the call connection can be directly reused. Optionally, as shown in S804 in Figure 8, the first set-top box and the second set-top box may perform a shared screen sharing media negotiation (or call media renegotiation).

[0163] Optionally, the first set-top box or the second set-top box can execute the shared screen projection media negotiation process when it is predicted that the screen projection parameters and the call parameters are inconsistent. For example, the first set-top box and the second set-top box can initiate shared screen projection media negotiation using commonly used parameters for screen projection.

[0164] After the shared screen projection media negotiation is completed, or when it is determined to adopt the direct multiplexing method, the user interface of the first set-top box can be adjusted to the screen to be projected, reminding the user that the screen projection can be initiated.

[0165] S805-S808 are similar to S505-S508, and can be referenced to each other, so they will not be repeated here.

[0166] S809: The first set-top box extracts the ES of the projection video information, plays the ES of the projection video information, extracts the PTS corresponding to the ES of the projection video information, and packages the ES and corresponding PTS of the projection video information according to the transmission format of the call channel.

[0167] Optionally, the first set-top box may also extract other original data of ES containing projection video information, such as PES, TS, etc.

[0168] S810: The first set-top box sends the ES and corresponding PTS of the projection video information packaged according to the transmission format of the call channel to the second set-top box.

[0169] Among them, the first set-top box sends the ES and corresponding PTS of the projection video information through the video channel of the multiplexed call connection.

[0170] S811: The first set-top box extracts the ES of the projection audio information, and extracts the PTS corresponding to the ES of the projection audio information, and packages the ES and corresponding PTS of the projection audio information according to the transmission format of the call channel.

[0171] Optionally, the first set-top box can also extract other original data of ES containing the projection audio information, such as PES, TS, etc.

[0172] S812: The first set-top box plays the audio stream and sends the audio stream to the second set-top box. The audio stream may include the ES and corresponding PTS of the projection audio information packaged in the transmission format of the call channel, or the audio stream may include the audio data of the call connection, or the audio stream may include the mixed audio data of the ES of the projection audio information and the audio data of the call connection.

[0173] Optionally, the user interface of the first set-top box can display prompt information for the user to select. The prompt information may include audio data of the call connection, screen projection audio information, or mixed audio data of the foregoing two. The first set-top box can determine the specific content of the audio stream based on the user's selection instructions.

[0174] The first set-top box sends the audio stream through the audio channel of the multiplexed call connection.

[0175] S813: The second set-top box plays the ES and audio stream of the projection video information.

[0176] In some embodiments, the parameters used by the mobile phone and the set-top box (such as encoding format, resolution information, etc.) can be negotiated. However, the call media parameters (i.e., shared screen projection parameters) of the shared screen projection information negotiated between the first set-top box and the second set-top box (i.e., the first parameter information in the aforementioned description) may be inconsistent with the screen projection parameters negotiated between the first set-top box and the mobile phone (i.e., the second parameter information in the aforementioned description). Figure 9 shows a schematic diagram of the process of reminding the user to re-project the screen when the projection parameters and the shared screen projection parameters are inconsistent in the scenario where the first set-top box and the second set-top box support multiple channels. In the scenario where multiple channels are supported, the shared screen projection media negotiation is completed first. The first set-top box negotiates the screen projection with the mobile phone based on the negotiated shared screen projection media parameters. If the parameters are inconsistent, the negotiation fails, and the user needs to be reminded that the parameters are inconsistent and the screen projection needs to be re-projected. The first set-top box can record the parameters supported by the mobile phone, and based on the parameters supported by the mobile phone, the shared screen projection media is renegotiated. After the negotiation is completed, the user is reminded that the status is accurate (ready), and the user projects the screen again to complete the screen projection media negotiation. Figure 10 shows a schematic diagram of the process of reminding the user to re-project when the projection parameters and shared projection parameters are inconsistent in the scenario where the first set-top box and the second set-top box support a single channel. In the scenario where single-channel is supported, compared with the scenario where multiple channels are supported, the channel for sharing projection information may reuse the channel for call connection, so it is optional to conduct shared projection media negotiation before projection (pre-judging that the projection parameters and call parameters will be inconsistent, you can initiate negotiation in advance), and other processes are consistent with the multi-channel scenario.

[0177] The following describes possible situations when parameters are inconsistent through different examples.

[0178] FIG11 shows an example of a renegotiation process when screen projection parameters and shared screen projection parameters are inconsistent. The process of this example may include the following steps:

[0179] S1101: The user of the first set-top box triggers the screen sharing function in the user interface corresponding to the first set-top box, and the first set-top box and the second set-top box perform shared screen sharing media negotiation (or call media renegotiation), or, in a single-channel scenario, the first set-top box and the second set-top box reuse the call channel.

[0180] In FIG. 11 , S1101 is exemplarily illustrated by the first set-top box and the second set-top box performing shared screen projection media negotiation.

[0181] Among them, the specific content of shared screen projection media negotiation can be found in the relevant description involved in the above embodiment, and will not be described in detail here.

[0182] S1102: The mobile phone user initiates screen projection through the mobile phone, and the mobile phone and the first set-top box can negotiate the screen projection media.

[0183] For example, a user can initiate a mirrored screen projection (such as miracast or Cast+) through a mobile phone.

[0184] Optionally, the main media parameters used in the screen projection media negotiation process may be parameters (such as encoding format, resolution, etc.) negotiated by the first set-top box and the second set-top box in the shared screen projection media negotiation process.

[0185] S1103: The screen projection media negotiation between the mobile phone and the first set-top box is unsuccessful, that is, the parameter negotiation fails. The first set-top box records the parameters supported by the mobile phone.

[0186] Among them, the user interface corresponding to the first set-top box can remind the user that the screen projection parameters and the shared screen projection parameters are inconsistent and the screen projection needs to be re-performed.

[0187] S1104: The first set-top box and the second set-top box may perform shared media re-negotiation (ie, call media re-negotiation).

[0188] The first set-top box and the second set-top box may negotiate based on parameters supported by the mobile phone during re-negotiation.

[0189] After the shared media is re-negotiated successfully, the user interface of the first set-top box can remind the user that screen projection is available.

[0190] S1105: The first set-top box and the mobile phone negotiate the screen projection media again, and the parameters of the screen projection media negotiation are consistent with the parameters of the shared screen projection media re-negotiation (such as the encoding format, resolution, etc.).

[0191] S1106: The first set-top box and the mobile phone successfully negotiate the screen projection media again. The first set-top box can send a screen projection instruction to the mobile phone to notify the mobile phone that it can transmit the screen projection audio and video information.

[0192] FIG12 shows an example of using other screen projection methods (such as screen recording or transcoding) to project the screen when the projection parameters and shared screen projection parameters are inconsistent. The process of this example may include the following steps:

[0193] S1201-S1203 are similar to S1101-S1103, and can be referred to each other, so they will not be repeated here.

[0194] S1204: The first set-top box determines that the parameters supported by the mobile phone are incompatible with the parameters supported by the call media (i.e., shared screen projection parameters), starts the screen recording or transcoding function, and enters the screen projection state, and the user interface of the first set-top box can remind the user that the screen can be projected.

[0195] S1205: The mobile phone user initiates screen projection through the mobile phone, and the mobile phone and the first set-top box can negotiate the screen projection media again.

[0196] For example, a user can initiate a mirrored screen projection (such as miracast or Cast+) through a mobile phone.

[0197] It should be understood that the screen projection media negotiation in S1205 can refer to the negotiation process of the screen projection scenario under the current screen recording or transcoding function, and will not be described in detail here.

[0198] S1206: The screen projection media negotiation between the mobile phone and the first set-top box is successful. The first set-top box can send a screen projection instruction to the mobile phone to notify the mobile phone to transmit the screen projection information (such as audio information and video information).

[0199] S1207: The mobile phone transmits the projection video information to the first set-top box.

[0200] For example, the screen projection video information can be the screen video stream of a mobile phone.

[0201] S1208: The mobile phone transmits the screen projection audio information to the first set-top box.

[0202] For example, the screen projection audio information can be the screen audio stream of a mobile phone.

[0203] It should be understood that the order of S1207 and S1208 is only an example and is not limited in this application.

[0204] S1209: The first set-top box records or transcodes the projection video information and the projection audio information.

[0205] S1210: The first set-top box packages the recorded (or transcoded) screen projection video information according to the transmission format of the call channel, and sends the packaged recorded (or transcoded) screen projection video information to the second set-top box.

[0206] S1211: The first set-top box packages the recorded (or transcoded) screen projection audio information according to the transmission format of the call channel, and sends the packaged recorded (or transcoded) screen projection audio information to the second set-top box.

[0207] In some embodiments, screen projection by the mobile phone to the first set-top box can trigger call media re-negotiation (i.e., shared screen projection media negotiation). As shown in the scenario in Figure 13, the user initiates screen projection through the mobile phone, and the mobile phone and the first set-top box conduct screen projection media negotiation, carrying media parameters (such as encoding format, resolution, etc.). The first set-top box does not respond to the supported media parameters for the time being, and instructs the mobile phone to wait for screen projection media negotiation. The first set-top box conducts call media re-negotiation (i.e., shared screen projection media negotiation) based on the media parameters supported by the mobile phone. After the negotiation is completed, the first set-top box replies to the mobile phone with the supported media parameters, and the screen projection media negotiation completes the remaining process, such as the process of stage 2 in the figure.

[0208] Based on the scenario shown in FIG13 , FIG14 shows an example of a process with consistent parameters when screen projection from a mobile phone to a first set-top box can trigger call media renegotiation. The process of this example may include the following steps:

[0209] S1401-S1403 are similar to the aforementioned S501-S503, and can be referred to each other, so they will not be repeated here.

[0210] S1404: The mobile phone user initiates screen projection through the mobile phone, and the mobile phone and the first set-top box conduct screen projection media negotiation. The screen projection media negotiation between the mobile phone and the first set-top box at this time is recorded as stage 1 of the screen projection media negotiation process.

[0211] For example, a user can initiate a mirrored screen projection (such as miracast or Cast+) through a mobile phone.

[0212] Optionally, the mobile phone and the first set-top box can negotiate screen projection media based on media parameters such as encoding format and resolution.

[0213] In phase 1 of the screen projection media negotiation process, the first set-top box does not respond to the supported media parameters and waits for negotiation.

[0214] S1405: The first set-top box records the media parameters supported by the mobile phone.

[0215] S1406: The first set-top box sends a screen projection media negotiation timeout waiting message to the mobile phone, so that the mobile phone waits for a response from the first set-top box.

[0216] S1406 is an optional step.

[0217] S1407: The first set-top box and the second set-top box perform call media re-negotiation (i.e., shared screen projection media negotiation) based on the media parameters supported by the mobile phone.

[0218] Optionally, if the first set-top box and the second set-top box support multiple channels, a new channel needs to be negotiated; if the first set-top box and the second set-top box support single channel, if the screen projection parameters and call parameters are consistent, they can be reused directly, otherwise the screen projection parameters and call parameters need to be renegotiated.

[0219] S1408: The first set-top box replies with the supported media parameters to the mobile phone, and the mobile phone and the first set-top box complete the screen projection media negotiation and complete the remaining process. At this time, the screen projection media negotiation between the mobile phone and the first set-top box is recorded as stage 2 of the screen projection media negotiation process.

[0220] S1409: The screen projection media negotiation between the mobile phone and the first set-top box is successful. The first set-top box can send a screen projection instruction to the mobile phone to notify the mobile phone that it can transmit screen projection information (such as audio information and video information).

[0221] Based on the scenario shown in FIG13 , FIG15 shows an example of a process for parameter inconsistency when screen projection from a mobile phone to a first set-top box can trigger call media renegotiation. The process of this example may include the following steps:

[0222] S1501-S1506 are similar to S1401-S1406, and can be referenced to each other, so they will not be repeated here.

[0223] S1507: The first set-top box and the second set-top box perform call media re-negotiation (i.e., shared screen projection media negotiation) based on the media parameters supported by the mobile phone. The re-negotiation fails, i.e., the screen projection parameters and the shared screen projection parameters are inconsistent.

[0224] S1508: The first set-top box and the second set-top box renegotiate the call media based on the media parameters supported by the second set-top box, and the negotiation is successful.

[0225] S1509: The first set-top box starts the screen recording or transcoding function and enters the screen projection state.

[0226] S1510: The first set-top box sends supported media parameters to the mobile phone, and the mobile phone and the first set-top box complete the remaining process of the screen projection media negotiation. At this time, the screen projection media negotiation between the mobile phone and the first set-top box is recorded as stage 2 of the screen projection media negotiation process.

[0227] S1511: The screen projection media negotiation is successful, and the first set-top box can send a screen projection instruction to the mobile phone to notify the mobile phone that the screen projection information (such as audio information and video information) can be transmitted.

[0228] S1512: The mobile phone transmits the projection video information to the first set-top box.

[0229] For example, the screen projection video information can be the screen video stream of a mobile phone.

[0230] S1513: The mobile phone transmits the screen projection audio information to the first set-top box.

[0231] For example, the screen projection audio information can be the screen audio stream of a mobile phone.

[0232] It should be understood that the order of S1512 and S1513 is only an example and is not limited in this application.

[0233] S1514: The first set-top box records or transcodes the projection video information and the projection audio information.

[0234] S1515: The first set-top box packages the recorded (or transcoded) screen projection video information according to the transmission format of the call channel, and sends the packaged recorded (or transcoded) screen projection video information to the second set-top box.

[0235] S1516: The first set-top box packages the recorded (or transcoded) screen projection audio information according to the transmission format of the call channel, and sends the packaged recorded (or transcoded) screen projection audio information to the second set-top box.

[0236] Based on the above embodiments, the present application also provides a communication device. Referring to FIG16 , the communication device 1600 may include a communication unit 1601 and a processing unit 1602. The communication unit 1601 is used for the communication device 1600 to communicate, such as receiving information (messages or data) or sending information (messages or data), and the processing unit 1602 is used to control and manage the actions of the communication device 1600. The processing unit 1602 may also control the steps performed by the communication unit 1601.

[0237] Exemplarily, the communication device 1600 may specifically be the first device in the above embodiment, a processor of the first device, or a chip, or a chip system, or a functional module, etc. Alternatively, the communication device 1600 may specifically be the second device in the above embodiment, a processor, or a chip, or a chip system, or a functional module, etc. in the second device.

[0238] In one embodiment, when the communication device 1600 is used to implement the function of the first device in the above embodiment, the communication unit 1601 can be used to obtain first instruction information when a call connection is established between the first device and the second device, and the first instruction information is used to indicate the sharing of screen projection information; and receive screen projection information from the third device based on the first instruction information; the processing unit 1602 can be used to extract the basic stream ES of the screen projection information and play the ES; the communication unit 1601 can also be used to send the ES to the second device.

[0239] In an optional implementation, the processing unit 1602 may also be used to negotiate first parameter information, where the first parameter information is used to share projection information during a call between the first device and the second device.

[0240] Exemplarily, the first parameter information includes an audio channel and / or a video channel used by the first device and the second device to establish shared screen projection information.

[0241] In an optional implementation, the processing unit 1602 may also be used to negotiate second parameter information, where the second parameter information is used for the third device to project the screen onto the first device.

[0242] In some embodiments, the screen projection information includes audio information and / or video information; further, when extracting the ES of the screen projection information, the processing unit 1602 can be used to: extract the ES of the audio information and / or the ES of the video information.

[0243] In one possible embodiment, when sending the ES to the second device, the communication unit 1601 can be used to: send the ES of the audio information to the second device through a first audio channel, and / or, send the ES of the video information to the second device through a first video channel; the first audio channel is the audio channel corresponding to the call connection between the first device and the second device, and the first video channel is the video channel corresponding to the call connection between the first device and the second device; or

[0244] An ES for sending the audio information to the second device through a second audio channel, and / or an ES for sending the video information to the second device through a second video channel; the second audio channel is an audio channel used for the shared screen projection information established by the first device and the second device, and the second video channel is a video channel used for the shared screen projection information established by the first device and the second device.

[0245] Optionally, the processing unit 1602 can also be used to extract the original packet time PTS corresponding to the ES of the projection information and record the PTS; the communication unit 1601 can also be used to send the PTS to the second device.

[0246] In some embodiments, the communication unit 1601 can also be used to interact with the second device to determine whether it has a first capability when the first device and the second device establish a call connection. The first capability is to support the channel corresponding to the call connection and to support channels other than the channel corresponding to the call connection.

[0247] In one example, the communication unit 1601 can also be used to send a first identifier to the second device, where the first identifier is used to indicate whether the first device has the first capability; and receive a second identifier from the second device, where the second identifier is used to indicate whether the second device has the first capability.

[0248] Optionally, the first identifier is an indication of the number of supported channels, or the first identifier is a preset code corresponding to the first device; the second identifier is an indication of the number of supported channels, or the second identifier is a preset code corresponding to the second device.

[0249] In an optional embodiment, the processing unit 1602 may also be configured to: determine whether the first parameter information and the second parameter information are consistent before the communication unit 1601 receives the projection information from the third device; the first parameter information is used to share the projection information during a call between the first device and the second device; the second parameter information is used for the third device to project the screen to the first device;

[0250] If yes, control the communication unit 1601 to send a screen projection instruction to the third device;

[0251] Otherwise, renegotiate the first parameter information and renegotiate the second parameter information. After determining that the renegotiated first parameter information and the second information are consistent, control the communication unit 1601 to send a screen projection instruction to the third device.

[0252] In some embodiments, the processing unit 1602 may be further configured to record parameter information supported by the third device when it is determined that the first parameter information is inconsistent with the second parameter information.

[0253] Optionally, when negotiating the first parameter information, the processing unit 1602 can be used to: negotiate the first parameter information based on second parameter information, and the second parameter information is used for the third device to project the screen to the first device.

[0254] In another embodiment, when the communication device 1600 is used to implement the function of the second device in the above embodiment, the communication unit 1601 can be used to receive the basic stream ES from the first device when a call connection is established between the first device and the second device, and the ES is extracted based on the screen projection information of the third device; the processing unit 1602 can be used to play the ES.

[0255] In an optional implementation, the processing unit 1602 may also be used to negotiate first parameter information, where the first parameter information is used to share projection information during a call between the first device and the second device.

[0256] Exemplarily, the first parameter information includes an audio channel and / or a video channel used by the first device and the second device to establish shared screen projection information.

[0257] Optionally, the screen projection information includes audio information and / or video information; the ES includes the ES of the audio information and / or the ES of the video information.

[0258] In some embodiments, when receiving the ES from the first device, the communication unit 1601 can be used to: receive the ES of the audio information from the first device through a first audio channel, and / or, receive the ES of the video information from the first device through a first video channel; the first audio channel is the audio channel corresponding to the call connection between the first device and the second device, and the first video channel is the video channel corresponding to the call connection between the first device and the second device; or

[0259] An ES that receives the audio information from the first device through a second audio channel, and / or an ES that receives the video information from the first device through a second video channel; the second audio channel is an audio channel used for the shared screen projection information established by the first device and the second device, and the second video channel is a video channel used for the shared screen projection information established by the first device and the second device.

[0260] In a possible manner, the communication unit 1601 may also be configured to receive the original packet time PTS corresponding to the ES from the first device.

[0261] In some embodiments, the communication unit 1601 can also be used to interact with the first device to determine whether it has a first capability when the first device and the second device establish a call connection. The first capability is to support the channel corresponding to the call connection and to support channels other than the channel corresponding to the call connection.

[0262] In one example, the communication unit 1601 can also be used to receive a first identifier from the first device, where the first identifier is used to indicate whether the first device has the first capability; and send a second identifier to the first device, where the second identifier is used to indicate whether the second device has the first capability.

[0263] Exemplarily, the first identifier is an indication of the number of supported channels, or the first identifier is a preset code corresponding to the first device; the second identifier is an indication of the number of supported channels, or the second identifier is a preset code corresponding to the second device.

[0264] Optionally, the communication unit 1601 can also be used to receive second parameter information from the first device, and the second parameter information is used for the third device to project the screen to the first device; when negotiating the first parameter information, the processing unit 1602 can be used to: negotiate the first parameter information based on the second parameter information.

[0265] It should be noted that the division of units in the embodiments of the present application is schematic and is merely a logical functional division. In actual implementation, other division methods may be used. The functional units in the embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0266] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0267] Based on the above embodiments, embodiments of the present application further provide a communication device. Referring to FIG. 17 , a communication device 1700 may include a processor 1702. Optionally, the communication device 1700 may further include a communication interface 1701. Optionally, the communication device 1700 may further include a memory 1703. The memory 1703 may be disposed within the communication device 1700 or external to the communication device 1700. The processor 1702 may control the communication interface 1701 to receive and send information, messages, or data.

[0268] Specifically, the processor 1702 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. The processor 1702 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0269] The communication interface 1701, the processor 1702, and the memory 1703 are interconnected. Optionally, the communication interface 1701, the processor 1702, and the memory 1703 are interconnected via a bus 1704; the bus 1704 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be classified as an address bus, a data bus, a control bus, and the like. For ease of illustration, FIG17 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0270] In an optional embodiment, the memory 1703 is used to store programs, etc. Specifically, the programs may include program code, which includes computer operating instructions. The memory 1703 may include RAM, or may also include non-volatile memory (non-volatile memory), such as one or more disk storage devices. The processor 1702 executes the application program stored in the memory 1703 to implement the above functions, thereby realizing the functions of the communication device 1700.

[0271] Exemplarily, the communication device 1700 may be the first device in the above embodiment; or may be the second device in the above embodiment.

[0272] In one embodiment, when the communication device 1700 implements the functions of the first device in the aforementioned method embodiment, the communication interface 1701 may implement the transceiver operations performed by the first device in the aforementioned method embodiment; and the processor 1702 may implement other operations performed by the first device in the aforementioned method embodiment, except for the transceiver operations. For specific details, please refer to the relevant descriptions in the aforementioned method embodiment and will not be described in detail here.

[0273] In another embodiment, when the communication device 1700 implements the functions of the second device in the aforementioned method embodiment, the communication interface 1701 may implement the transceiver operations performed by the second device in the aforementioned method embodiment; and the processor 1702 may implement other operations performed by the second device in the aforementioned method embodiment in addition to the transceiver operations. For specific details, please refer to the relevant descriptions in the aforementioned method embodiment and will not be described in detail here.

[0274] Based on the above embodiments, an embodiment of the present application provides a system, which may include the first device, the first device and the third device involved in the above embodiments.

[0275] An embodiment of the present application also provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a computer, the computer can implement the screen projection sharing method provided by the above method embodiment.

[0276] An embodiment of the present application also provides a computer program product, which is used to store a computer program. When the computer program is executed by a computer, the computer can implement the screen projection sharing method provided by the above method embodiment.

[0277] An embodiment of the present application also provides a chip, including a processor, which is coupled to a memory and is used to call a program in the memory so that the chip can implement the screen projection sharing method provided by the above method embodiment.

[0278] An embodiment of the present application also provides a chip, which is coupled to a memory and is used to implement the screen projection sharing method provided in the above method embodiment.

[0279] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0280] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or box in the flow chart and / or block diagram, as well as the combination of the flow chart and / or box in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more flow charts and / or one or more boxes in the block diagram.

[0281] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0282] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0283] Obviously, those skilled in the art may make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include these modifications and variations.

Claims

1. A screen projection sharing method, characterized in that, Including: When a call connection is established between a first device and a second device, the first device obtains first instruction information for instructing to share screen projection information; Based on the first instruction information, the first device receives the screen projection information from a third device and extracts an elementary stream (ES) of the screen projection information; The first device plays the ES and sends the ES to the second device.

2. The method according to claim 1, wherein The method further includes: Negotiating first parameter information for sharing screen projection information during a call between the first device and the second device.

3. The method according to claim 2, characterized in that The first parameter information includes an audio channel and / or a video channel used for the shared screen projection information to be established between the first device and the second device.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Negotiating second parameter information for the third device to project the screen to the first device.

5. The method according to any one of claims 1-4, characterized in that The screen projection information includes audio information and / or video information; The first device extracts the ES of the screen projection information, including: The first device extracts the ES of the audio information and / or the ES of the video information.

6. The method according to claim 5, wherein The first device sends the ES to the second device, including: The first device sends the ES of the audio information to the second device through a first audio channel, and / or sends the ES of the video information to the second device through a first video channel; the first audio channel is the audio channel corresponding to the call connection between the first device and the second device, and the first video channel is the video channel corresponding to the call connection between the first device and the second device; or The first device sends the ES of the audio information to the second device through a second audio channel, and / or sends the ES of the video information to the second device through a second video channel; the second audio channel is the audio channel used for the shared screen projection information established between the first device and the second device, and the second video channel is the video channel used for the shared screen projection information established between the first device and the second device.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: The first device extracts a presentation time stamp (PTS) corresponding to the ES of the screen projection information; The first device records the PTS and sends the PTS to the second device.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: When the first device and the second device establish a call connection, the first device and the second device interact to determine whether they have a first capability, where the first capability is to support the channel corresponding to the call connection and also support channels other than the channel corresponding to the call connection.

9. The method according to claim 8, wherein The first device and the second device interact to determine whether they have the first capability, including: The first device sends a first identifier to the second device, where the first identifier is used to indicate whether the first device has the first capability; The first device receives a second identifier from the second device, where the second identifier is used to indicate whether the second device has the first capability.

10. The method according to claim 9, characterized in that, The first identifier is an indication of the number of supported channels, or the first identifier is a preset code corresponding to the first device. The second identifier is an indication of the number of supported channels, or the second identifier is a preset code corresponding to the second device.

11. The method according to any one of claims 1 to 10, characterized in that, Before the first device receives the screen mirroring information from the third device, the method further includes: The first device determines whether first parameter information and second parameter information are consistent; the first parameter information is used for sharing screen mirroring information during a call between the first device and the second device; the second parameter information is used for the third device to perform screen mirroring to the first device. If so, the first device sends a screen mirroring instruction to the third device. Otherwise, renegotiate the first parameter information and renegotiate the second parameter information. After determining that the renegotiated first parameter information and the second information are consistent, the first device sends a screen mirroring instruction to the third device.

12. The method according to claim 11, wherein The method further includes: When the first device determines that the first parameter and the second parameter information are inconsistent, the first device records the parameter information supported by the third device.

13. The method according to any one of claims 2-4, 11-12, characterized in that, Negotiating the first parameter information includes: Negotiating the first parameter information according to the second parameter information, where the second parameter information is used for the third device to perform screen mirroring to the first device.

14. A screen projection sharing method, characterized in that, Includes: When a call connection is established between the first device and the second device, the second device receives an elementary stream (ES) from the first device, and the ES is extracted based on the screen mirroring information of the third device. The second device plays the ES.

15. The method according to claim 14, characterized in that, The method further includes: Negotiating the first parameter information, where the first parameter information is used for sharing screen mirroring information during a call between the first device and the second device.

16. The method according to claim 15, wherein The first parameter information includes the audio channel and / or video channel used by the first device and the second device to establish shared screen mirroring information.

17. The method according to any one of claims 14-16, characterized in that, The screen mirroring information includes audio information and / or video information; the ES includes the ES of the audio information and / or the ES of the video information.

18. The method according to claim 17, wherein The second device receiving the ES from the first device includes: The second device receives the ES of the audio information from the first device through a first audio channel, and / or receives the ES of the video information from the first device through a first video channel; the first audio channel is the audio channel corresponding to the call connection between the first device and the second device, and the first video channel is the video channel corresponding to the call connection between the first device and the second device; or The second device receives the ES of the audio information from the first device through a second audio channel, and / or receives the ES of the video information from the first device through a second video channel; the second audio channel is the audio channel used by the first device and the second device to establish shared screen mirroring information, and the second video channel is the video channel used by the first device and the second device to establish shared screen mirroring information.

19. The method according to any one of claims 14 to 18, characterized in that The method further includes: The second device receives the presentation time stamp (PTS) of the original packet corresponding to the ES from the first device.

20. The method according to any one of claims 14-19, characterized in that, The method further includes: When the first device and the second device establish a call connection, the second device and the first device interact to determine whether the first capability is available. The first capability supports the channel corresponding to the call connection and also supports channels other than the channel corresponding to the call connection.

21. The method according to claim 20, characterized in that, The interaction between the second device and the first device to determine whether the first capability is available includes: The second device receives a first identifier from the first device, and the first identifier is used to indicate whether the first device has the first capability; The second device sends a second identifier to the first device, and the second identifier is used to indicate whether the second device has the first capability.

22. The method according to claim 21, wherein The first identifier is an indication of the number of supported channels, or the first identifier is a preset code corresponding to the first device; The second identifier is an indication of the number of supported channels, or the second identifier is a preset code corresponding to the second device.

23. The method according to claim 15 or 16, characterized in that The second device negotiates first parameter information with the first device, including: The second device receives second parameter information from the first device, and the second parameter information is used for a third device to project a screen to the first device; Negotiate the first parameter information according to the second parameter information.

24. A communication device, characterized in that, It includes a module or unit for executing the method according to any one of claims 1-13, or includes a module or unit for executing the method according to any one of claims 14-23.

25. A communication device, characterized in that, It includes a processor, and the processor is coupled to a memory, wherein: The processor is configured to call computer instructions in the memory to cause the communication device to execute the method according to any one of claims 1-13, or execute the method according to any one of claims 14-23.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called by the computer, they are used to execute the method according to any one of claims 1-13, or execute the method according to any one of claims 14-23.

27. A computer program product, characterized in that, It contains instructions that, when running on a computer, cause the method according to any one of claims 1-13, or the method according to any one of claims 14-23 to be executed.

28. A chip, characterized in that, The chip is coupled to the memory and is configured to read and execute program instructions stored in the memory to implement the method according to any one of claims 1-13, or implement the method according to any one of claims 14-23.

Images