Screen projection system, method, apparatus, storage medium, and computer program product

Abstract

The present application discloses a screen projection system, a method, an apparatus, a storage medium, and a computer program product, which belong to the technical field of computers. The method comprises: a source device sends UI attribute data and a UI structure of a screen projection area to a target device, so that the target device can independently create, reconstruct, render, and display the corresponding screen projection area on the basis of the UI structure and the UI attribute data at the UI level. In the present solution, the source device does not need to render the image and then perform video encoding and transmission, which reduces the power consumption of the source device; and it is not necessary for the source device to send the user-related data to the target device, so that the risk of a user's private data being leaked is reduced. Moreover, screen projection can be still be achieved when the target device has not installed a required screen projection application, and thus that multi-device synergy can be supported. In addition, the target device automatically renders and displays the screen projection area, which can be adapted to the screen resolution of the target device and improve the display effect.

Description

Screen projection system, method, device, storage medium and computer program product

[0001] This application claims priority to Chinese patent application No. 202311287387.9 filed on September 28, 2023, entitled “Screen projection system, method, device, storage medium and computer program product”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a screen projection method, device, storage medium and computer program product. Background Art

[0003] Currently, screen projection technologies such as wireless screen projection, multi-screen collaboration, and relay projection are widely used in people's lives and work. For example, mobile devices such as mobile phones and tablets can serve as source devices, which can project or migrate screen content to destination devices such as computers, TVs, and projectors for viewing or user operations on larger screens.

[0004] There are two main ways to implement screen projection technology in the related art. The first way is the screen mirroring method, and the second way is the continuation method, which is also called migration or flow. In the screen mirroring method, after the source device generates a screen image by rendering, it encodes the screen image frame by frame to form a video stream. The source device sends the video stream to the destination device. The destination device can display the screen image after decoding the video stream, thereby projecting the screen image of the source device onto the destination device. In the continuation method, the source device and the destination device need to have the same application installed. During the running of the application, the source device exports the relevant data of the application and sends it to the destination device. After receiving the data sent by the source, the destination device starts the same application and runs the application based on the received data, thereby realizing the continuation of the application from the source device to the destination device.

[0005] However, in the screen mirroring method, all the work of the application needs to be completed on the source device, including the rendering operations with large computational workload, and the source device is also required to perform video encoding and transmission. When performing high-definition screen projection, the power consumption of encoding and decoding and video transmission is relatively high. In addition, the rendering operation is completed on the source device, and the screen image is rendered according to the screen resolution of the source device. In this way, it is impossible to adapt to the screen resolution of the destination device, resulting in the screen image displayed on the destination device and the screen resolution of the destination device may not match. In the subsequent method, the destination device is required to install the same application, and multi-device collaboration is not supported. In addition, the source device needs to send the user-related data of the application to the destination device, which may involve the leakage of user privacy data.

[0006] Summary of the Invention

[0007] This application provides a screen projection system, method, device, storage medium, and computer program product that can adapt to the screen resolution of the target device, improve the screen projection effect, reduce the amount of data sent, reduce the power consumption of the source device, and reduce the risk of user privacy data leakage. The technical solution is as follows:

[0008] In a first aspect, a screen projection system is provided, which includes a source-end screen projection module and a destination-end screen projection module; the source-end screen projection module is used to send a user interface (UI) structure and UI attribute data of a first projection area to the destination-end screen projection module, so as to project the first projection area from a source device where the source-end screen projection module is located to a destination device where the destination-end screen projection module is located, wherein the UI structure indicates a layout relationship between multiple UI nodes in the first projection area, and the UI attribute data indicates rendering-related attributes of each UI node in the multiple UI nodes; the destination-end screen projection module is used to receive the UI structure and UI attribute data, and display the first projection area on the destination device based on the UI structure and the UI attribute data.

[0009] The destination device can rebuild and render the corresponding projection area based on the UI structure and UI attribute data at the UI level. This solution does not require the source device to render the image and then perform video encoding and transmission, which reduces the power consumption of the source device. It also does not require the source device to send user-related data to the destination device, reducing the risk of user privacy data leakage. Moreover, this solution can also achieve screen projection even when the destination device does not have an application that requires screen projection installed, thereby supporting multi-device collaboration. In addition, the destination device renders and displays the projection area by itself, which can adapt to the screen resolution of the destination device and improve the display effect.

[0010] The target-end screen projection module is specifically configured to: load the UI structure through the UI framework of the target device to layout the multiple UI nodes in the first display area, and set the attributes of each of the multiple UI nodes based on the UI attribute data to reconstruct and render the first projection area. In other words, this solution is specifically implemented based on the UI framework of the target device.

[0011] The source screen projection module is also used to build the UI structure and generate the UI attribute data through the UI framework of the source device according to the application instructions. In other words, this solution relies on the UI layer below the application layer.

[0012] In one possible implementation, the destination screen projection module is further configured to send first control information to the source screen projection module to update some or all UI nodes in the first projection area by reversely controlling the source device. In other words, this solution supports the destination device reversely controlling the source device.

[0013] The source device runs a first application, and the first projection area is the projection area of ​​the first application.

[0014] In a possible implementation, the destination device does not have the first application installed. That is, this solution does not require the destination device to have the same application installed.

[0015] In one possible implementation, the first projection area is part or all of the first page. That is, this solution supports projection of a partial page area.

[0016] In the second aspect, a screen projection method is provided, which is applied to a source device, and the method includes: sending a user interface UI structure and UI attribute data of a first projection area to a destination device to project the first projection area to the destination device; wherein, the UI structure indicates the layout relationship between multiple UI nodes in the first projection area, and the UI attribute data indicates the rendering-related attributes of each UI node in the multiple UI nodes, and the UI structure and the UI attribute data are used to display the first projection area on the destination device.

[0017] That is, the destination device can construct, rebuild, and render the corresponding projection area based on the UI structure and UI attribute data at the UI level. This solution does not require the source device to render the image and then perform video encoding and transmission, which reduces the power consumption of the source device. It also does not require the source device to send user-related data to the destination device, reducing the risk of user privacy data leakage. Moreover, this solution can also achieve screen projection even when the destination device does not have an application that requires screen projection installed, thereby supporting multi-device collaboration. In addition, the destination device renders and displays the projection area by itself, which can adapt to the screen resolution of the destination device and improve the display effect.

[0018] Among them, before sending the UI structure and UI attribute data of the first projection area to the destination device, the method also includes: constructing the UI structure and generating the UI attribute data through the UI framework of the source device according to application instructions.

[0019] In one possible implementation, the multiple UI nodes include a first UI node corresponding to a first source-end component, the first source-end component belongs to a first-category component, and the first-category component has a corresponding image drawn by a non-UI framework; the UI attribute data includes a drawing instruction of the first source-end component, and the drawing instruction of the first source-end component is used to draw an image not rendered by the first source-end component through the non-UI framework; or, the UI attribute data includes an image rendered by the first source-end component. It should be understood that the first-category component can be considered a non-standard component. For example, the first-category component includes developer-defined components or some special components preset by the system. For the first-category component, the source device synchronizes the drawing instruction or the rendered image to the destination device.

[0020] In one possible implementation, the multiple UI nodes include a second UI node corresponding to a second source-end component, the second source-end component belongs to a second-category component, and the second-category component has a corresponding image drawn by the UI framework; the UI attribute data includes drawing attribute data of the second source-end component, and the drawing attribute data of the second source-end component is used to draw the corresponding image of the second source-end component through the UI framework. It should be understood that the second-category component can be considered as a standard component. For example, the second-category component includes some standard components preset by the system. For the second-category component, the source device synchronizes the attribute data used to draw the corresponding component to the destination device.

[0021] In one possible implementation, the source device runs a first application, and the first projection area is the projection area of ​​the first application. The method further includes: sending application resources of the first application to the destination device, the application resources including the resources required for rendering the multiple UI nodes, and the application resources and the UI attribute data are used together by the destination device to reconstruct and render the first projection area. That is, the source device may also need to synchronize application resources to the destination device. Among them, application resources may include configuration data, pictures, text, etc. in the application package.

[0022] In one possible implementation, the method further includes: receiving first manipulation information sent by the destination device, the first manipulation information indicating a first operation of the destination user on the first projection area; performing a second operation on the first projection area based on the first manipulation information, and updating the UI structure and / or UI attribute data of the first projection area; and sending UI update data of the first projection area to the destination device, the UI update data being used by the destination device to update the first projection area through the UI framework of the destination device. That is, this solution supports the destination device to control the source device.

[0023] In one possible implementation, the UI attribute data includes the identifiers of the multiple UI nodes, the first control information includes the identifier of the third UI node and the control instruction corresponding to the first operation, the control instruction indicates the second operation, and the third UI node is one of the multiple UI nodes; performing the second operation on the first projection area based on the first control information includes: based on the identifier of the third UI node, performing the second operation on the third UI node according to the control instruction. That is, this solution supports the destination device to reversely control individual UI nodes, and the control information synchronized by the destination device to the source device is a specific control instruction.

[0024] In one possible implementation, the UI attribute data includes the identifiers of the multiple UI nodes, the first control information includes the identifier of the fourth UI node and the operation information of the first operation, and the fourth UI node is one of the multiple UI nodes; performing the second operation on the first projection area based on the first control information includes: parsing the operation information of the first operation to determine the second operation; and performing the second operation on the fourth UI node based on the identifier of the fourth UI node. That is, this solution supports the destination device to reversely control individual UI nodes, and the control information synchronized by the destination device to the source device is the user's operation information, and the source device needs to parse the operation information to obtain the second operation corresponding to the corresponding operation instruction.

[0025] In one possible implementation, the source device runs a first application, and the first projection area is a projection area within a first page of the first application. The method further includes: when the first application switches from the first page to a second page, sending UI structure and UI attribute data of a second projection area to the destination device to project the second projection area to the destination device, where the second projection area is a projection area within the second page. That is, when a page switch occurs, the source device resends the UI structure and UI attribute data of the new projection area.

[0026] In a possible implementation, the source device supports multiple screen projection modes; the method further includes: displaying a current screen projection mode, where the current screen projection mode is one of the multiple screen projection modes.

[0027] In one possible implementation, the source device supports multiple projection modes, with different projection areas having different projection modes. Before sending the user interface (UI) structure and UI attribute data of the first projection area to the destination device, the method further includes: determining the projection mode of the first projection area from the multiple projection modes using a mode adaptation algorithm. In other words, this solution supports adaptive switching of projection modes during the projection process.

[0028] In a third aspect, a screen projection method is provided, which is applied to a destination device, and the method includes: receiving a user interface UI structure and UI attribute data of a first screen projection area sent by a source device, wherein the UI structure indicates a layout relationship between multiple UI nodes in the first screen projection area, and the UI attribute data indicates rendering-related attributes of each UI node in the multiple UI nodes; and displaying the first screen projection area based on the UI structure and the UI attribute data.

[0029] That is, the destination device can construct, rebuild, and render the corresponding projection area based on the UI structure and UI attribute data at the UI level. This solution does not require the source device to render the image and then perform video encoding and transmission, which reduces the power consumption of the source device. It also does not require the source device to send user-related data to the destination device, reducing the risk of user privacy data leakage. Moreover, this solution can also achieve screen projection even when the destination device does not have an application that requires screen projection installed, thereby supporting multi-device collaboration. In addition, the destination device renders and displays the projection area by itself, which can adapt to the screen resolution of the destination device and improve the display effect.

[0030] In one possible implementation, displaying the first projection area based on the UI structure and the UI attribute data includes: loading the UI structure through the UI framework of the destination device to layout the multiple UI nodes in the first display area; and setting the attributes of each of the multiple UI nodes based on the UI attribute data to reconstruct and render the first projection area. In other words, this solution is specifically implemented based on the UI framework of the destination device.

[0031] In one possible implementation, the multiple UI nodes include a first UI node corresponding to a first source-end component, the first source-end component belongs to a first-category component, and the first-category component has a corresponding image drawn by a non-UI framework; the UI attribute data includes a drawing instruction of the first source-end component, and the drawing instruction of the first source-end component is used to draw an image not rendered by the first source-end component through the non-UI framework; or, the UI attribute data includes an image rendered by the first source-end component. It should be understood that the first-category component can be considered a non-standard component. For example, the first-category component includes developer-defined components or some special components preset by the system. For the first-category component, the source device synchronizes the drawing instruction or the rendered image to the destination device.

[0032] Wherein, in the case where the UI attribute data includes the drawing instruction of the first source-end component, the UI structure is loaded through the UI framework of the destination device to layout the first UI node in the first display area, including: creating the first destination-end component by loading the UI structure through the UI framework to obtain the first UI node; setting the attributes of the first UI node based on the UI attribute data, including: drawing the image of the first destination-end component through the non-UI framework based on the drawing instruction of the first source-end component to obtain an unrendered image of the first UI node.

[0033] In a case where the UI attribute data includes an image rendered by the first source-end component, the UI structure is loaded through the UI framework of the destination device to layout the first UI node in the first display area, including: loading the UI structure through the UI framework to create a second destination-end component to obtain the first UI node, the second destination-end component being different from the first source-end component; setting the attributes of the first UI node based on the UI attribute data, including: based on the image rendered by the first source-end component, by setting the UI attributes of the second destination-end component, so that the first UI node carries the image rendered by the first source-end component.

[0034] In one possible implementation, the multiple UI nodes include a second UI node corresponding to a second source-end component, the second source-end component belongs to a second-category component, the second-category component has a corresponding image drawn by the UI framework, and the UI attribute data includes the drawing attribute data of the second source-end component; the UI structure is loaded through the UI framework of the destination device to layout the second UI node in the first display area, including: loading the UI structure through the UI framework to create a third destination-end component to obtain the second UI node, the third destination-end component is the same as the second source-end component; setting the attributes of the second UI node based on the UI attribute data, including: setting the attributes of the second source-end component through the UI framework based on the drawing attribute data of the second source-end component to draw the corresponding image of the second UI node. It should be understood that the second-category component can be considered as a standard component. For example, the second-category component includes some standard components preset by the system. For the second-category component, the source device synchronizes the attribute data used to draw the corresponding component to the destination device.

[0035] In one possible implementation, the method further includes: in response to a first operation of a destination user on the first projection area, sending first manipulation information to the source device, wherein the first manipulation information is used to instruct the source device to perform a second operation on the first projection area and update the UI structure and / or UI attribute data of the first projection area; receiving UI update data of the first projection area sent by the source device; and updating the first projection area based on the UI update data of the first projection area. That is, this solution supports the destination device to control the source device.

[0036] In one possible implementation, the source device supports multiple projection modes, with different projection modes available for different projection areas. The method further includes sending real-time computing capability information of the destination device to the source device, where the source device uses a mode adaptation algorithm to determine the projection mode for the current projection area from the multiple projection modes. In other words, this solution supports adaptive switching of projection modes during the projection process.

[0037] In a fourth aspect, a screen projection device is provided, wherein the screen projection device has the function of implementing the screen projection method described in the second or third aspect. The screen projection device includes one or more modules, which are used to implement the screen projection method described in the second or third aspect.

[0038] In a fifth aspect, an electronic device is provided, comprising a processor and a memory, wherein the memory is used to store a program for executing the screen projection method provided in the second or third aspect, and to store data involved in implementing the screen projection method provided in the second or third aspect. The processor is configured to execute the program stored in the memory. The electronic device may further include a communication bus for establishing a connection between the processor and the memory.

[0039] In a sixth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, which, when executed on a computer, enables the computer to execute the screen projection method described in the second or third aspect above.

[0040] In the seventh aspect, a computer program product containing instructions is provided, which, when run on a computer, enables the computer to execute the screen projection method described in the second or third aspect above.

[0041] The technical effects obtained in the above-mentioned fourth to seventh aspects are similar to the technical effects obtained by the corresponding technical means in the first to third aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] FIG1 is a schematic diagram of a screen mirroring method according to an embodiment of the present application;

[0043] FIG2 is a schematic diagram of a continuous screen projection method according to an embodiment of the present application;

[0044] FIG3 is a system architecture diagram of a screen projection method provided in an embodiment of the present application;

[0045] FIG4 is a schematic diagram of the software and hardware architecture of an electronic device provided in an embodiment of the present application;

[0046] FIG5 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0047] FIG6 is a flow chart of a screen projection method provided in an embodiment of the present application;

[0048] FIG7 is a flowchart of another screen projection method provided in an embodiment of the present application;

[0049] FIG8 is a schematic diagram of a screen projection method for non-standard components in the screen projection method provided in an embodiment of the present application;

[0050] FIG9 is a schematic diagram of a mode adaptation provided by an embodiment of the present application;

[0051] FIG10 is a flowchart of a screen projection process in a page update situation provided by an embodiment of the present application;

[0052] FIG11 is a flowchart of another screen projection method provided in an embodiment of the present application;

[0053] FIG12 is a schematic diagram of the core steps of a screen projection method provided in an embodiment of the present application;

[0054] FIG13 is a schematic diagram of a screen projection area on a source device provided in an embodiment of the present application;

[0055] FIG14 is a schematic diagram of a screen projection effect on a destination device in the solution provided by an embodiment of the present application;

[0056] FIG15 is a schematic diagram of a screen projection effect of a mirror mode on a target device in the related art provided by an embodiment of the present application;

[0057] FIG16 is a schematic diagram of another screen projection area of ​​the source device in the solution provided by an embodiment of the present application;

[0058] FIG17 is a schematic diagram of a screen projection area of ​​a destination device corresponding to FIG16 in the solution provided in an embodiment of the present application;

[0059] FIG18 is a schematic structural diagram of a screen projection device provided in an embodiment of the present application;

[0060] Figure 19 is a structural schematic diagram of another screen projection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0062] At present, screen projection technologies such as wireless screen projection, multi-screen collaboration, and connection have been widely used in people's lives and work. For example, mobile terminals such as mobile phones and tablets can be used as source devices, and the source devices can project or migrate the screen content to target devices such as computers, TVs, and projectors for viewing on a larger screen or performing user operations. Of course, large-screen devices can also be projected to small-screen devices. For example, the video call application on the computer is connected to the mobile phone so that the user can continue the video call using the mobile phone. It should also be understood that the source device can be one or more. For example, in a multi-channel screen projection scenario, multiple source devices can be projected to the same target device at the same time to display the screen images corresponding to the multiple source devices on the target device in split screen. Similarly, the target device can be one or more. For example, the source device can share the screen image with multiple target devices so that users of the multiple target devices can view the screen images corresponding to the source device.

[0063] There are two main ways to implement screen projection technology in related technologies. The first method is the screen mirroring projection method, and the second method is the continuous screen projection method, which is also called migration or flow. For easy understanding, the execution process of the application is first introduced. The execution process of the application usually includes three steps. The first step is the logical operation of the application, that is, the application layer of the device runs the UI components and application function logic developed by the application developer to trigger a series of UI tasks or instructions. The second step is for the device to load its own system UI framework (which can be simply referred to as UI framework) to execute this series of UI tasks or instructions, thereby building or updating the application page. The third step is for the device to render the display page on the screen through its own graphics system based on the constructed page.

[0064] FIG1 is a schematic diagram of a screen mirroring method according to an embodiment of the present application. Referring to FIG1 , in the screen mirroring method, after the source device generates a screen image by rendering, it encodes the screen image frame by frame to form a video stream. The source device sends the video stream to the destination device via the network. The destination device can display the screen image after decoding the video stream through its own graphics system, thereby realizing the projection of the screen image of the source device onto the destination device. However, when performing wireless high-definition screen projection or multi-channel screen projection, the power consumption of the source device for encoding and decoding is large, and since the video stream needs to be continuously transmitted on the network, the power consumption of the wireless transmission module (such as wireless fidelity (WiFi) chip) on the source device is large. For high-definition screen projection, the key frame in the video stream is about 100-200KB in size, and the amount of network transmission data is large. When the network quality is poor, it is easy to cause a jamming phenomenon on the destination device. In addition, the rendering operation is completed on the source device, and the screen image is rendered according to the screen resolution of the source device. In this way, it is impossible to adapt to the screen resolution of the destination device, resulting in the screen image displayed on the destination device and the screen resolution of the destination device may not match.

[0065] Figure 2 is a schematic diagram of the continuous screen projection method involved in an embodiment of the present application. Referring to Figure 2, in the continuous screen projection method, the source device and the destination device need to have the same application installed. During the operation of the application, the source device exports the relevant data of the application (including application status data, etc.) and sends it to the destination device through the network. After receiving the data sent by the source, the destination device starts the same application and runs the application by writing the application status data, thereby connecting the application from the source device to the destination device for operation. However, in the continuous screen projection method, the destination device is required to install the same application, and does not support multi-device collaboration and the destination device's reverse control of the source device. In addition, the source device needs to send the user-related data of the application to the destination device, which may involve the leakage of user privacy data.

[0066] In order to solve the above-mentioned problems in the related art, an embodiment of the present application provides a new screen projection method, which will be introduced in detail below.

[0067] FIG3 is a diagram of the system architecture involved in a screen projection method provided in an embodiment of the present application. Referring to FIG3 , the system architecture includes a source device and a destination device. A wireless or wired communication connection is established between the source device and the destination device. The source device includes a source-end screen projection module, and the destination device includes a destination-end screen projection module.

[0068] The source device runs the application to be projected and sends the application's UI data stream to the destination device based on its own system UI framework. This means synchronizing the internal UI state data, including UI structure and UI attribute data, with the destination device. The destination device then uses its own UI framework to construct and render the corresponding projection area based on the received UI data stream.

[0069] Among them, the application can be a system program that comes with the source device when it leaves the factory, or a non-system program installed on the source device after leaving the factory, or a small program, a quick application, and other programs that do not need to be installed and can be used immediately. That is, as long as the source device can present a picture, it can be projected according to this solution. In addition, the projection area can be the entire display area of ​​the source device, or it can be part of the display area of ​​the source device. Taking the source device running the first application as an example, the projection area can be part or all of the first page in the first application.

[0070] Taking the entire area of ​​the first page in the first application as an example, in a specific implementation, the source device is used to send the UI structure and UI attribute data of the first page to the destination device after constructing the first page through its own UI framework during the running of the first application. The destination device is used to receive the UI structure and UI attribute data of the first page, and based on the UI structure and UI attribute data of the first page, construct multiple UI nodes of the first page through its own UI framework, and render and display the first page through its own graphics system. In some cases, the source device also needs to send the application resources of the first application (including the resources required to render the multiple UI nodes in the first page) to the destination device. The application resources are such as icon resources and text resources in the application package. The application resources and UI attribute data are used together to set the attributes of the UI nodes. That is, the destination device is used to load the UI structure, UI attribute data and application resources through the UI framework to construct the first page. Among them, when the first page is the first page to be projected, the destination device needs to load the UI framework first. After loading the UI framework, the destination device can construct each page to be projected through its own UI framework.

[0071] It should be understood that this solution does not limit whether the destination device has the first application installed.

[0072] Figure 4 is a schematic diagram of the hardware and software architecture of an electronic device provided in an embodiment of the present application. The electronic device can serve as a source device or a destination device. The electronic device includes an application portion, a system UI framework (referred to as the UI framework), a network synchronization module, a service management module, a processor, memory, and a network card.

[0073] Among them, the application part includes applications installed after leaving the factory (referred to as applications in Figure 4) and system applications installed at the factory (referred to as dedicated applications in Figure 4). The application has an application logic part and application resources and configuration files. The application logic part includes program logic (Logic) and pages (Pages). Application resources and configuration files include media resources (media) and original documents (rawfile). In the embodiment of the present application, the dedicated application in the destination device includes a UI display module, while the dedicated application in the source device may not include a UI display module, but may also include a UI display module. The UI display module also includes two parts: Logic and Pages.

[0074] The UI framework consists of a front-end framework layer, an engine layer, and a UI proxy module. The front-end framework layer includes application programming interfaces (APIs), specialized components (such as non-standard components), standard components, and component events. APIs include the JavaScript new application program interface (JS NAPI). The engine layer is responsible for parsing, building, and rendering UI components.

[0075] In an embodiment of the present application, each UI node corresponds to a component, and the component is used to determine the UI node. The components include first-class components and second-class components. The first-class component draws the corresponding image by the non-UI framework. In other words, the first-class component can be constructed by the UI framework but the corresponding image is drawn by the non-UI framework. For example, the corresponding image is drawn by running the first program code by the non-UI framework, and the first program code is provided by other frameworks or application developers. The second-class component draws the corresponding image by the UI framework. In other words, the second-class component can be constructed and draw the corresponding image by the UI framework. Among them, the UI node corresponding to the first-class component is called the first-class node, and the UI node corresponding to the second-class component is called the second-class node. In a specific implementation, the first-class component includes non-system pre-set components, such as custom components of application developers, and may also include some special components pre-set by the system, such as surface components, web components, and video components. The second-class component includes some standard components pre-set by the system, such as Image components. In some embodiments, the first-class component can also be referred to as non-standard components, and the second-class component can be referred to as standard components. That is, custom components and special components can be referred to as non-standard components, and other components can be referred to as standard components.

[0076] It should be understood that one component corresponds to one or more UI nodes, and images of one or more UI nodes can be generated by drawing one component.

[0077] Processors include central processing units (CPUs) and / or graphics processing units (GPUs). Processors, memory, and network cards provide hardware support for application execution, data processing, and data transmission, specifically providing system and network resources.

[0078] It should be understood that the UI display module, UI proxy module, network synchronization module and service management module in FIG4 are newly added modules in the embodiment of the present application relative to the electronic device in the related art.

[0079] Next, the roles of the above-mentioned UI display module, UI agent module, network synchronization module and service management module in the embodiment of the present application will be introduced in conjunction with Figure 4.

[0080] At the beginning of screen projection, the service management module of the source device and the service management module of the destination device are used to negotiate the screen projection mode to confirm which of the multiple screen projection modes is currently in use. The multiple screen projection modes include mirror mode, UI data stream mode (i.e., a screen projection mode provided in an embodiment of the present application), etc. When the use of the UI data stream mode is confirmed, the destination device starts its own UI display module. After the source device executes the application logic of the application to be projected and determines or updates the UI structure and UI attribute data of the area to be projected through the UI framework, the UI proxy module of the source device obtains the UI structure and UI attribute data or UI update data, and sends it to the network synchronization module of the destination device through the network synchronization module of the source device. The UI proxy module of the destination device creates a display area or updates the UI based on the UI structure and UI attribute data or UI update data of the area to be projected, and then renders the corresponding image through the engine layer of the destination device, and displays the projection area through the UI display module of the destination device.

[0081] The embodiments of the present application do not limit the specific devices of the source device and the destination device. For example, the source device can be a mobile phone, tablet computer, desktop computer, laptop computer, etc., and the destination device can be a mobile phone, laptop computer, desktop computer, projector, etc.

[0082] The system architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that, with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0083] Please refer to Figure 5, which is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. In one possible implementation, the electronic device is the source device or destination device shown in Figure 3. The electronic device includes one or more processors 501, a communication bus 502, a memory 503, and one or more communication interfaces 504.

[0084] The processor 501 includes a CPU, a GPU, a network processing (NP), a microprocessor, or one or more integrated circuits for implementing the solution of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. In one possible implementation, the PLD is a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0085] Communication bus 502 is used to transmit information between the aforementioned components. In one possible implementation, communication bus 502 is divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the figure shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0086] In one possible implementation, the memory 503 is a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compact disc, a laser disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium, or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 503 exists independently and is connected to the processor 501 via the communication bus 502, or the memory 503 is integrated with the processor 501.

[0087] The communication interface 504 uses any device, such as a transceiver, for communicating with other devices or communication networks. The communication interface 504 includes a wired communication interface and, in one possible implementation, also includes a wireless communication interface. Examples of wired communication interfaces include Ethernet interfaces. In one possible implementation, the Ethernet interface is an optical interface, an electrical interface, or a combination thereof. The wireless communication interface is a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0088] In some embodiments, the electronic device includes multiple processors, such as processor 501 and processor 505 shown in Figure 5. Each of these processors is a single-core processor or a multi-core processor. In one possible implementation, the processor here refers to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0089] In a specific implementation, as an embodiment, the electronic device further includes an output device 506 and an input device 507. The output device 506 communicates with the processor 501 and can display information in a variety of ways. For example, the output device 506 is a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 507 communicates with the processor 501 and can receive user input in a variety of ways. For example, the input device 507 is a mouse, a keyboard, a touch screen device, or a sensor device.

[0090] In some embodiments, the memory 503 is used to store program code 510 for executing the solution of the present application, and the processor 501 is capable of executing the program code 510 stored in the memory 503. The program code includes one or more software modules, and the electronic device can implement the screen projection method provided in the embodiments of Figure 6 or Figure 7 below through the processor 501 and the program code 510 in the memory 503.

[0091] FIG6 is a flowchart of a screen projection method provided in an embodiment of the present application. The method is applied to a screen projection system, and the screen projection system includes a source device and a destination device. Specifically, the source device includes a source-end screen projection module, and the destination device includes a destination-end screen projection module. The method can be specifically applied to the source-end screen projection module and the destination-end screen projection module. Please refer to FIG6 , the method includes the following steps.

[0092] Step 601: The source device sends the UI structure and UI attribute data of the first projection area to the destination device to project the first projection area to the destination device.

[0093] Among them, the UI structure indicates the layout relationship between multiple UI nodes in the first projection area, the UI attribute data indicates the rendering-related attributes of each UI node in the multiple UI nodes, and the UI structure and UI attribute data are used to display the first projection area on the target device.

[0094] That is, the source device synchronizes the UI layer data to the destination device. Before synchronizing the UI structure and UI attribute data, the source device may need to first build the UI structure and generate the UI attribute data through the source device's UI framework according to the application instructions.

[0095] In some embodiments, after the screen projection process starts, the source device sends application resources of the first application to the destination device. The first application refers to the application to which the first screen projection area belongs, and the application resources are the resources required to render the first screen projection area.

[0096] The first projection area is part or all of the first page in the first application. The source device is running the first application, and the destination device is not running the first application, for example, the destination device does not have the first application installed. The first application can be a system application or a non-system application.

[0097] Step 602: The destination device receives the UI structure and UI attribute data of the first projection area sent by the source device.

[0098] Step 603: The destination device displays the first projection area based on the UI structure and UI attribute data.

[0099] Among them, the destination device loads the UI structure through the UI framework of the destination device to layout the multiple UI nodes in the first display area, and sets the properties of each UI node in the multiple UI nodes based on the UI attribute data to reconstruct and render the first projection area.

[0100] In this solution, the source device sends the UI structure and UI attribute data of the projection area to the destination device, so that the destination device can rebuild and render the corresponding projection area based on the UI structure and UI attribute data at the UI level. This solution does not require the source device to perform video encoding and transmission after rendering the image, which reduces the power consumption of the source device. It also does not require the source device to send user-related data to the destination device, reducing the risk of user privacy data leakage. Moreover, this solution can also achieve projection when the destination device does not have an application that requires projection installed, thereby supporting multi-device collaboration. In addition, the projection area is rendered and displayed by the destination device itself, which can adapt to the screen resolution of the destination device and improve the display effect.

[0101] Next, taking the synchronization of application resources, UI structure, and UI attribute data from a source device to a destination device as an example, the specific implementation of the embodiment of FIG. 6 will be described in detail with reference to FIG. 7 .

[0102] FIG7 is a flowchart of another screen projection method provided in an embodiment of the present application. The method is applied to a screen projection system, and the screen projection system includes a source device and a destination device. Specifically, the source device includes a source-end screen projection module, and the destination device includes a destination-end screen projection module. The method can be specifically applied to the source-end screen projection module and the destination-end screen projection module. Please refer to FIG7 , the method includes the following steps.

[0103] Step 701: The source device sends application resources of a first application to the destination device.

[0104] The source device runs a first application. After the screen projection process begins, the source device sends the application resources of the first application to the destination device. The application resources include the resources required to render the projection area, such as configuration data, images (such as icons), text, and other media resources in the application package.

[0105] It should be understood that synchronizing application resources is to prepare for subsequent rendering and display on the destination device, and the destination device may use some, all, or none of the application resources in the subsequent rendering and display. Based on this, in another possible implementation, the source device synchronizes the used application resources to the destination device each time a projection area is projected. That is, the source device synchronizes the application resources required for the current projection area in real time, rather than synchronizing all application resources to non-destination devices at the beginning of the projection process.

[0106] As can be seen from the foregoing, the projection mode provided in the embodiment of the present application is a UI data stream mode, and the UI data stream mode refers to a mode in which corresponding UI attribute data is transmitted to all UI nodes in the same projection area. The projection modes in the related art include mirror mode and connection mode. If the source device only supports the UI data stream mode, or the source device supports multiple projection modes but is configured to use the UI data stream mode by default, then after the projection process starts, the source device directly sends the application resources of the first application to the destination device. If the source device supports multiple projection modes but is not configured to use a certain projection mode by default, then before sending the application resources of the first application to the destination device, the source device determines that the current projection mode is the UI data stream mode among the multiple projection modes, and then performs the step of sending the application resources of the first application to the destination device. That is, when the source device supports multiple projection modes, the source device and the destination device can first determine the current projection mode from the multiple projection modes.

[0107] There are multiple ways to determine the screen projection mode between the source device and the destination device. Here are three of them.

[0108] In a first implementation method of determining the current screen projection mode, the source device sends a first request to the destination device, and the first request is used to obtain the device resource information of the destination device. The destination device receives the first request sent by the source device, and in response to the first request, sends the device resource information of the destination device to the source device. The source device receives the device resource information of the destination device sent by the destination device, and based on the device resource information of the destination device, determines the current screen projection mode from the multiple screen projection modes, such as determining that the current screen projection mode is the UI data stream mode, that is, the device resource information of the destination device is used by the source device to determine that the current screen projection mode is the UI data stream mode from the multiple screen projection modes. In a possible implementation method, the source device sends a first message to the destination device, and the first message indicates the current screen projection mode, such as indicating that the current screen projection mode is the UI data stream mode.

[0109] In the second implementation method of determining the current screen projection mode, the source device sends the device resource information of the source device to the destination device. The destination device receives the device resource information of the source device sent by the source device, and based on the device resource information of the source device, determines that the current screen projection mode is the UI data stream mode, and sends a first response to the source device. The first response indicates the current screen projection mode, for example, indicating that the current screen projection mode is the UI data stream mode. The source device receives the first response sent by the destination device and obtains the current screen projection mode by parsing the first response.

[0110] In the third implementation of determining the current projection mode, the source device selects the current projection mode from multiple projection modes based on the detected user action, such as if the user selects the UI data streaming mode. In other words, this solution allows users (such as technicians) to select the projection mode on demand.

[0111] The above-mentioned device resource information indicates one or more of the software and hardware capabilities, operating system version and UI framework version of the corresponding device. Among them, the software and hardware capabilities of the device include the CPU architecture of the device, whether it has a GPU, the GPU architecture it has, the graphics card configuration and other software and hardware specifications. In the first two implementation methods of determining the current screen projection mode, the conditions for determining that the current screen projection mode is the UI data stream mode based on the device resource information include that the operating system and UI framework of the target device need to be compatible with the operating system and UI framework of the source device, and the software and hardware of the target device support the construction and rendering of the display page through the UI data stream mode.

[0112] In an embodiment of the present application, the multiple projection modes supported by the source device also include one or more of a mirror mode, a hybrid mode, and a connection mode. Among them, the mirror mode refers to a mode in which all UI nodes in the same projection area are rendered to obtain a rendered image of the corresponding projection area and transmitted. The hybrid mode is also called a media stream + UI data stream mode. The hybrid mode refers to a mode in which the corresponding UI attribute data is transmitted to the second UI node, and the third UI node is rendered to obtain a rendered image of the third UI node and transmitted. The second UI node and the third UI node are different UI nodes in the same projection area. The connection mode refers to a mode in which the first application is connected to the destination device to run and the source device stops running the first application.

[0113] It should be understood that a projection area includes multiple UI nodes, and the mixed mode can be understood as using different projection modes for two different UI nodes in the same projection. For example, the content of the second UI node in a projection area is a video, and the content of the third UI node is an icon. Then, the UI data stream mode can be used for the second UI node, for example, the source device transmits the link of the video to the destination device, and the media stream mode can be used for the third UI node, that is, the source device transmits the rendered image of the icon to the destination device.

[0114] In one possible implementation, when the source device includes the service management module and the network synchronization module shown in Figure 4, the source device can negotiate the screen projection mode with the destination device through the service management module, and send the application resources of the first application to the destination device through the network synchronization module.

[0115] Step 702: The source device constructs a UI structure of the first projection area and generates UI attribute data of the first projection area.

[0116] The source device constructs the UI structure of the first projection area and generates UI attribute data of the first projection area while executing step 701 or after executing step 701. Specifically, the source device constructs the UI structure of the first projection area and generates UI attribute data of the first projection area through the UI framework of the source device. Before projecting the first projection area, the source device needs to load the UI framework of the source device first, such as loading the UI framework when the source device starts the first application, and then constructing the projection area of ​​each page through the loaded UI framework.

[0117] The first projection area is part or all of the first page in the first application. When the first projection area is a partial area of ​​the first page, the source device constructs the UI structure of the first page and generates UI attribute data for the first page. Alternatively, the source device may only construct the UI structure of the first projection area and generate UI attribute data for the first projection area.

[0118] In one possible implementation, the first projection area can be determined based on a user's selection of part or all of the source device's display area, meaning this solution supports user selection of the projection area. Alternatively, the first projection area can be determined based on the default settings of the first application, meaning this solution supports automatic projection of a portion of a page within the application.

[0119] The UI structure of the first projection area includes multiple UI nodes, each UI node corresponds to a component, and the multiple UI nodes include a first UI node. The source device obtains the first UI node by creating a first source-end component, that is, the first UI node in the source device corresponds to the first source-end component, and the first source-end component belongs to a first-class component. The first-class component draws the corresponding image by a non-UI framework. For example, the first-class component is drawn by running a first program code through a non-UI framework, and the first program code is provided by the developer of other frameworks or the first application.

[0120] In the case that the multiple UI nodes include the first UI node, the above-mentioned UI attribute data includes the drawing instruction of the first source-end component, the drawing instruction of the first source-end component is the instruction generated by the source device in the process of drawing the first source-end component, and the drawing instruction of the first source-end component is used by the destination device to draw the image not rendered by the first source-end component through the non-UI framework; or, the above-mentioned UI attribute data includes the image rendered by the first source-end component, and the image rendered by the first source-end component is obtained by the source device rendering the image not rendered by the first source-end component after drawing the first source-end component.

[0121] In an embodiment of the present application, the first source-end component corresponding to the first UI node belongs to the first category of components. The first category of components includes custom components of the application developer and / or some special components pre-set by the system, such as surface, web, and video components. In some embodiments, the first category of components can be referred to as non-standard components. The above-mentioned first component includes at least one custom component and / or at least one special component. The drawing instruction of the first category of nodes includes the above-mentioned first program code.

[0122] It should be understood that, generally speaking, a component is used to determine at least one UI node, that is, one or more UI nodes can be obtained by creating a certain component. If a certain component belongs to a first-class component, then each UI node determined by the component belongs to a first-class node. Of course, in other embodiments, a UI node is determined by at least one component, that is, a UI node can be drawn by one or more components, and these one or more components may include first-class components and / or second-class components. In the case where the components required to draw a certain UI node include at least one first-class component, the UI node is a first-class node. It can be seen that in a specific implementation, the correspondence between components and UI nodes can be one-to-one, one-to-many or many-to-one, which can be understood as being related to the granularity of defining UI nodes.

[0123] Simply put, there are two ways of casting screens corresponding to non-standard components, as shown in Figure 8. One is that the source device synchronizes the drawing instructions (or drawing tasks) corresponding to the non-standard components to the destination device. The destination device responds to the received drawing instructions, builds and draws the same UI node, such as drawing the UI node through the same CustomPaint Node (abbreviated as CP_N) component as on the source device, and renders and displays it to present the final effect. The other is that the source device synchronizes the image rendered by the non-standard component to the destination device, and the destination device constructs the UI node through a standard component (such as the ImageNode (abbreviated as Img_N) component) and renders and displays the image of the UI node. That is, the UI attribute data of the first type of node includes the drawing instructions or rendered images corresponding to the non-standard component. It should be understood that the image rendered by the non-standard component synchronized by the source device to the destination device can be a static picture or a dynamic picture, that is, an animation. While the source device synchronizes the image rendered by the non-standard component to the destination device, it can also synchronize the corresponding control size.

[0124] The above-mentioned multiple UI nodes usually also include a second UI node, and the second UI node includes a corresponding component as a standard component. The UI attribute data of the second UI node may include UI control status data such as text, pictures, animations, videos, text attributes, cursor positions, sliding positions, colors, etc. of the corresponding node, that is, the UI attribute data of the second UI node include the drawing attribute data of the second UI node, that is, the attribute data that can be used to draw the second UI node. The component corresponding to the second UI node belongs to the second category of components, and the second category of components draws the corresponding image by the UI framework. In other words, the second category of components can be constructed and draw the corresponding image by the UI framework. That is, for standard components, what is synchronized from the source device to the destination device is the attribute data that can be used to draw the corresponding UI node. Of course, in some other embodiments, the drawing instructions of certain standard components can also be synchronized.

[0125] In an embodiment of the present application, the UI structure of multiple UI nodes in a projection area can be represented as a tree structure or other structures.

[0126] In one possible implementation, after constructing the UI structure of the first projection area, the source device generates a unique projection area identifier for the first projection area, and the projection area identifier can be recorded as a source profile identifier (SPID). In one possible implementation, the source device can also generate a unique node identifier for each UI node in the multiple UI nodes, and the node identifier can be recorded as a source frame identifier (SFID). In the case where the first projection area is the entire area of ​​the first page, the source device generates a unique page identifier for the first page, that is, in this case, the above-mentioned projection area identifier is also called a page identifier.

[0127] Step 703: The source device sends the UI structure and UI attribute data of the first projection area to the destination device.

[0128] The source device may send the UI structure and UI attribute data of the first projection area to the destination device either simultaneously or after executing step 701. The application resources of the first application and the UI structure and UI attribute data of the first page are used together on the destination device, and the first projection area is reconstructed and rendered using the UI framework of the destination device.

[0129] When the source device includes the UI proxy module and the network synchronization module shown in Figure 4, the source device can obtain the UI structure and UI attribute data of the first projection area through the UI proxy module, and send the UI structure and UI attribute data of the first projection area to the destination device through the network synchronization module.

[0130] As can be seen from the foregoing, the source device in this solution can support multiple projection modes. In this case, in one possible implementation, different projection areas may have different projection modes. Therefore, before sending the UI structure and UI attribute data of the first projection area to the destination device, the source device determines that the projection mode of the first projection area is the UI data stream mode from the multiple projection modes through a mode adaptation algorithm.

[0131] In one possible implementation, before projecting each projection area, the source device can use a mode adaptation algorithm to determine the projection mode for the current projection area from multiple projection modes. Alternatively, the source device can use a mode adaptation algorithm to determine the projection mode for the current projection area from multiple projection modes at intervals, such as 1 minute, 3 minutes, etc.

[0132] Among them, one implementation method of the source device determining that the projection mode of the first projection area is the UI data stream mode from the multiple projection modes through a mode adaptive algorithm is: based on the UI structure and UI attribute data of the first projection area, and / or the real-time computing capability information of the destination device, determining that the projection mode of the first projection area is the UI data stream mode from the multiple projection modes through a mode adaptive algorithm.

[0133] For example, the destination device sends real-time computing capability information of the destination device to the source device, and the real-time computing capability information is used by the source device to determine the projection mode of the current projection area from the multiple projection modes through a mode adaptation algorithm.

[0134] For example, when the real-time computing capability of the destination device becomes weak due to a long period of time spent performing rendering operations, the source device may determine that the projection mode of the current projection area is the mirror mode from among a plurality of projection modes.

[0135] When the input of the mode adaptation algorithm includes the UI structure and UI attribute data of the first projection area, the mode adaptation algorithm is used to analyze the UI structure and UI attribute data of the first projection area to determine whether each UI node in the UI structure can be reconstructed on the destination device through the UI data stream mode, and the amount of data of the UI structure and UI attribute data of the first projection area that needs to be transmitted in the UI data stream mode.

[0136] For example, if the component corresponding to a UI node in the second projection area is a non-standard component, and the source device analyzes the UI node and determines that the UI node cannot be reconstructed on the destination device through the UI data stream mode, then the source device can determine that the projection mode of the second projection area is mirror mode or hybrid mode. If the amount of UI structure and UI attribute data of the second projection area required to be transmitted by the UI data stream mode is greater than a threshold, the source device can determine that the projection mode of the second projection area is mirror mode or hybrid mode.

[0137] It should be understood that the source device can perform a comprehensive analysis of the UI structure and UI attribute data of the first projection area and the real-time computing capability information of the destination device through a mode adaptation algorithm to determine the projection mode of the first projection area.

[0138] It should also be understood that when the screen projection is just started, the current projection mode is determined in step 701, and during the projection process, the projection mode of the current projection area can also be determined by the mode adaptation algorithm in step 703. That is, this solution supports switching the projection mode during the projection process to reduce the power consumption of the source device and the destination device and improve the projection effect.

[0139] Figure 9 is a schematic diagram of a mode adaptation provided by an embodiment of the present application. Referring to Figure 9, the source device supports three projection modes, which are mirror mode (i.e., transmitting media stream), UI data stream mode (i.e., transmitting UI data stream) and mixed mode (i.e., transmitting media stream + UI data stream). The UI structure of the projection area constructed by the source device is a tree structure including multiple UI nodes, and the multiple UI nodes include a first type of node filled with white (corresponding to the first type of component) and a second type of node filled with black (corresponding to the second type of component) as shown in Figure 9. The source device determines the projection mode of the projection area from the above three projection modes through UI structure analysis and mode adaptation, and projects the projection area according to the determined projection mode.

[0140] Among them, in the mirror mode, the source device needs to encode and form a media stream (i.e., a video stream) and send it, and the destination device needs to receive and decode the media stream. In the UI data stream mode, the source device needs to build and update the UI structure of the projection area, and the destination device needs to rebuild and update the projection area, that is, to rebuild and update the UI. In the hybrid mode, the source device needs to encode and send part of the rendered image of the projection area (such as the image rendered by the first type of node), as well as transmit the UI structure of the projection area and the UI attribute data of the rest of the projection area (such as the UI attribute data of the second type of node). That is, the source device needs to perform stream encoding and UI construction and update. The destination device needs to decode part of the image of the projection area and reconstruct the rest of the projection area. That is, the destination device needs to perform stream decoding and UI reconstruction and update.

[0141] In one possible implementation, if the source device supports multiple projection modes, the source device can also display the current projection mode, which is one of the multiple projection modes. The embodiments of the present application do not limit the manner in which the source device displays the current projection mode, including the display position, display font, display color, etc. of the current projection mode.

[0142] In one possible implementation, the source device sends the UI structure and UI attribute data of the first projection area to the destination device, and also sends the identifier of the first projection area, such as the SPID of the first projection area. The UI attribute data sent by the source device includes the identifiers of multiple UI nodes in the first projection area, such as the SFID of each of the multiple UI nodes.

[0143] In one possible implementation, the source device can serialize the data to be sent (including the UI nodes and UI attribute data of the first projection area) to obtain serialized page data and send the serialized page data to the destination device. Among them, serialization processing can refer to assembling (or encoding) into a string or binary array (such as a binary Buffer) or a file. Of course, the source device can also process the data to be sent according to other data processing methods, and the embodiments of the present application are not limited to this.

[0144] In a specific implementation, the source device may send a page reconstruction message to the destination device, where the page reconstruction message carries the above serialized page data.

[0145] Step 704: The destination device receives the application resources of the first application and the UI structure and UI attribute data of the first projection area.

[0146] In the case where the destination device includes the network synchronization module shown in Figure 4, the destination device receives the application resources and the UI structure and UI attribute data of the first projection area sent by the source device through the network synchronization module.

[0147] In one possible implementation, when the source device sends serialized page data, the destination device deserializes the received serialized page data to obtain the application resources of the first application and the UI structure and UI attribute data of the first projection area. The deserialization process of the destination device is an operation that matches and is symmetrical with the serialization process of the source device. For example, the deserialization process refers to decoding the serialized page data.

[0148] Step 705: The destination device displays the first projection area based on the application resources and the UI structure and UI attribute data of the first projection area.

[0149] In an embodiment of the present application, the destination device loads the UI structure through the UI framework of the destination device to layout the above-mentioned multiple UI nodes in the first display area, and sets the properties of each UI node in the multiple UI nodes based on the UI attribute data to reconstruct and render the first projection area. That is, the destination device can reconstruct and render the first projection area through the UI framework. Among them, the destination device can layout the above-mentioned multiple UI nodes in the first display area by creating multiple UI nodes in the first blank page. That is, the above-mentioned multiple UI nodes are carried by the first blank page.

[0150] In the case where the destination device includes the UI proxy module and the UI display module shown in Figure 4, the destination device constructs and renders the first projection area through the UI proxy mode in the UI framework, and displays the rendered first projection area through the UI display module.

[0151] Among them, the first display area can be part or all of the display area of ​​the destination device, that is, the first display area is the full-screen area or partial screen area of ​​the destination device. It should be noted that the first projection area of ​​the source device can also be part or all of the display area of ​​the source device. In the case where the source device displays the first projection area on the full screen, the destination device can display the first projection area on the full screen or not fully, such as when the destination device is provided with a display area occupied by the projection screen. Similarly, in the case where the source device does not display the first projection area on the full screen, the destination device can display the first projection area on the full screen or not fully.

[0152] As can be seen above, the source device synchronizes different data for Type 1 and Type 2 nodes. The following details how the destination device reconstructs Type 1 and Type 2 nodes.

[0153] First, the reconstruction method of the first type of nodes (corresponding to non-standard components) is introduced.

[0154] Taking the first UI node among the multiple UI nodes as an example, which belongs to the first type of node, when the UI attribute data includes the drawing instructions of the first source-end component, the destination device loads the above-mentioned UI structure through the UI framework to construct (or layout) the multiple UI nodes, including creating the first destination-end component to obtain the first UI node, and drawing the image of the first destination-end component based on the drawing instructions of the first source-end component to obtain an image not rendered by the first source-end component.

[0155] Among them, the first destination component is a system pre-installed component of the destination device, for example, the first destination component includes the first system pre-installed component of the destination device. In a specific implementation, the first destination component includes one or more standard components, special components or components provided by other frameworks. The first destination component has the same or similar functions as the first source component mentioned above. The type of the first destination component can be the same as the type of the first source component. As shown in Figure 8, the first source component and the first destination component are both CustomPaint Node components. Of course, the type of the second component can also be different from that of the first component.

[0156] In the case where the UI attribute data includes an image rendered by the first source-end component, the destination device loads the above-mentioned UI structure through the UI framework to construct the multiple UI nodes, which includes creating a second destination-end component to obtain the first UI node, and based on the image rendered by the first source-end component, by setting the UI attributes of the second destination-end component, so that the first UI node carries the image rendered by the first source-end component.

[0157] It should be understood that the image rendered by the first source-side component includes rendered pixels and may also include properties such as control size and animation. The destination device sets the properties of the second destination-side component to set the rendered pixels, control size, animation, and other properties of the first UI node into the first UI node, thereby obtaining the image rendered by the first source-side component.

[0158] Among them, the second destination component is also a system preset component of the destination device, for example, the second destination component includes the second system preset component of the destination device. In a specific implementation, the second destination component may also include one or more standard components, special components or components provided by other frameworks. The second destination component and the first source component may have different functions, and the type of the second destination component may be different from the type of the first source component, that is, the second destination component is different from the first source component. For example, the first source component is the CustomPaint Node component shown in Figure 8, and the second destination component is the ImageNode component shown in Figure 8.

[0159] Next, we introduce the reconstruction method of the second type of nodes (corresponding to standard components).

[0160] Taking the second UI node among the multiple UI nodes as an example, which belongs to the second type of node, as can be seen from the above, the UI attribute data of the second UI node may include the drawing attribute data of the second UI node, such as the UI control state data such as the text, picture, animation, video, text attributes, cursor position, sliding position, color, etc. of the second UI node. The destination device loads the above UI structure through the UI framework to construct the multiple UI nodes, which includes creating a third destination component to obtain the second UI node, and setting the attributes of the third destination component through the UI framework based on the drawing attribute data of the second UI node to draw the corresponding image of the second UI node.

[0161] The third destination component is of the same type as the component used by the source device to draw the second UI node (hereinafter referred to as the second source component). For example, the third destination component and the second source component are the same standard component. Of course, in some possible embodiments, the third destination component and the second source component can also be of different types, as long as they have the same or similar functions.

[0162] In one possible implementation, after constructing the UI structure of the first projection area, the destination device generates a unique projection area identifier for the first projection area, which can be recorded as a destination profile identifier (DPID). The destination device records the correspondence between the SPID and the DPID of the first projection area, such as storing the SPID and the DPID of the first projection area in a page mapping table. In the case where the first projection area is the entire area of ​​the first page, the destination device generates a unique page identifier for the first page, that is, in this case, the above-mentioned projection area identifier is also called a page identifier.

[0163] In one possible implementation, the destination device may further generate a unique node identifier for each of the multiple UI nodes constructed, which may be recorded as a destination frame identifier (DFID). The destination device records the correspondence between the SFID and the DFID of each UI node, for example, by storing the correspondence between the SFID and DFID of each UI node in a node mapping table.

[0164] The above describes the implementation process of projecting the first projection area from the source device to the destination device through steps 701 to 705. Next, the projection process is described when the first projection area is updated, that is, the UI structure and / or UI attribute data of the first projection area is updated, and when the first projection area is switched to the second projection area.

[0165] When the UI structure and / or UI attribute data of the first projection area is updated, the source device sends the UI update data of the first projection area to the destination device, and the UI update data is used by the destination device to update the first projection area. The destination device receives the UI update data of the first projection area sent by the source device and updates the first page based on the UI update data of the first projection area.

[0166] There are at least three situations in which the first projection area is updated: the first is when the destination device controls the source device, the second is when the user operates the source device, and the third is when the source device automatically updates.

[0167] In the first case where the first projection area is updated (i.e., the destination device controls the source device), the destination device sends a first manipulation message to the source device in response to the destination user's first operation on the first projection area, where the first manipulation message is used to instruct the source device to perform a second operation on the first projection area and update the UI structure and / or UI attribute data of the first projection area. The source device receives the first manipulation message sent by the destination device, where the first manipulation message indicates the destination user's first operation on the first projection area. Based on the first manipulation message, the source device performs the second operation on the first projection area and updates the UI structure and / or UI attribute data of the first projection area.

[0168] Among them, the second operation may be the same as or different from the first operation, and the objects of the second operation and the first operation may be the same or different. For example, in one example, the first projection area displayed by the destination device is playing a video, and the destination user long presses the right area of ​​the video screen to fast forward or rewind the video. After the long press operation of the destination user is reflected on the source device, the source device performs the fast forward or rewind operation, that is, in this example, the first operation is a long press operation, and the object of the first operation is the UI node corresponding to the right area of ​​the video screen. The second operation is a fast forward or rewind operation, and the object of the second operation is the UI node corresponding to the progress bar. In another example, the first projection area displayed by the destination device includes a pause button. The destination user clicks the pause button. After the click operation is reflected on the source device, the source device also performs the same click operation, that is, in this example, the first operation and the second operation are the same, both are click operations, and the objects of the first operation and the second operation are the same, both are UI nodes corresponding to the button.

[0169] Based on this, the first manipulation information sent by the destination device to the source device may include different information according to actual conditions, which will be described below.

[0170] As can be seen from the above, the UI structure of the first projection area includes multiple UI nodes, and the UI attribute data includes the identifiers of the multiple UI nodes. In some cases, the first control information includes the identifier of the third UI node and the control instruction corresponding to the first operation. The control instruction indicates the second operation, and the third UI node is one of the multiple UI nodes. One implementation method for the source device to perform the second operation on the first projection area based on the first control information is: based on the identifier of the third UI node, according to the control instruction, perform the second operation on the third UI node. Specifically, the source device searches for the third UI node from the multiple UI nodes based on the identifier of the third UI node, and performs the second operation on the found third UI node according to the control instruction.

[0171] Exemplarily, the destination user clicks on the first UI control (i.e., the third UI node) in the first projection area displayed on the destination device, triggering a click event on the first UI control. The destination device responds to the destination user's click operation by sending the identifier of the first UI control and a click instruction corresponding to the click event to the source device. Based on the identifier of the first UI control, the source device searches for the first UI control from multiple UI nodes and performs a click operation on the first UI control according to the click instruction.

[0172] If the destination device records the correspondence between the SFID and DFID of each UI node, after detecting a first user operation on a third UI node, the destination device searches for the SFID of the third UI node from the correspondence between SFIDs and DFIDs based on the DFID of the third UI node. The destination device obtains a manipulation instruction corresponding to the first operation and sends the SFID of the third UI node and the manipulation instruction corresponding to the first operation to the source device. The source device searches for the third UI node from multiple UI nodes based on the SFID of the third UI node.

[0173] In other cases, the first manipulation information includes an identifier of a fourth UI node and operation information of the first operation, and the fourth UI node is one of the multiple UI nodes. Another implementation method for the source device to perform the second operation on the first projection area based on the first manipulation information is to parse the operation information of the first operation to determine the second operation, and perform the second operation on the fourth UI node based on the identifier of the fourth UI node. The source device can find the fourth UI node from the multiple UI nodes based on the SFID of the fourth UI node.

[0174] Exemplarily, the destination user long presses the right area of ​​the video screen in the first projection area, triggering a drag operation on the progress bar (i.e., the fourth UI node). The destination device responds to the destination user's long press operation by sending the identifier of the fourth UI node and the operation information of the drag operation to the source device. The source device parses the operation information of the drag operation to determine the fast forward or fast rewind operation, and performs the fast forward or fast rewind operation on the progress bar based on the identifier of the progress bar.

[0175] In the second case where the first projection area is updated (i.e., the user operates the source device), the source device updates the UI structure and / or UI attribute data of the first page in response to the source user's third operation on the first projection area.

[0176] It should be understood that the destination user is the user who operates the destination device, and the source user is the user who operates the source device. The destination user and the source user may be the same user or different users.

[0177] In the third case where the first projection area is updated (i.e., the source device is automatically updated), the source device may automatically update the first projection area based on the application code or application default settings, for example, the source device automatically switches the scrolling image in the first projection area.

[0178] The first scenario where the first projection area is updated is triggered by the destination device, while the second and third scenarios are triggered by the source device. Therefore, the second and third scenarios can be classified into the same category, namely, the source device triggering the update. In other words, the three scenarios can be considered as two categories: one where the destination device triggers the update, and the other where the source device triggers the update.

[0179] In order to reduce the amount of UI data transmitted from the source device to the destination device, reduce the lag of the destination device, and improve the screen projection effect, when the first projection area is updated, the source device can synchronize the changed part of the first projection area to the destination device, and the unchanged part will not be sent repeatedly. That is, the UI update data of the first projection area indicates the changed UI nodes and UI attribute data among the multiple UI nodes. The destination device updates the changed UI nodes in the first projection area based on the UI update data of the first projection area.

[0180] Exemplarily, the first UI node in the first projection area is a UI icon. After the user clicks the first UI node on the source device, the color or transparency of the first UI node changes, for example, the color changes from gray to blue. Then, the UI update data of the first projection area includes the identifier of the first UI node and the updated color information.

[0181] The source device may serialize the UI update data of the first projection area to obtain serialized page update data and send it to the destination device. The destination device deserializes the received serialized page update data to obtain the UI update data of the first projection area. In addition, the source device may send a page update message to the destination device, and the page update message carries the serialized page update data.

[0182] Taking the first projection area as the entire area of ​​the first page in the first application as an example, Figure 10 is a projection flow chart for a page update provided by an embodiment of the present application. Figure 10 shows the above two types of page update situations.

[0183] Referring to Figure 10 , in the first scenario, a user operates a destination device. In response to the user operation, the destination device determines the DFID of the UI node corresponding to the user operation and, based on the UI node's DFID, searches the node mapping table for the UI node's SFID. The destination device then obtains the manipulation instruction corresponding to the manipulation event for the UI node and sends the UI node's SFID and the corresponding manipulation instruction to the source device. After receiving the UI node's SFID and the corresponding manipulation instruction, the source device locates the corresponding UI node based on the UI node's SFID and performs the corresponding operation on the UI node according to the manipulation instruction, injecting a manipulation event into the UI node. The source device updates the logic of the first application to determine whether a page switch has occurred, i.e., whether a new page has been entered. If a new page has been entered, the source device executes the UI construction process for the new page. If a new page has not been entered, the source device obtains updated data for the UI structure and UI attribute data of the current page, serializes the obtained updated data to obtain serialized page update data, and sends a page update message to the destination device, carrying the serialized page update data. The destination device receives and parses the page update message to obtain serialized page update data, deserializes the serialized page update data to obtain the updated data for the current page. Based on the updated data for the current page, the destination device updates the UI structure of the current page and the properties of the corresponding UI nodes. The destination device then refreshes the displayed page.

[0184] In the second scenario, the user operates the source device, which responds by updating the logic of the first application and determining whether a page switch has occurred, i.e., whether a new page has been entered. Alternatively, the source device updates the logic of the first application according to the application code or default settings, and after the update, determines whether a page switch has occurred. A similar process to the first scenario is then followed to ultimately refresh the displayed page on the destination device.

[0185] When the projection area switches from the first projection area to the second projection area (for example, the first page switches to the second page), the source device constructs the UI structure of the second projection area and generates UI attribute data of the second projection area, and sends the UI structure and UI attribute data of the second projection area to the destination device. The above-mentioned application resources and the UI structure and UI attribute data of the second projection area are used by the destination device to reconstruct and render the second projection area through the UI framework. That is, the source device performs operations similar to the above-mentioned steps 702 and 703 to synchronize the UI structure and UI attribute data of the second projection area to the destination device. The destination device receives the UI structure and UI attribute data of the second projection area, and based on the above-mentioned application resources and the UI structure and UI attribute data of the second projection area, reconstructs and renders the second projection area through the UI framework of the destination device. That is, the destination device performs operations similar to the above-mentioned steps 704 and 704 to switch the projection area.

[0186] Here, taking the switching from the first page to the second page as an example, page switching includes page jump, page rollback, etc.

[0187] Next, please refer to Figures 11 and 12 to exemplify the screen projection method provided by the embodiment of the present application. It should be noted that Figure 11 is an example of an example in which the screen projection area is the entire area of ​​the page.

[0188] FIG11 is a flow chart of another screen projection method provided in an embodiment of the present application. Referring to FIG11 , the method includes the following steps.

[0189] Step 1101: The source device executes the application logic portion of the first application.

[0190] Step 1102: The source device constructs a UI page, that is, constructs the page to be projected by loading its own system UI framework.

[0191] Step 1103: The source device obtains the UI structure of the constructed UI page. The UI structure can be a tree structure or a graph structure. The UI structure includes parent-child relationships, layout data, layer relationships, and other information between multiple UI nodes. In embodiments of the present application, the UI structure of a page can be obtained by directly obtaining the page structure information of the UI page or by obtaining UI creation instructions. The source device may also generate and record the SPID of the constructed UI page.

[0192] Step 1104: The source device obtains UI attribute data for the constructed UI page. This UI attribute data includes the identifier, type, and status of each UI node in the UI page. Status data includes UI control status, such as text, images, media, rendered images or drawing instructions / tasks corresponding to non-standard components, text attributes, cursor position, slide position, text, color, control size, and animation. The source device may also generate and record the SFID for each UI node.

[0193] Step 1105: The source device serializes the data to be sent (including the UI structure and UI attribute data of the page to be projected) to obtain serialized page data. Serialization can refer to assembling (or encoding) into a string or binary buffer or file.

[0194] Step 1106: The source device sends a page reconstruction message to the destination device via the network. The page reconstruction message carries the serialized page data.

[0195] Step 1107: The destination device receives and parses the page reconstruction message from the source device to obtain serialized page data.

[0196] Step 1108: The destination device deserializes the serialized page data to obtain the UI structure and UI attribute data of the current page.

[0197] Step 1109: The destination device reconstructs the UI structure of the page through the system UI framework based on the UI structure of the current page or the UI creation instruction, and generates the DPID of the page. In one possible implementation, the destination device records the DPID in the page mapping table.<SPID,DPID> .

[0198] Step 1110: The destination device sets the properties of each UI node in the current page based on the UI property data of the current page, including creating or configuring the UI node type and setting the UI property data to the UI node. For non-standard components, the destination device may draw the corresponding UI node based on the drawing instructions corresponding to the non-standard components, or set the corresponding rendered image, control size or animation on the corresponding UI node. The destination device may also generate and record the DFID of each UI node. In one possible implementation, the destination device records the DFID in the node mapping table.<SFID,DFID> .

[0199] Step 1111: The destination device refreshes the displayed page, i.e., renders the displayed page, for example, by displaying the page through a UI display module. The destination device can dynamically set the size of the display window, so that the page can adapt to the window size when displayed in the display window.

[0200] It should be noted that if the projection mode of the first page in the first application is UI data stream mode or mixed mode, then the source device has already sent the application resources of the first application to the destination device before projecting the second page. In this case, there is no need to repeatedly send the application resources of the first application to the destination device during the projection process after the first page. The embodiment of Figure 11 above shows the projection process of a page after the first page.

[0201] In an embodiment of the present application, after the screen projection is completed, for example, after the screen projection connection between the source device and the destination device is disconnected, the destination device may choose to delete all or part of the received data, for example, the destination device releases the UI display module. In other words, after the screen projection is completed, the source device may recycle or manage the resources of the other end. In one possible implementation, the destination device determines whether to delete the received data based on the security level and / or user settings.

[0202] Exemplarily, after a screen projection connection is established between a source device and a destination device, a first prompt message is displayed on the source device, and the first prompt message is used to instruct the user to choose whether to trust the destination device. If the user chooses to trust the destination device by operating the source device, the destination device does not delete the received data after the screen projection is completed. If the user chooses not to trust the destination device by operating the source device, the destination device deletes the received data after the screen projection is completed. Alternatively, after the screen projection is completed, the source device displays a second prompt message, and the second prompt message instructs the user whether to choose to delete the data received by the destination device. If the user chooses to delete by operating the source device, the destination device deletes the received data after the screen projection is completed. If the user chooses not to delete by operating the source device, the destination device does not delete the received data after the screen projection is completed.

[0203] Figure 12 is a schematic diagram of the core steps of a screen projection method provided by an embodiment of the present application. Figure 12 is a schematic diagram obtained by combining this solution with the hardware and software architecture of the electronic device shown in Figure 4. Referring to Figure 12, in one implementation of the screen projection method, the source device and the destination device both include the following five steps: negotiate the current screen projection mode through interactive hardware and software capabilities and operating system and UI framework versions; synchronize application resources and configuration data; build and synchronize UI structures; update UI nodes / events; disconnect the screen projection connection, and recycle or manage destination resources. Among them, the first and fifth steps can be implemented through the service management modules at both ends, and the second to fourth steps can be implemented through the UI frameworks (including UI proxy modules) and network synchronization modules at both ends.

[0204] To verify the performance of the UI data flow model provided in the embodiments of this application, a comparative test of the UI data flow model and related technologies was conducted on the open source OpenHarmony system and the ArkUI framework. The test results are shown in Table 1.

[0205] Table 1

[0206] Among them, the test indicators include network transmission data volume, reverse control delay, source power consumption and image quality. The mirror mode in the related technology requires video encoding to transmit video streams (also called media streams). I frames and P frames represent key frames and non-key frames in video encoding, respectively. Each I frame is about 50-100 kilobytes (KB), and each P frame is about 10-20KB. Based on the number of I frames and P frames that can be transmitted per second in mirror mode, the network transmission data volume is about 1-2 megabytes per second (MB / s). The network transmission data volume of the UI data flow mode in this solution is about 10-100KB / page in the case of UI reconstruction, and the network transmission data volume is basically less than 1KB / page in the case of page update. Furthermore, mirror mode requires transmission of every frame when the screen changes, while UI data stream mode only transmits data when the UI attribute data of the page is updated or the page is switched. For example, if a page includes a GIF and the screen of the page is constantly changing due to the GIF, then mirror mode requires transmission of every frame, while UI data stream mode only needs to transmit the UI attribute data of the GIF through the UI data stream after the UI node of the GIF is first created. There is no need to transmit the UI-related data of the GIF later. As can be seen, the amount of network transmission data in UI data stream mode is greatly reduced compared to mirror mode.

[0207] Reverse control latency refers to the time from when a user performs an action on the destination device to when the destination device updates the page. In related technologies, the reverse control latency in office collaboration applications is approximately 110-120 milliseconds (ms). In this solution, the reverse control latency in the UI data stream mode is approximately 60-80ms. This shortens the reverse control latency, resulting in faster response times on the destination device and reduced lag.

[0208] Source power consumption refers to the power consumption of the source device. In related art, the source power consumption in shopping and office applications is approximately 110 milliamperes (mA), and in video applications is approximately 220mA. In other words, in related art, the source power consumption is approximately 110-220mA. In this solution, the source power consumption in UI data streaming mode is approximately 30-70mA.

[0209] When the screen resolution of the source device is 2K and the refresh rate is 60 Hz, the target device in mirroring mode in related technologies cannot adapt to the resolution, resulting in lag or screen distortion. However, this solution can achieve native 2K resolution and 60 Hz refresh rate on the target device through screen resolution adaptation without lag or screen distortion.

[0210] Figure 13 is a schematic diagram of the screen projection area on the source device provided in an embodiment of the present application. Figure 14 is a schematic diagram of a screen projection effect on the destination device in this solution provided in an embodiment of the present application. Figure 15 is a schematic diagram of a screen projection effect in mirror mode on the destination device in the related technology provided in an embodiment of the present application. It can be clearly seen from Figures 13 to 15 that the picture after screen projection in this solution will not be stretched, while the picture after screen projection in the related technology will be stretched. This is precisely because in this solution, the destination device layouts and renders the various UI nodes in the display screen by itself through UI reconstruction, and the reconstructed UI nodes adapt to the screen resolution of the destination device.

[0211] Figure 16 is another schematic diagram of the projection area of ​​the source device in this solution provided by an embodiment of the present application. Figure 17 is a schematic diagram of a projection area of ​​the destination device corresponding to Figure 16 in this solution provided by an embodiment of the present application. It can be seen from Figures 16 and 17 that in this solution, there may be some differences between the source device and the destination device for the layout of the above-mentioned multiple UI nodes, that is, the destination device can adaptively adjust the layout of the multiple UI nodes according to information such as screen resolution. For example, there is a difference in the display position of certain UI nodes on the source device and the destination device, such as being displayed in the center on the source device and displayed on the right on the destination device.

[0212] In summary, in an embodiment of the present application, the source device sends the UI structure and UI attribute data of the projection area to the destination device, so that the destination device can rebuild and render the corresponding projection area based on the UI structure and UI attribute data at the UI level. This solution does not require the source device to perform video encoding and transmission after rendering the image, which reduces the power consumption of the source device, and does not require the source device to send user-related data to the destination device, reducing the risk of user privacy data leakage. Moreover, this solution can also achieve screen projection when the destination device does not have an application that requires screen projection installed, thereby supporting multi-device collaboration. In addition, the destination device renders and displays the projection area by itself, which can adapt to the screen resolution of the destination device and improve the display effect.

[0213] Figure 18 is a schematic diagram of the structure of a screen projection device provided in an embodiment of the present application. The screen projection device can be implemented as part or all of an electronic device by software, hardware, or a combination of both. The electronic device can be the electronic device shown in Figure 5. In this embodiment of the present application, the screen projection device is included in the source device. Referring to Figure 18, the device includes: a sending module 1801.

[0214] The sending module 1801 is used to send the user interface UI structure and UI attribute data of the first projection area to the destination device to project the first projection area to the destination device;

[0215] Among them, the UI structure indicates the layout relationship between multiple UI nodes in the first projection area, the UI attribute data indicates the rendering-related attributes of each UI node in the multiple UI nodes, and the UI structure and UI attribute data are used to display the first projection area on the target device.

[0216] In a possible implementation, the device further includes:

[0217] The construction module is used to build the UI structure and generate UI attribute data through the UI framework of the source device according to the application instructions.

[0218] In a possible implementation, the multiple UI nodes include a first UI node corresponding to a first source-end component, the first source-end component belongs to a first-category component, and a corresponding image of the first-category component is drawn by a non-UI framework;

[0219] The UI attribute data includes a drawing instruction of the first source-end component, where the drawing instruction of the first source-end component is used to draw an image not rendered by the first source-end component through a non-UI framework; or, the UI attribute data includes an image rendered by the first source-end component.

[0220] In a possible implementation, the multiple UI nodes include a second UI node corresponding to a second source-end component, the second source-end component belongs to a second-category component, and a corresponding image is drawn by the UI framework for the second-category component;

[0221] The UI attribute data includes drawing attribute data of the second source-end component, and the drawing attribute data of the second source-end component is used to draw a corresponding image of the second source-end component through the UI framework.

[0222] In a possible implementation, the source device runs a first application, the first projection area is the projection area of ​​the first application, and the sending module 1801 is further configured to:

[0223] The application resources of the first application are sent to the destination device, where the application resources include the resources required to render the multiple UI nodes. The application resources and the UI attribute data are used together by the destination device to reconstruct and render the first projection area.

[0224] In a possible implementation, the device further includes:

[0225] a receiving module, configured to receive first manipulation information sent by a destination device, where the first manipulation information indicates a first operation performed by a destination user on a first projection area;

[0226] An updating module, configured to perform a second operation on the first projection area based on the first manipulation information, and update a UI structure and / or UI attribute data of the first projection area;

[0227] The sending module 1801 is also used to send UI update data of the first projection area to the destination device. The UI update data is used by the destination device to update the first projection area through the UI framework of the destination device.

[0228] In one possible implementation, the UI attribute data includes identifiers of the multiple UI nodes, the first manipulation information includes an identifier of a third UI node and a manipulation instruction corresponding to the first operation, the manipulation instruction indicates the second operation, and the third UI node is one of the multiple UI nodes;

[0229] The update module is specifically configured to: perform a second operation on the third UI node based on the identifier of the third UI node and in accordance with the manipulation instruction.

[0230] In one possible implementation, the UI attribute data includes identifiers of the multiple UI nodes, the first manipulation information includes an identifier of a fourth UI node and operation information of the first operation, and the fourth UI node is one of the multiple UI nodes;

[0231] The update module is specifically used to:

[0232] parsing the operation information of the first operation to determine the second operation;

[0233] Based on the identifier of the fourth UI node, a second operation is performed on the fourth UI node.

[0234] In one possible implementation, the source device runs a first application, the first projection area is a projection area on a first page of the first application, and the sending module 1801 is further configured to:

[0235] When the first application switches from the first page to the second page, the UI structure and UI attribute data of the second projection area are sent to the destination device to project the second projection area to the destination device. The second projection area is the projection area in the second page.

[0236] In one possible implementation, the source device supports multiple screen projection modes; the apparatus further includes:

[0237] The display module is used to display the current projection mode, which is one of multiple projection modes.

[0238] In one possible implementation, the source device supports multiple screen projection modes, and different screen projection modes exist in different screen projection areas; the apparatus further includes:

[0239] The determination module is used to determine the projection mode of the first projection area from multiple projection modes through a mode adaptation algorithm.

[0240] In an embodiment of the present application, the source device sends the UI structure and UI attribute data of the projection area to the destination device, so that the destination device can rebuild and render the corresponding projection area based on the UI structure and UI attribute data at the UI level. This solution does not require the source device to perform video encoding and transmission after rendering the image, which reduces the power consumption of the source device, and does not require the source device to send user-related data to the destination device, reducing the risk of user privacy data leakage. Moreover, this solution can also achieve projection when the destination device does not have an application that requires projection installed, thereby supporting multi-device collaboration. In addition, the projection area is rendered and displayed by the destination device itself, which can adapt to the screen resolution of the destination device and improve the display effect.

[0241] It should be noted that the screen projection device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when projecting the screen. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the screen projection device provided in the above embodiment and the screen projection method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0242] Figure 19 is a schematic diagram of the structure of another screen projection device provided in an embodiment of the present application. The screen projection device can be implemented as part or all of an electronic device by software, hardware, or a combination of both. The electronic device can be the electronic device shown in Figure 5. In this embodiment of the present application, the device is included in the destination device. Referring to Figure 19, the device includes: a receiving module 1901 and a display module 1902.

[0243] Receiving module 1901, configured to receive a user interface UI structure and UI attribute data of a first projection area sent by a source device, where the UI structure indicates a layout relationship between multiple UI nodes in the first projection area, and the UI attribute data indicates rendering-related attributes of each UI node in the multiple UI nodes;

[0244] Display module 1902 is used to display the first projection area based on the UI structure and UI attribute data.

[0245] In a possible implementation, the display module 1902 is specifically configured to:

[0246] Loading a UI structure through a UI framework of a destination device to layout a plurality of UI nodes in a first display area;

[0247] The attributes of each UI node in the plurality of UI nodes are set based on the UI attribute data to reconstruct and render the first projection area.

[0248] In a possible implementation, the multiple UI nodes include a first UI node corresponding to a first source-end component, the first source-end component belongs to a first-category component, and a corresponding image of the first-category component is drawn by a non-UI framework;

[0249] The UI attribute data includes a drawing instruction of the first source-end component, where the drawing instruction of the first source-end component is used to draw an image not rendered by the first source-end component through a non-UI framework; or, the UI attribute data includes an image rendered by the first source-end component.

[0250] In a possible implementation, when the UI attribute data includes a drawing instruction of the first source-end component, the display module 1902 is specifically configured to:

[0251] Loading the UI structure through the UI framework to create a first destination component to obtain a first UI node;

[0252] Based on the drawing instruction of the first source-end component, the image of the first destination-end component is drawn through the non-UI framework to obtain an unrendered image of the first UI node.

[0253] In one possible implementation, when the UI attribute data includes an image rendered by the first source-end component, the display module 1902 is specifically configured to:

[0254] Creating a second destination component by loading the UI structure through the UI framework to obtain the first UI node, where the second destination component is different from the first source component;

[0255] Based on the image rendered by the first source-end component, the UI property of the second destination-end component is set so that the first UI node carries the image rendered by the first source-end component.

[0256] In one possible implementation, the multiple UI nodes include a second UI node corresponding to a second source-end component, the second source-end component belongs to a second-category component, a corresponding image is drawn by a UI framework for the second-category component, and the UI attribute data includes drawing attribute data of the second source-end component;

[0257] The display module 1902 is specifically used for:

[0258] Loading the UI structure through the UI framework creates a third destination component to obtain a second UI node, where the third destination component is the same as the second source component;

[0259] Based on the drawing attribute data of the second source-end component, the attributes of the second source-end component are set through the UI framework to draw a corresponding image of the second UI node.

[0260] In a possible implementation, the device further includes:

[0261] A sending module, configured to send first manipulation information to a source device in response to a first operation performed by a destination user on the first projection area, wherein the first manipulation information is used to instruct the source device to perform a second operation on the first projection area and update the UI structure and / or UI attribute data of the first projection area;

[0262] The receiving module 1901 is further configured to receive UI update data of the first projection area sent by the source device;

[0263] The update module is used to update the first projection area based on the UI update data of the first projection area.

[0264] In one possible implementation, the source device supports multiple screen projection modes, and different screen projection modes exist in different screen projection areas; the apparatus further includes:

[0265] The sending module is used to send the real-time computing capability information of the destination device to the source device. The real-time computing capability information is used by the source device to determine the projection mode of the current projection area from multiple projection modes through a mode adaptation algorithm.

[0266] In an embodiment of the present application, the source device sends the UI structure and UI attribute data of the projection area to the destination device, so that the destination device can rebuild and render the corresponding projection area based on the UI structure and UI attribute data at the UI level. This solution does not require the source device to perform video encoding and transmission after rendering the image, which reduces the power consumption of the source device, and does not require the source device to send user-related data to the destination device, reducing the risk of user privacy data leakage. Moreover, this solution can also achieve projection when the destination device does not have an application that requires projection installed, thereby supporting multi-device collaboration. In addition, the projection area is rendered and displayed by the destination device itself, which can adapt to the screen resolution of the destination device and improve the display effect.

[0267] It should be noted that the screen projection device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when projecting the screen. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the screen projection device provided in the above embodiment and the screen projection method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0268] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of the present application may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0269] It should be understood that the "at least one" mentioned herein refers to one or more, and "a plurality of" refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0270] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0271] The above description is an embodiment provided for this application and is not intended to limit this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A screen projection system, characterized in that: The system includes a source-end screen projection module and a destination-end screen projection module; The source-end screen projection module is used to send the user interface UI structure and UI attribute data of the first screen projection area to the destination-end screen projection module, so as to project the first screen projection area from the source device where the source-end screen projection module is located to the destination device where the destination-end screen projection module is located, wherein the UI structure indicates the layout relationship between multiple UI nodes in the first screen projection area, and the UI attribute data indicates the rendering-related attributes of each UI node in the multiple UI nodes; The target-end screen projection module is used to receive the UI structure and UI attribute data, and display the first screen projection area on the target device based on the UI structure and the UI attribute data.

2. The system according to claim 1, characterized in that The target end screen projection module is specifically used for: The UI structure is loaded through the UI framework of the target device to layout the multiple UI nodes in the first display area, and the properties of each UI node in the multiple UI nodes are set based on the UI property data to reconstruct and render the first projection area.

3. The system according to claim 1 or 2, characterized in that: The source-end screen projection module is also used to construct the UI structure and generate the UI attribute data through the UI framework of the source device according to application instructions.

4. The system according to any one of claims 1 to 3, characterized in that: The destination-end screen projection module is also used to send first control information to the source-end screen projection module, so as to update part or all of the UI nodes in the first screen projection area by reversely controlling the source device.

5. The system according to any one of claims 1 to 4, characterized in that: The source device runs a first application, and the first screen projection area is the screen projection area of ​​the first application.

6. The system according to claim 5, characterized in that The destination device does not have the first application installed.

7. The system according to any one of claims 1 to 6, characterized in that: The first projection area is part or all of the area of ​​the first page.

8. A screen projection method, characterized in that: Applied to a source device, the method comprises: Sending the user interface UI structure and UI attribute data of the first projection area to the destination device to project the first projection area to the destination device; Among them, the UI structure indicates the layout relationship between multiple UI nodes in the first projection area, the UI attribute data indicates the rendering-related properties of each UI node among the multiple UI nodes, and the UI structure and the UI attribute data are used to display the first projection area on the destination device.

9. The method according to claim 8, characterized in that Before sending the UI structure and UI attribute data of the first projection area to the destination device, the method further includes: According to the application instruction, the UI structure is constructed and the UI attribute data is generated through the UI framework of the source device.

10. The method according to claim 8 or 9, characterized in that The multiple UI nodes include a first UI node corresponding to a first source-end component, the first source-end component belongs to a first type of component, and a corresponding image of the first type of component is drawn by a non-UI framework; The UI attribute data includes a drawing instruction of the first source-end component, where the drawing instruction of the first source-end component is used to draw an image not rendered by the first source-end component through the non-UI framework; or, The UI attribute data includes an image rendered by the first source-end component.

11. The method according to any one of claims 8 to 10, characterized in that: The multiple UI nodes include a second UI node corresponding to a second source-end component, the second source-end component belongs to a second type of component, and the second type of component has a corresponding image drawn by the UI framework; The UI attribute data includes drawing attribute data of the second source-end component, and the drawing attribute data of the second source-end component is used to draw a corresponding image of the second source-end component through the UI framework.

12. The method according to any one of claims 8 to 11, characterized in that: The source device runs a first application, the first screen projection area is a screen projection area of ​​the first application, and the method further includes: Send application resources of the first application to the destination device, the application resources including resources required for rendering the multiple UI nodes, and the application resources and the UI attribute data are used together by the destination device to reconstruct and render and display the first projection area.

13. The method according to any one of claims 8 to 12, characterized in that: The method further comprises: Receiving first manipulation information sent by the destination device, where the first manipulation information indicates a first operation of a destination user on the first projection area; Based on the first manipulation information, perform a second operation on the first projection area, and update the UI structure and / or UI attribute data of the first projection area; Send UI update data of the first screen projection area to the destination device, where the UI update data is used by the destination device to update the first screen projection area through the UI framework of the destination device.

14. The method according to claim 13, characterized in that The UI attribute data includes identifiers of the multiple UI nodes, the first manipulation information includes an identifier of a third UI node and a manipulation instruction corresponding to the first operation, the manipulation instruction indicates the second operation, and the third UI node is one of the multiple UI nodes; The performing a second operation on the first projection area based on the first manipulation information includes: Based on the identifier of the third UI node, and in accordance with the manipulation instruction, the second operation is performed on the third UI node.

15. The method according to claim 13, characterized in that The UI attribute data includes identifiers of the multiple UI nodes, the first manipulation information includes an identifier of a fourth UI node and operation information of the first operation, and the fourth UI node is one of the multiple UI nodes; The performing a second operation on the first projection area based on the first manipulation information includes: parsing the operation information of the first operation to determine the second operation; Based on the identifier of the fourth UI node, the second operation is performed on the fourth UI node.

16. The method according to any one of claims 8 to 15, characterized in that: The source device runs a first application, the first screen projection area is a screen projection area in a first page of the first application, and the method further includes: When the first application switches from the first page to the second page, the UI structure and UI attribute data of the second projection area are sent to the destination device to project the second projection area to the destination device, and the second projection area is the projection area in the second page.

17. The method according to any one of claims 8 to 16, characterized in that: The source device supports multiple screen projection modes; and the method further includes: Display the current screen projection mode, where the current screen projection mode is one of the multiple screen projection modes.

18. The method according to any one of claims 8 to 17, characterized in that: The source device supports multiple screen projection modes, and different screen projection areas have different screen projection modes; Before sending the user interface UI structure and UI attribute data of the first projection area to the destination device, the method further includes: The projection mode of the first projection area is determined from the multiple projection modes through a mode adaptation algorithm.

19. A screen projection method, characterized in that: Applied to a target device, the method comprises: The receiving source device sends a user interface UI structure and UI attribute data of the first projection area, wherein the UI structure indicates the first projection area a layout relationship between a plurality of UI nodes in the domain, the UI attribute data indicating rendering-related attributes of each of the plurality of UI nodes; The first projection area is displayed based on the UI structure and the UI attribute data.

20. The method of claim 19, wherein: The displaying of the first projection area based on the UI structure and the UI attribute data includes: Loading the UI structure through the UI framework of the destination device to layout the multiple UI nodes in the first display area; The attributes of each UI node among the multiple UI nodes are set based on the UI attribute data to reconstruct and render the first projection area.

21. The method according to claim 19 or 20, characterized in that The multiple UI nodes include a first UI node corresponding to a first source-end component, the first source-end component belongs to a first type of component, and a corresponding image of the first type of component is drawn by a non-UI framework; The UI attribute data includes a drawing instruction of the first source-end component, where the drawing instruction of the first source-end component is used to draw an image not rendered by the first source-end component through the non-UI framework; or, The UI attribute data includes an image rendered by the first source-end component.

22. The method according to claim 21, characterized in that In a case where the UI attribute data includes a drawing instruction of the first source-end component, loading the UI structure through a UI framework of the destination device to layout the first UI node in a first display area includes: Loading the UI structure through the UI framework to create a first destination component to obtain the first UI node; Setting the attribute of the first UI node based on the UI attribute data includes: Based on the drawing instruction of the first source-end component, the image of the first destination-end component is drawn through the non-UI framework to obtain an unrendered image of the first UI node.

23. The method according to claim 19 or 20, characterized in that In a case where the UI attribute data includes an image rendered by the first source-end component, loading the UI structure through a UI framework of the destination device to layout the first UI node in the first display area includes: Loading the UI structure through the UI framework to create a second destination component to obtain the first UI node, where the second destination component is different from the first source component; Setting the attribute of the first UI node based on the UI attribute data includes: Based on the image rendered by the first source-end component, by setting the UI property of the second destination-end component, the first UI node carries the image rendered by the first source-end component.

24. The method according to any one of claims 19 to 23, characterized in that: The multiple UI nodes include a second UI node corresponding to a second source-end component, the second source-end component belongs to a second type of component, the second type of component has a corresponding image drawn by the UI framework, and the UI attribute data includes drawing attribute data of the second source-end component; Loading the UI structure through the UI framework of the target device to layout the second UI node in the first display area includes: Loading the UI structure through the UI framework to create a third destination component to obtain the second UI node, the third destination component being the same as the second source component; Setting the attribute of the second UI node based on the UI attribute data includes: Based on the drawing attribute data of the second source-end component, the attributes of the second source-end component are set through the UI framework to draw a corresponding image of the second UI node.

25. The method according to any one of claims 19 to 24, characterized in that The method further comprises: In response to a first operation of a destination user on the first projection area, sending first manipulation information to the source device, where the first manipulation information is used to instruct the source device to perform a second operation on the first projection area and update a UI structure and / or UI attribute data of the first projection area; Receiving UI update data of the first screen projection area sent by the source device; Update the first projection area based on the UI update data of the first projection area.

26. The method according to any one of claims 19 to 25, characterized in that The source device supports multiple screen projection modes, and different screen projection areas have different screen projection modes; the method further includes: The real-time computing capability information of the destination device is sent to the source device, where the real-time computing capability information is used by the source device to determine the projection mode of the current projection area from the multiple projection modes through a mode adaptation algorithm.

27. A screen projection device, characterized in that: Included in a source device, the apparatus comprises: A sending module, used to send the user interface UI structure and UI attribute data of the first projection area to the destination device, so as to project the first projection area to the destination device; Among them, the UI structure indicates the layout relationship between multiple UI nodes in the first projection area, the UI attribute data indicates the rendering-related properties of each UI node among the multiple UI nodes, and the UI structure and the UI attribute data are used to display the first projection area on the destination device.

28. A screen projection device, characterized in that: Included in the destination device, the device includes: A receiving module, configured to receive a user interface UI structure and UI attribute data of a first projection area sent by a source device, wherein the UI structure indicates a layout relationship between multiple UI nodes in the first projection area, and the UI attribute data indicates rendering-related attributes of each UI node in the multiple UI nodes; A display module is used to display the first projection area based on the UI structure and the UI attribute data.

29. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the steps of any one of the methods of claims 1-18, or implements the steps of any one of the methods of claims 19-26.

30. A computer program product, characterized in that The computer program product stores computer instructions, and when the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 18 are implemented, or the steps of the method described in any one of claims 19 to 26 are implemented.

Images