Screen projection system, method and device, storage medium and computer program product

Abstract

The invention discloses a screen projection system, method and device, a storage medium and a computer program product, and belongs to the technical field of computers. The method comprises the steps that a source device sends UI structures and UI attribute data of a screen projection area to a target device, so that the target device can build, reconstruct, render and display the corresponding screen projection area based on the UI structures and the UI attribute data of the UI level. According to the scheme, the source device does not need to carry out video coding and transmission after rendering the image, the power consumption of the source device is reduced, the source device does not need to send user related data to the target device, and the risk of user privacy data leakage is reduced. Moreover, according to the scheme, screen projection can be realized under the condition that the target equipment is not provided with an application needing screen projection, so that multi-equipment collaboration can be supported. In addition, the target equipment automatically renders and displays the screen projection area, the screen resolution of the target equipment can be self-adapted, and the display effect is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a screen projection method, device, storage medium and computer program product. Background Art

[0002] Currently, screen projection technologies such as wireless screen projection, multi-screen collaboration, and screen continuation have been widely used in people's lives and work. For example, mobile terminals such as mobile phones and tablets can be used as source devices, and the source devices can project or migrate screen content to target devices such as computers, TVs, and projectors for viewing or user operations on a larger screen.

[0003] There are two main ways to implement screen projection technology in the related art. The first way is the screen mirroring method, and the second way is the continuation method, which is also called migration or flow. In the screen mirroring method, after the source device generates a screen image by rendering, it encodes the screen image frame by frame to form a video stream. The source device sends the video stream to the destination device. The destination device can display the screen image after decoding the video stream, thereby projecting the screen image of the source device onto the destination device. In the continuation method, the source device and the destination device need to have the same application installed. During the running of the application, the source device exports the relevant data of the application and sends it to the destination device. After receiving the data sent by the source, the destination device starts the same application and runs the application based on the received data, thereby realizing the continuation of the application from the source device to the destination device.

[0004] However, in the screen mirroring method, all the work of the application needs to be completed on the source device, including the rendering operation with large computational intensity, and the source device is also required to perform video encoding and transmission. When performing high-definition screen projection, the power consumption of encoding and decoding and video transmission is relatively large. In addition, the rendering operation is completed on the source device, and the screen image is rendered according to the screen resolution of the source device. In this way, it is impossible to adapt to the screen resolution of the destination device, resulting in the screen image displayed on the destination device may not match the screen resolution of the destination device. In the subsequent method, the destination device is required to install the same application, and multi-device collaboration is not supported. In addition, the source device needs to send the user-related data of the application to the destination device, which may involve the leakage of user privacy data. Summary of the invention

[0005] The present application provides a screen projection system, method, device, storage medium and computer program product, which can adapt to the screen resolution of the target device, improve the screen projection effect, reduce the amount of data sent, reduce the power consumption of the source device, and reduce the risk of user privacy data leakage. The technical solution is as follows:

[0006] In a first aspect, a screen projection system is provided, the system comprising a source-end screen projection module and a destination-end screen projection module; the source-end screen projection module is used to send a user interface (UI) structure and UI attribute data of a first projection area to the destination-end screen projection module, so as to project the first projection area from a source device where the source-end screen projection module is located to a destination device where the destination-end screen projection module is located, wherein the UI structure indicates a layout relationship between multiple UI nodes in the first projection area, and the UI attribute data indicates rendering-related attributes of each UI node in the multiple UI nodes; the destination-end screen projection module is used to receive the UI structure and UI attribute data, and display the first projection area on the destination device based on the UI structure and the UI attribute data.

[0007] The destination device can rebuild and render the corresponding projection area based on the UI structure and UI attribute data at the UI level. This solution does not require the source device to render the image and then perform video encoding and transmission, which reduces the power consumption of the source device. It also does not require the source device to send user-related data to the destination device, reducing the risk of user privacy data leakage. Moreover, this solution can also achieve screen projection even if the destination device does not have an application that requires screen projection installed, thereby supporting multi-device collaboration. In addition, the destination device renders and displays the projection area by itself, which can adapt to the screen resolution of the destination device and improve the display effect.

[0008] The target screen projection module is specifically used to: load the UI structure through the UI framework of the target device to layout the multiple UI nodes in the first display area, and set the properties of each UI node in the multiple UI nodes based on the UI property data to rebuild and render the first screen projection area. That is, this solution is specifically implemented based on the UI framework of the target device.

[0009] The source screen projection module is also used to construct the UI structure and generate the UI attribute data through the UI framework of the source device according to the application instructions. That is, this solution relies on the UI layer below the application layer.

[0010] Optionally, the target screen projection module is also used to send the first control information to the source screen projection module to update some or all UI nodes in the first screen projection area by reversely controlling the source device. That is, this solution supports the target device to reversely control the source device.

[0011] The source device runs a first application, and the first projection area is the projection area of ​​the first application.

[0012] Optionally, the destination device does not have the first application installed. That is, this solution does not require the destination device to have the same application installed.

[0013] Optionally, the first projection area is a part or all of the area of ​​the first page. That is, this solution supports projection of a part of the page area.

[0014] In a second aspect, a screen projection method is provided, which is applied to a source device, and the method includes: sending a user interface UI structure and UI attribute data of a first projection area to a destination device to project the first projection area to the destination device; wherein the UI structure indicates a layout relationship between multiple UI nodes in the first projection area, the UI attribute data indicates rendering-related attributes of each UI node in the multiple UI nodes, and the UI structure and the UI attribute data are used to display the first projection area on the destination device.

[0015] That is, the destination device can rebuild and render the corresponding projection area based on the UI structure and UI attribute data at the UI level. This solution does not require the source device to render the image and then perform video encoding and transmission, which reduces the power consumption of the source device. It also does not require the source device to send user-related data to the destination device, reducing the risk of user privacy data leakage. Moreover, this solution can also achieve screen projection when the destination device does not have an application that requires screen projection installed, thereby supporting multi-device collaboration. In addition, the destination device renders and displays the projection area by itself, which can adapt to the screen resolution of the destination device and improve the display effect.

[0016] Among them, before sending the UI structure and UI attribute data of the first projection area to the destination device, the method also includes: building the UI structure and generating the UI attribute data through the UI framework of the source device according to application instructions.

[0017] Optionally, the multiple UI nodes include a first UI node corresponding to a first source-end component, the first source-end component belongs to a first category component, and the first category component has a corresponding image drawn by a non-UI framework; the UI attribute data includes a drawing instruction of the first source-end component, and the drawing instruction of the first source-end component is used to draw an image not rendered by the first source-end component through the non-UI framework; or, the UI attribute data includes an image rendered by the first source-end component. It should be understood that the first category component can be considered as a non-standard component. For example, the first category component includes developer-defined components or some special components preset by the system. For the first category component, the source device synchronizes the drawing instruction or the rendered image to the destination device.

[0018] Optionally, the multiple UI nodes include a second UI node corresponding to a second source-end component, the second source-end component belongs to a second-category component, and the second-category component has a corresponding image drawn by the UI framework; the UI attribute data includes drawing attribute data of the second source-end component, and the drawing attribute data of the second source-end component is used to draw the corresponding image of the second source-end component through the UI framework. It should be understood that the second-category component can be considered as a standard component. For example, the second-category component includes some standard components preset by the system. For the second-category component, the source device synchronizes the attribute data used to draw the corresponding component to the destination device.

[0019] Optionally, the source device runs a first application, and the first projection area is the projection area of ​​the first application. The method further includes: sending application resources of the first application to the destination device, the application resources including resources required for rendering the multiple UI nodes, and the application resources and the UI attribute data are used together for the destination device to reconstruct and render and display the first projection area. That is, the source device may also need to synchronize application resources to the destination device. Among them, application resources may include configuration data, pictures, text, etc. in the application package.

[0020] Optionally, the method further includes: receiving first control information sent by the destination device, the first control information indicating a first operation of the destination user on the first projection area; based on the first control information, performing a second operation on the first projection area, and updating the UI structure and / or UI attribute data of the first projection area; sending UI update data of the first projection area to the destination device, the UI update data is used by the destination device to update the first projection area through the UI framework of the destination device. That is, this solution supports the destination device to reversely control the source device.

[0021] Optionally, the UI attribute data includes the identifiers of the multiple UI nodes, the first control information includes the identifier of the third UI node and the control instruction corresponding to the first operation, the control instruction indicates the second operation, and the third UI node is one of the multiple UI nodes; performing the second operation on the first projection area based on the first control information includes: performing the second operation on the third UI node according to the control instruction based on the identifier of the third UI node. That is, this solution supports the destination device to reversely control individual UI nodes, and the control information synchronized by the destination device to the source device is a specific control instruction.

[0022] Optionally, the UI attribute data includes the identifiers of the multiple UI nodes, the first control information includes the identifier of a fourth UI node and the operation information of the first operation, and the fourth UI node is one of the multiple UI nodes; performing a second operation on the first projection area based on the first control information includes: parsing the operation information of the first operation to determine the second operation; and performing the second operation on the fourth UI node based on the identifier of the fourth UI node. That is, this solution supports the destination device to reversely control individual UI nodes, and the control information synchronized by the destination device to the source device is the user's operation information, and the source device needs to parse the operation information to obtain the second operation corresponding to the corresponding operation instruction.

[0023] Optionally, the source device runs a first application, and the first projection area is a projection area in a first page of the first application. The method further includes: when the first application switches from the first page to the second page, sending the UI structure and UI attribute data of the second projection area to the destination device to project the second projection area to the destination device, wherein the second projection area is a projection area in the second page. That is, when a page switch occurs, the source device resends the UI structure and UI attribute data of the new projection area.

[0024] Optionally, the source device supports multiple screen projection modes; the method also includes: displaying a current screen projection mode, where the current screen projection mode is one of the multiple screen projection modes.

[0025] Optionally, the source device supports multiple projection modes, and different projection areas have different projection modes; before sending the user interface UI structure and UI attribute data of the first projection area to the destination device, the method further includes: determining the projection mode of the first projection area from the multiple projection modes by a mode adaptive algorithm. That is, this solution supports adaptive switching of projection modes during the projection process.

[0026] In a third aspect, a screen projection method is provided, which is applied to a target device, and the method includes: receiving a user interface UI structure and UI attribute data of a first screen projection area sent by a source device, the UI structure indicating a layout relationship between multiple UI nodes in the first screen projection area, and the UI attribute data indicating rendering-related attributes of each UI node in the multiple UI nodes; and displaying the first screen projection area based on the UI structure and the UI attribute data.

[0027] That is, the destination device can rebuild and render the corresponding projection area based on the UI structure and UI attribute data at the UI level. This solution does not require the source device to render the image and then perform video encoding and transmission, which reduces the power consumption of the source device. It also does not require the source device to send user-related data to the destination device, reducing the risk of user privacy data leakage. Moreover, this solution can also achieve screen projection when the destination device does not have an application that requires screen projection installed, thereby supporting multi-device collaboration. In addition, the destination device renders and displays the projection area by itself, which can adapt to the screen resolution of the destination device and improve the display effect.

[0028] Optionally, displaying the first projection area based on the UI structure and the UI attribute data includes: loading the UI structure through the UI framework of the destination device to layout the multiple UI nodes in the first display area; and setting the attributes of each UI node in the multiple UI nodes based on the UI attribute data to reconstruct and render the first projection area. That is, this solution is specifically implemented based on the UI framework of the destination device.

[0029] Optionally, the multiple UI nodes include a first UI node corresponding to a first source-end component, the first source-end component belongs to a first category component, and the first category component has a corresponding image drawn by a non-UI framework; the UI attribute data includes a drawing instruction of the first source-end component, and the drawing instruction of the first source-end component is used to draw an image not rendered by the first source-end component through the non-UI framework; or, the UI attribute data includes an image rendered by the first source-end component. It should be understood that the first category component can be considered as a non-standard component. For example, the first category component includes developer-defined components or some special components preset by the system. For the first category component, the source device synchronizes the drawing instruction or the rendered image to the destination device.

[0030] Among them, in the case that the UI attribute data includes the drawing instructions of the first source-end component, the UI structure is loaded through the UI framework of the destination device to layout the first UI node in the first display area, including: loading the UI structure through the UI framework to create the first destination-end component to obtain the first UI node; setting the attributes of the first UI node based on the UI attribute data, including: based on the drawing instructions of the first source-end component, drawing the image of the first destination-end component through the non-UI framework to obtain an unrendered image of the first UI node.

[0031] In a case where the UI attribute data includes an image rendered by the first source-end component, the UI structure is loaded through the UI framework of the destination device to layout the first UI node in the first display area, including: loading the UI structure through the UI framework to create a second destination-end component to obtain the first UI node, the second destination-end component being different from the first source-end component; setting the attributes of the first UI node based on the UI attribute data, including: based on the image rendered by the first source-end component, setting the UI attributes of the second destination-end component so that the first UI node carries the image rendered by the first source-end component.

[0032] Optionally, the multiple UI nodes include a second UI node corresponding to a second source component, the second source component belongs to a second type of component, the second type of component is drawn by the UI framework with a corresponding image, and the UI attribute data includes the drawing attribute data of the second source component; loading the UI structure through the UI framework of the destination device to layout the second UI node in the first display area includes: loading the UI structure through the UI framework to create a third destination component to obtain the second UI node, the third destination component is the same as the second source component; setting the attributes of the second UI node based on the UI attribute data includes: setting the attributes of the second source component through the UI framework based on the drawing attribute data of the second source component to draw the corresponding image of the second UI node. It should be understood that the second type of component can be considered as a standard component, for example, the second type of component includes some standard components preset by the system. For the second type of component, the source device synchronizes the attribute data used to draw the corresponding component to the destination device.

[0033] Optionally, the method further includes: in response to a first operation of a destination user on the first projection area, sending a first control information to the source device, the first control information being used to instruct the source device to perform a second operation on the first projection area and update the UI structure and / or UI attribute data of the first projection area; receiving UI update data of the first projection area sent by the source device; and updating the first projection area based on the UI update data of the first projection area. That is, this solution supports the destination device to reversely control the source device.

[0034] Optionally, the source device supports multiple screen projection modes, and different projection areas have different projection modes; the method further includes: sending real-time computing capability information of the destination device to the source device, and the real-time computing capability information is used by the source device to determine the projection mode of the current projection area from the multiple projection modes through a mode adaptive algorithm. That is, this solution supports adaptive switching of projection modes during the projection process.

[0035] In a fourth aspect, a screen projection device is provided, wherein the screen projection device has the function of implementing the screen projection method behavior in the second aspect or the third aspect. The screen projection device includes one or more modules, and the one or more modules are used to implement the screen projection method provided in the second aspect or the third aspect.

[0036] In a fifth aspect, an electronic device is provided, comprising a processor and a memory, wherein the memory is used to store a program for executing the screen projection method provided in the second aspect or the third aspect, and to store data involved in implementing the screen projection method provided in the second aspect or the third aspect. The processor is configured to execute the program stored in the memory. The electronic device may further comprise a communication bus for establishing a connection between the processor and the memory.

[0037] In a sixth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the screen projection method described in the second aspect or the third aspect.

[0038] In the seventh aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the screen projection method described in the second aspect or the third aspect.

[0039] The technical effects obtained in the above-mentioned fourth to seventh aspects are similar to the technical effects obtained by the corresponding technical means in the first to third aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a schematic diagram of a screen mirroring method involved in an embodiment of the present application;

[0041] Figure 2 is a schematic diagram of a continuous screen projection method involved in an embodiment of the present application;

[0042] Figure 3 This is a system architecture diagram involved in a screen projection method provided in an embodiment of the present application;

[0043] Figure 4 It is a schematic diagram of the software and hardware architecture of an electronic device provided in an embodiment of the present application;

[0044] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0045] Figure 6 is a flow chart of a screen projection method provided in an embodiment of the present application;

[0046] Figure 7is a flow chart of another screen projection method provided in an embodiment of the present application;

[0047] Figure 8 It is a schematic diagram of a screen projection method for non-standard components in the screen projection method provided in an embodiment of the present application;

[0048] Fig. 9 is a schematic diagram of a mode adaptation provided by an embodiment of the present application;

[0049] Fig.10 This is a screen projection flow chart for a page update provided in an embodiment of the present application;

[0050] Fig.11 is a flow chart of another screen projection method provided in an embodiment of the present application;

[0051] Fig.12 This is a schematic diagram of the core steps of a screen projection method provided in an embodiment of the present application;

[0052] Fig.13 This is a schematic diagram of a screen projection area on a source device provided in an embodiment of the present application;

[0053] Fig.14 This is a schematic diagram of a screen projection effect on a target device in the solution provided in an embodiment of the present application;

[0054] Fig.15 This is a schematic diagram of a screen projection effect of a mirror mode on a target device in the related technology provided by an embodiment of the present application;

[0055] Fig.16 is another screen projection area schematic diagram of the source device in the solution provided by the embodiment of the present application;

[0056] Fig.17 This is the solution provided in the embodiment of the present application. Fig.16 A schematic diagram of a projection area of ​​the corresponding target device;

[0057] Fig.18 It is a structural schematic diagram of a screen projection device provided in an embodiment of the present application;

[0058] Fig.19 It is a structural schematic diagram of another screen projection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0060] At present, screen projection technologies such as wireless screen projection, multi-screen collaboration, and continuation have been widely used in people's lives and work. For example, mobile terminals such as mobile phones and tablets can be used as source devices, and the source devices can project or migrate screen content to target devices such as computers, TVs, and projectors for viewing on a larger screen or for user operations. Of course, large-screen devices can also be projected to small-screen devices. For example, the video call application on the computer is connected to the mobile phone so that the user can continue the video call using the mobile phone. It should also be understood that the source device can be one or more. For example, in a multi-channel screen projection scenario, multiple source devices can be simultaneously projected to the same destination device to display the screen images corresponding to multiple source devices on the destination device in split screen. Similarly, the destination device can be one or more. For example, the source device can share the screen image with multiple destination devices so that users of the multiple destination devices can view the screen images corresponding to the source device.

[0061] There are two main ways to implement screen projection technology in the related art. The first method is the screen projection method of screen mirroring, and the second method is the continuous screen projection method, which is also called migration or flow. For easy understanding, the execution process of the application is first introduced. The execution process of the application usually includes three steps. The first step is the logical operation of the application, that is, the application layer of the device runs the UI components and application function logic developed by the application developer to trigger a series of UI tasks or instructions. The second step is for the device to load its own system UI framework (which can be referred to as UI framework for short) to execute this series of UI tasks or instructions, thereby building or updating the application page. The third step is for the device to render the display page on the screen through its own graphics system based on the constructed page.

[0062] Figure 1 Schematic diagram of the screen mirroring method involved in the embodiment of the present application. Figure 1In the screen mirroring projection mode, after the source device generates the screen image by rendering, it encodes the screen image frame by frame to form a video stream. The source device sends the video stream to the destination device through the network. The destination device can display the screen image after decoding the video stream through its own graphics system, thereby realizing the projection of the screen image of the source device onto the destination device. However, when performing wireless high-definition screen projection or multi-channel screen projection, the power consumption of the source device for encoding and decoding is relatively large, and because the video stream needs to be continuously transmitted on the network, the power consumption of the wireless transmission module (such as wireless fidelity (WiFi) chip) on the source device is relatively large. For high-definition screen projection, the key frame in the video stream is about 100-200KB in size, and the amount of network transmission data is large. When the network quality is poor, the destination device is prone to jamming. In addition, the rendering operation is completed on the source device, and the screen image is rendered according to the screen resolution of the source device, so it cannot adapt to the screen resolution of the destination device, resulting in the screen image displayed on the destination device and the screen resolution of the destination device may not match.

[0063] Figure 2 is a schematic diagram of a continuous screen projection method involved in an embodiment of the present application. Figure 2 In the continued screen projection method, the source device and the destination device need to have the same application installed. During the running of the application, the source device exports the relevant data of the application (including application status data, etc.) and sends it to the destination device through the network. After receiving the data sent by the source, the destination device starts the same application and runs the application by writing the application status data, thereby connecting the application from the source device to the destination device. However, in the continued screen projection method, the destination device is required to install the same application, and does not support multi-device collaboration and the destination device's reverse control of the source device. In addition, the source device needs to send the user-related data of the application to the destination device, which may involve the leakage of user privacy data.

[0064] In order to solve the above-mentioned problems in the related art, an embodiment of the present application provides a new screen projection method, which will be introduced in detail below.

[0065] Figure 3 is a system architecture diagram involved in a screen projection method provided in an embodiment of the present application. Figure 3 The system architecture includes a source device and a destination device. A wireless or wired communication connection is established between the source device and the destination device. The source device includes a source screen projection module, and the destination device includes a destination screen projection module.

[0066] The source device is used to run the application to be projected, and sends the UI data stream of the application to the destination device based on its own system UI framework, that is, synchronizes the UI internal state data to the destination device, including UI structure and UI attribute data. The destination device is used to construct and render the corresponding projection area through its own UI framework based on the received UI data stream.

[0067] Among them, the application can be a system program that comes with the source device when it leaves the factory, or a non-system program installed after the source device leaves the factory, or a small program, a quick application, and other programs that do not need to be installed and can be used immediately. That is, as long as the source device can present the screen, it can be projected according to this solution. In addition, the projection area can be all display areas of the source device, or it can be part of the display area of ​​the source device. Taking the source device running the first application as an example, the projection area can be part or all of the first page in the first application.

[0068] Taking the entire area of ​​the first page in the first application as an example, in a specific implementation, the source device is used to send the UI structure and UI attribute data of the first page to the destination device after constructing the first page through its own UI framework during the running of the first application. The destination device is used to receive the UI structure and UI attribute data of the first page, and based on the UI structure and UI attribute data of the first page, construct multiple UI nodes of the first page through its own UI framework, and render and display the first page through its own graphics system. Optionally, in some cases, the source device also needs to send the application resources of the first application (including the resources required to render multiple UI nodes in the first page) to the destination device. The application resources are such as icon resources, text resources, etc. in the application package. The application resources and UI attribute data are used together to set the attributes of the UI nodes. That is, the destination device is used to load the UI structure, UI attribute data and application resources through the UI framework to construct the first page. Among them, when the first page is the first page to be projected, the destination device needs to load the UI framework first. After loading the UI framework, the destination device can construct each page to be projected through its own UI framework.

[0069] It should be understood that this solution does not limit whether the destination device has the first application installed.

[0070] Figure 4 1 is a schematic diagram of the hardware and software architecture of an electronic device provided in an embodiment of the present application. The electronic device can be used as a source device or a destination device. The electronic device includes an application part, a system UI framework (referred to as UI framework), a network synchronization module, a service management module, a processor, a memory, and a network card.

[0071] The application part includes the applications installed after leaving the factory ( Figure 4), and system applications installed at the factory ( Figure 4 The application has an application logic part and application resources and configuration files. The application logic part includes program logic (Logic) and pages (Pages). The application resources and configuration files include media resources (media) and raw files (rawfile). In the embodiment of the present application, the dedicated application in the destination device includes a UI display module, while the dedicated application in the source device may not include a UI display module, but may also include a UI display module. The UI display module also includes two parts: Logic and Pages.

[0072] The UI framework includes the front-end framework layer, the engine layer and the UI proxy module. The front-end framework layer includes application programming interfaces, special components (such as some non-standard components), standard components and component events, and application programming interfaces such as JavaScript new application program interface (JSNAPI). The engine layer is used for UI component parsing, construction and page rendering.

[0073] In an embodiment of the present application, each UI node corresponds to a component, and the component is used to determine the UI node, and the component includes a first type of component and a second type of component. The first type of component draws the corresponding image by the non-UI framework. In other words, the first type of component can be constructed by the UI framework but the corresponding image is drawn by the non-UI framework. For example, the corresponding image is drawn by running the first program code by the non-UI framework, and the first program code is provided by other frameworks or application developers. The second type of component draws the corresponding image by the UI framework. In other words, the second type of component can be constructed by the UI framework and draw the corresponding image. Among them, the UI node corresponding to the first type of component is called the first type of node, and the UI node corresponding to the second type of component is called the second type of node. In a specific implementation, the first type of component includes non-system preset components, such as custom components of application developers, and may also include some special components preset by the system, such as surface components, web components, and video components. The second type of component includes some standard components preset by the system, such as Image components, etc. In some embodiments, the first type of component can also be called a non-standard component, and the second type of component can be called a standard component, that is, custom components and special components can be called non-standard components, and other components can be called standard components.

[0074] It should be understood that one component corresponds to one or more UI nodes, and images of one or more UI nodes can be generated by drawing one component.

[0075] The processor includes a central processing unit (CPU) and / or a graphics processing unit (GPU). The processor, memory and network card provide hardware support for application running, data processing and data transmission, including providing system resources and network resources.

[0076] It should be understood that Figure 4 The UI display module, UI proxy module, network synchronization module and service management module are newly added modules in the embodiment of the present application relative to the electronic device in the related art.

[0077] Next, we will combine Figure 4 The roles of the above-mentioned UI display module, UI proxy module, network synchronization module and service management module in the embodiments of the present application are introduced.

[0078] At the beginning of screen projection, the service management module of the source device and the service management module of the destination device are used to negotiate the screen projection mode to confirm which of the multiple screen projection modes is currently used. The multiple screen projection modes include mirror mode, UI data stream mode (i.e., a screen projection mode provided in an embodiment of the present application), etc. When the use of the UI data stream mode is confirmed, the destination device starts its own UI display module. After the source device executes the application logic of the application to be projected and determines or updates the UI structure and UI attribute data of the area to be projected through the UI framework, the UI proxy module of the source device obtains the UI structure and UI attribute data or UI update data, and sends it to the network synchronization module of the destination device through the network synchronization module of the source device. The UI proxy module of the destination device creates a display area or updates the UI based on the UI structure and UI attribute data or UI update data of the area to be projected, and then renders the corresponding image through the engine layer of the destination device, and displays the projection area through the UI display module of the destination device.

[0079] The embodiment of the present application does not limit what the source device and the destination device are specifically. For example, the source device can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, etc., and the destination device can be a mobile phone, a laptop computer, a desktop computer, a projector, etc.

[0080] The system architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art will appreciate that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0081] Please refer to Figure 5 , Figure 5 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Optionally, the electronic device is Figure 3 The electronic device includes one or more processors 501 , a communication bus 502 , a memory 503 and one or more communication interfaces 504 .

[0082] The processor 501 includes a CPU, a GPU, a network processor (NP), a microprocessor, or one or more integrated circuits for implementing the solution of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. Optionally, the PLD is a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0083] The communication bus 502 is used to transmit information between the above components. Optionally, the communication bus 502 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0084] Optionally, the memory 503 is a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compressed optical disc, a laser disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 503 exists independently and is connected to the processor 501 via the communication bus 502, or the memory 503 is integrated with the processor 501.

[0085] The communication interface 504 uses any transceiver-like device for communicating with other devices or communication networks. The communication interface 504 includes a wired communication interface, and optionally, also includes a wireless communication interface. Among them, the wired communication interface is, for example, an Ethernet interface. Optionally, the Ethernet interface is an optical interface, an electrical interface, or a combination thereof. The wireless communication interface is a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0086] Optionally, in some embodiments, the electronic device includes multiple processors, such as Figure 5 The processor 501 and the processor 505 shown in . Each of these processors is a single-core processor, or a multi-core processor. Optionally, the processor here refers to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0087] In a specific implementation, as an embodiment, the electronic device also includes an output device 506 and an input device 507. The output device 506 communicates with the processor 501 and can display information in a variety of ways. For example, the output device 506 is a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device or a projector. The input device 507 communicates with the processor 501 and can receive user input in a variety of ways. For example, the input device 507 is a mouse, a keyboard, a touch screen device or a sensor device.

[0088] In some embodiments, the memory 503 is used to store the program code 510 for executing the solution of the present application, and the processor 501 can execute the program code 510 stored in the memory 503. The program code includes one or more software modules, and the electronic device can implement the following by the processor 501 and the program code 510 in the memory 503. Figure 6 or Figure 7 The screen projection method provided by the embodiment.

[0089] Figure 6 This is a flow chart of a screen projection method provided in an embodiment of the present application. The method is applied to a screen projection system, and the screen projection system includes a source device and a destination device. Specifically, the source device includes a source screen projection module, and the destination device includes a destination screen projection module. The method can be specifically applied to the source screen projection module and the destination screen projection module. Please refer to Figure 6 , the method comprises the following steps.

[0090] Step 601: The source device sends the UI structure and UI attribute data of the first projection area to the destination device to project the first projection area to the destination device.

[0091] Among them, the UI structure indicates the layout relationship between multiple UI nodes in the first projection area, the UI attribute data indicates the rendering-related properties of each UI node among the multiple UI nodes, and the UI structure and UI attribute data are used to display the first projection area on the target device.

[0092] That is, the source device synchronizes the UI layer data to the destination device. Before synchronizing the UI structure and UI attribute data, the source device may need to first build the UI structure and generate the UI attribute data through the UI framework of the source device according to the application instruction.

[0093] In some embodiments, after the screen projection process starts, the source device sends application resources of the first application to the destination device, where the first application refers to the application to which the first screen projection area belongs, and the application resources are the resources required to render the first screen projection area.

[0094] The first projection area is a part or all of the first page in the first application. The source device runs the first application, and the destination device does not run the first application, for example, the destination device does not install the first application. The first application can be a system application or a non-system application.

[0095] Step 602: The destination device receives the UI structure and UI attribute data of the first projection area sent by the source device.

[0096] Step 603: The destination device displays the first projection area based on the UI structure and UI attribute data.

[0097] Among them, the target device loads the UI structure through the UI framework of the target device to layout the multiple UI nodes in the first display area, and sets the properties of each UI node in the multiple UI nodes based on the UI property data to reconstruct and render the first projection area.

[0098] In this solution, the source device sends the UI structure and UI attribute data of the projection area to the destination device, so that the destination device can rebuild and render the corresponding projection area based on the UI structure and UI attribute data at the UI level. This solution does not require the source device to perform video encoding and transmission after rendering the image, which reduces the power consumption of the source device. It also does not require the source device to send user-related data to the destination device, reducing the risk of user privacy data leakage. Moreover, this solution can also achieve screen projection when the destination device does not have an application that requires screen projection installed, thereby supporting multi-device collaboration. In addition, the destination device renders and displays the projection area by itself, which can adapt to the screen resolution of the destination device and improve the display effect.

[0099] Next, we will take the example of synchronizing application resources, UI structure, and UI attribute data from the source device to the destination device. Figure 7 right Figure 6 The specific implementation method of the embodiment is introduced in detail.

[0100] Figure 7 This is a flowchart of another screen projection method provided by an embodiment of the present application. The method is applied to a screen projection system, and the screen projection system includes a source device and a destination device. Specifically, the source device includes a source screen projection module, and the destination device includes a destination screen projection module. The method can be specifically applied to the source screen projection module and the destination screen projection module. Please refer to Figure 7 , the method comprises the following steps.

[0101] Step 701: The source device sends application resources of a first application to the destination device.

[0102] The source device runs a first application. After the screen projection process starts, the source device sends the application resources of the first application to the destination device, and the application resources include the resources required for rendering the projection area, such as configuration data, pictures (such as icons), texts and other media resources in the application package.

[0103] It should be understood that the synchronization of application resources is to prepare for the subsequent rendering and display of the destination device, and the destination device may use part or all of the application resources in the subsequent rendering and display, or may not use the application resources. Based on this, in another possible implementation, when each screen projection area is projected, the source device synchronizes the used application resources to the destination device, that is, the source device synchronizes the application resources required by the current screen projection area in real time, rather than synchronizing all application resources to non-destination devices at the beginning of the screen projection process.

[0104] As can be seen from the foregoing, the projection mode provided in the embodiment of the present application is a UI data stream mode, and the UI data stream mode refers to a mode in which corresponding UI attribute data is transmitted to all UI nodes in the same projection area. The projection modes in the related art include mirror mode and continuation mode. If the source device only supports the UI data stream mode, or the source device supports multiple projection modes, but is configured to use the UI data stream mode by default, then after the projection process starts, the source device directly sends the application resources of the first application to the destination device. If the source device supports multiple projection modes, but is not configured to use a certain projection mode by default, before sending the application resources of the first application to the destination device, the source device determines that the current projection mode is a UI data stream mode among the multiple projection modes, and then executes the step of sending the application resources of the first application to the destination device. That is, when the source device supports multiple projection modes, the source device and the destination device can first determine the current projection mode from the multiple projection modes.

[0105] There are many ways to determine the screen projection mode between the source device and the destination device. Here are three of them.

[0106] In a first implementation method of determining the current screen projection mode, the source device sends a first request to the destination device, and the first request is used to obtain device resource information of the destination device. The destination device receives the first request sent by the source device, and in response to the first request, sends the device resource information of the destination device to the source device. The source device receives the device resource information of the destination device sent by the destination device, and based on the device resource information of the destination device, determines the current screen projection mode from the multiple screen projection modes, such as determining that the current screen projection mode is a UI data stream mode, that is, the device resource information of the destination device is used by the source device to determine that the current screen projection mode is a UI data stream mode from the multiple screen projection modes. Optionally, the source device sends a first message to the destination device, and the first message indicates the current screen projection mode, such as indicating that the current screen projection mode is a UI data stream mode.

[0107] In the second implementation method of determining the current screen projection mode, the source device sends the device resource information of the source device to the destination device. The destination device receives the device resource information of the source device sent by the source device, determines that the current screen projection mode is the UI data stream mode based on the device resource information of the source device, and sends a first response to the source device, the first response indicating the current screen projection mode, for example, indicating that the current screen projection mode is the UI data stream mode. The source device receives the first response sent by the destination device, and obtains the current screen projection mode by parsing the first response.

[0108] In the third implementation of determining the current projection mode, the source device selects the current projection mode from multiple projection modes based on the detected user operation, for example, the user selects the current projection mode as the UI data stream mode. That is, this solution supports users (such as technicians, etc.) to select the projection mode as needed.

[0109] The above-mentioned device resource information indicates one or more of the software and hardware capabilities, operating system version and UI framework version of the corresponding device. Among them, the software and hardware capabilities of the device include the CPU architecture of the device, whether it has a GPU, the GPU architecture it has, the graphics card configuration and other software and hardware specifications. In the first two implementation methods of determining the current screen projection mode, the conditions for determining that the current screen projection mode is the UI data stream mode based on the device resource information include that the operating system and UI framework of the target device need to be compatible with the operating system and UI framework of the source device, and the software and hardware of the target device support the construction and rendering of the display page through the UI data stream mode.

[0110] In an embodiment of the present application, the multiple projection modes supported by the source device also include one or more of a mirror mode, a hybrid mode, and a continuation mode. Among them, the mirror mode refers to a mode in which all UI nodes in the same projection area are rendered to obtain a rendered image of the corresponding projection area and transmit it. The hybrid mode is also called a media stream + UI data stream mode. The hybrid mode refers to a mode in which the corresponding UI attribute data is transmitted to the second UI node, and the third UI node is rendered to obtain a rendered image of the third UI node and transmit it. The second UI node and the third UI node are different UI nodes in the same projection area. The continuation mode refers to a mode in which the first application is continued to run on the destination device and the source device stops running the first application.

[0111] It should be understood that a projection area includes multiple UI nodes, and the mixed mode can be understood as using different projection modes for two different UI nodes in the same projection. For example, if the content of the second UI node in a projection area is a video, and the content of the third UI node is an icon, then the UI data stream mode can be used for the second UI node, for example, the source device transmits the link of the video to the destination device, and the media stream mode is used for the third UI node, that is, the source device transmits the rendered image of the icon to the destination device.

[0112] Optionally, the source device includes Figure 4 In the case of the service management module and the network synchronization module shown, the source device can negotiate the screen projection mode with the destination device through the service management module, and send the application resources of the first application to the destination device through the network synchronization module.

[0113] Step 702: The source device constructs a UI structure of the first projection area and generates UI attribute data of the first projection area.

[0114] The source device constructs the UI structure of the first projection area and generates UI attribute data of the first projection area while executing step 701 or after executing step 701. Specifically, the source device constructs the UI structure of the first projection area and generates UI attribute data of the first projection area through the UI framework of the source device. Among them, before projecting the first projection area, the source device needs to load the UI framework of the source device first, such as loading the UI framework when the source device starts the first application, and then constructing the projection area of ​​each page through the loaded UI framework.

[0115] The first projection area is part or all of the first page in the first application. When the first projection area is a partial area of ​​the first page, the source device constructs the UI structure of the first page and generates UI attribute data of the first page. Alternatively, the source device may only construct the UI structure of the first projection area and generate UI attribute data of the first projection area.

[0116] Optionally, the first projection area can be determined based on the user's selection operation of part or all of the display area of ​​the source device, that is, this solution supports the user to select the projection area. Alternatively, the first projection area is determined based on the default setting of the first application, that is, this solution supports automatic projection of part of the page area in the application.

[0117] The UI structure of the first projection area includes multiple UI nodes, each UI node corresponds to a component, and the multiple UI nodes include a first UI node. The source device obtains the first UI node by creating a first source-end component, that is, the first UI node in the source device corresponds to the first source-end component, and the first source-end component belongs to a first category component. The first category component draws the corresponding image by a non-UI framework. For example, the first category component is drawn by running a first program code through a non-UI framework, and the first program code is provided by the developer of other frameworks or the first application.

[0118] In the case where the multiple UI nodes include a first UI node, the above-mentioned UI attribute data include a drawing instruction of a first source-end component, the drawing instruction of the first source-end component is an instruction generated by a source device in a process of drawing the first source-end component, and the drawing instruction of the first source-end component is used by a destination device to draw an image that is not rendered by the first source-end component through a non-UI framework; or, the above-mentioned UI attribute data include an image rendered by the first source-end component, and the image rendered by the first source-end component is obtained by rendering the image that is not rendered by the first source-end component by the source device after drawing the first source-end component.

[0119] In the embodiment of the present application, the first source component corresponding to the first UI node belongs to the first category of components, and the first category of components includes custom components of application developers and / or some special components preset by the system, such as surface, web, video components, etc. In some embodiments, the first category of components can be referred to as non-standard components. The above-mentioned first component includes at least one custom component and / or at least one special component. The drawing instruction of the first category of nodes includes the above-mentioned first program code.

[0120] It should be understood that, generally speaking, a component is used to determine at least one UI node, that is, one or more UI nodes can be obtained by creating a certain component. If a certain component belongs to a first-class component, then each UI node determined by the component belongs to a first-class node. Of course, in other embodiments, a UI node is determined by at least one component, that is, a UI node can be drawn by one or more components, and these one or more components may include first-class components and / or second-class components. In the case where at least one first-class component is included in the components required to draw a certain UI node, the UI node is a first-class node. It can be seen that in a specific implementation, the correspondence between components and UI nodes can be one-to-one, one-to-many, or many-to-one, which can be understood as being related to the granularity of defining UI nodes.

[0121] Simply put, there are two ways to project screens corresponding to non-standard components, such as Figure 8 As shown, one is that the source device synchronizes the drawing instructions (or drawing tasks) corresponding to the non-standard components to the destination device. The destination device responds to the received drawing instructions, builds and draws the same UI node, such as drawing the UI node through the same CustomPaint Node (abbreviated as CP_N) component as on the source device, and renders and displays it to present the final effect. The other is that the source device synchronizes the image rendered by the non-standard component to the destination device. The destination device constructs the UI node through a standard component (such as the ImageNode (abbreviated as Img_N) component) and renders and displays the image of the UI node. That is, the UI attribute data of the first type of node includes the drawing instructions or rendered images corresponding to the non-standard components. It should be understood that the image rendered by the non-standard component synchronized by the source device to the destination device can be a static picture or a dynamic picture, that is, an animation. While the source device synchronizes the image rendered by the non-standard component to the destination device, it can also synchronize the corresponding control size.

[0122] The above-mentioned multiple UI nodes usually also include a second UI node, and the second UI node includes a corresponding component as a standard component. The UI attribute data of the second UI node may include UI control state data such as text, pictures, animations, videos, text attributes, cursor positions, sliding positions, colors, etc. of the corresponding node, that is, the UI attribute data of the second UI node includes the drawing attribute data of the second UI node, that is, the attribute data that can be used to draw the second UI node. The component corresponding to the second UI node belongs to the second category component, and the second category component draws the corresponding image by the UI framework. In other words, the second category component can be constructed and draw the corresponding image by the UI framework. That is, for standard components, what the source device synchronizes to the destination device is the attribute data that can be used to draw the corresponding UI node. Of course, in some other embodiments, the drawing instructions of certain standard components can also be synchronized.

[0123] In an embodiment of the present application, the UI structure of multiple UI nodes in a projection area can be represented as a tree structure or other structures.

[0124] Optionally, after constructing the UI structure of the first projection area, the source device generates a unique projection area identifier for the first projection area, which can be recorded as a source profile identifier (SPID). Optionally, the source device can also generate a unique node identifier for each UI node in the multiple UI nodes, which can be recorded as a source frame identifier (SFID). In which, when the first projection area is the entire area of ​​the first page, the source device generates a unique page identifier for the first page, that is, in this case, the above-mentioned projection area identifier is also called a page identifier.

[0125] Step 703: The source device sends the UI structure and UI attribute data of the first projection area to the destination device.

[0126] The source device may send the UI structure and UI attribute data of the first projection area to the destination device while executing step 701 or after executing step 701. The application resources of the first application and the UI structure and UI attribute data of the first page are used by the destination device to rebuild and render the first projection area through the UI framework of the destination device.

[0127] The source device includes Figure 4 In the case of the UI proxy module and network synchronization module shown, the source device can obtain the UI structure and UI attribute data of the first projection area through the UI proxy module, and send the UI structure and UI attribute data of the first projection area to the destination device through the network synchronization module.

[0128] As can be seen from the foregoing, the source device in this solution can support multiple projection modes. In this case, optionally, different projection areas may have different projection modes. Then, before the source device sends the UI structure and UI attribute data of the first projection area to the destination device, it determines that the projection mode of the first projection area is the UI data stream mode from the multiple projection modes through a mode adaptation algorithm.

[0129] Optionally, before projecting each projection area, the source device may determine the projection mode of the current projection area from multiple projection modes through a mode adaptive algorithm. Alternatively, the source device may determine the projection mode of the current projection area from multiple projection modes through a mode adaptive algorithm at a time interval, and the time interval may be 1 minute, 3 minutes, etc.

[0130] Among them, one implementation method of the source device determining that the projection mode of the first projection area is the UI data stream mode from the multiple projection modes through a mode adaptive algorithm is: based on the UI structure and UI attribute data of the first projection area, and / or the real-time computing capability information of the target device, determining that the projection mode of the first projection area is the UI data stream mode from the multiple projection modes through a mode adaptive algorithm.

[0131] For example, the destination device sends real-time computing capability information of the destination device to the source device, and the real-time computing capability information is used by the source device to determine the projection mode of the current projection area from the multiple projection modes through a mode adaptation algorithm.

[0132] For example, when the real-time computing capability of the destination device becomes weak due to a long period of time spent performing rendering operations, the source device may determine that the projection mode of the current projection area is the mirror mode from among a plurality of projection modes.

[0133] When the input of the mode adaptation algorithm includes the UI structure and UI attribute data of the first projection area, the mode adaptation algorithm is used to analyze the UI structure and UI attribute data of the first projection area to determine whether each UI node in the UI structure can be reconstructed on the destination device through the UI data stream mode, and the amount of data of the UI structure and UI attribute data of the first projection area that needs to be transmitted in the UI data stream mode.

[0134] Exemplarily, the component corresponding to a UI node in the second projection area is a non-standard component. The source device analyzes the UI node and determines that the UI node cannot be reconstructed on the destination device through the UI data stream mode. Then the source device can determine that the projection mode of the second projection area is a mirror mode or a hybrid mode. When the amount of data of the UI structure and UI attribute data of the second projection area required to be transmitted by the UI data stream mode is greater than a threshold, the source device can determine that the projection mode of the second projection area is a mirror mode or a hybrid mode.

[0135] It should be understood that the source device can determine the projection mode of the first projection area by comprehensively analyzing the UI structure and UI attribute data of the first projection area and the real-time computing capability information of the target device through a mode adaptation algorithm.

[0136] It should also be understood that when the screen projection is just started, the current projection mode is determined in step 701, and during the projection process, the projection mode of the current projection area can also be determined by the mode adaptation algorithm in step 703. That is, this solution supports switching of projection modes during the projection process to reduce the power consumption of the source device and the destination device and improve the projection effect.

[0137] Fig. 9is a schematic diagram of a mode adaptation provided by an embodiment of the present application. Fig. 9 The source device supports three screen projection modes, namely, mirror mode (i.e., transmitting media stream), UI data stream mode (i.e., transmitting UI data stream), and hybrid mode (i.e., transmitting media stream + UI data stream). The UI structure of the screen projection area constructed by the source device is a tree structure including multiple UI nodes, including Fig. 9 As shown, the first type of nodes filled with white (corresponding to the first type of components) and the second type of nodes filled with black (corresponding to the second type of components), the source device determines the projection mode of the projection area from the above three projection modes through UI structure analysis and mode adaptation, and projects the projection area according to the determined projection mode.

[0138] Among them, in the mirror mode, the source device needs to encode and form a media stream (ie, a video stream) and send it, and the destination device needs to receive and decode the media stream. In the UI data stream mode, the source device needs to build and update the UI structure of the projection area, and the destination device needs to rebuild and update the projection area, that is, to rebuild and update the UI. In the hybrid mode, the source device needs to encode and send part of the rendered image of the projection area (such as the image rendered by the first type of node), as well as transmit the UI structure of the projection area and the UI attribute data of the rest of the projection area (such as the UI attribute data of the second type of node). That is, the source device needs to perform stream encoding and UI construction and update. The destination device needs to decode part of the image of the projection area and reconstruct the rest of the projection area. That is, the destination device needs to perform stream decoding and UI reconstruction and update.

[0139] Optionally, when the source device supports multiple projection modes, the source device can also display the current projection mode, which is one of the multiple projection modes. The embodiments of the present application do not limit the way in which the source device displays the current projection mode, including not limiting the display position, display font, display color, etc. of the current projection mode.

[0140] Optionally, the source device sends an identifier of the first projection area, such as the SPID of the first projection area, while sending the UI structure and UI attribute data of the first projection area to the destination device. The UI attribute data sent by the source device includes identifiers of multiple UI nodes in the first projection area, such as the SFID of each UI node in the multiple UI nodes.

[0141] Optionally, the source device can serialize the data to be sent (including the UI nodes and UI attribute data of the first projection area) to obtain serialized page data, and send the serialized page data to the destination device. Among them, serialization processing can refer to assembling (or encoding) into a string or binary array (such as a binary Buffer) or a file. Of course, the source device can also process the data to be sent according to other data processing methods, which is not limited to this embodiment of the present application.

[0142] In a specific implementation, the source device may send a page reconstruction message to the destination device, where the page reconstruction message carries the above serialized page data.

[0143] Step 704: The destination device receives application resources of the first application and UI structure and UI attribute data of the first projection area.

[0144] The target devices include Figure 4 In the case of the network synchronization module shown, the destination device receives the application resources and the UI structure and UI attribute data of the first projection area sent by the source device through the network synchronization module.

[0145] Optionally, when the source device sends serialized page data, the destination device deserializes the received serialized page data to obtain the application resources of the first application and the UI structure and UI attribute data of the first projection area. The deserialization processing of the destination device is an operation that matches and is symmetrical to the serialization processing of the source device. For example, the deserialization processing refers to decoding the serialized page data.

[0146] Step 705: The destination device displays the first projection area based on the application resources and the UI structure and UI attribute data of the first projection area.

[0147] In an embodiment of the present application, the destination device loads the UI structure through the UI framework of the destination device to layout the above-mentioned multiple UI nodes in the first display area, and sets the properties of each UI node in the multiple UI nodes based on the UI property data to reconstruct and render the first projection area. That is, the destination device can reconstruct and render the first projection area through the UI framework. Among them, the destination device can layout the above-mentioned multiple UI nodes in the first display area by creating multiple UI nodes in the first blank page. That is, the above-mentioned multiple UI nodes are carried by the first blank page.

[0148] The target devices include Figure 4 In the case of the UI proxy module and UI display module shown, the target device constructs and renders the first projection area through the UI proxy mode in the UI framework, and displays the rendered first projection area through the UI display module.

[0149] Among them, the first display area can be part or all of the display area of ​​the destination device, that is, the first display area is the full-screen area or partial screen area of ​​the destination device. It should be noted that the first projection area of ​​the source device can also be part or all of the display area of ​​the source device. In the case where the source device displays the first projection area in full screen, the destination device can either display the first projection area in full screen or not fully screen, for example, the destination device is provided with a display area occupied by the projection. Similarly, in the case where the source device does not display the first projection area in full screen, the destination device can either display the first projection area or not fully screen.

[0150] As can be seen from the above, for the first type of nodes and the second type of nodes, the data synchronized by the source device is different. Next, the implementation method of the destination device to reconstruct the first type of nodes and the second type of nodes is described in detail.

[0151] First, the reconstruction method of the first type of nodes (corresponding to non-standard components) is introduced.

[0152] Taking the first UI node among the multiple UI nodes as an example, which belongs to the first type of node, when the UI attribute data includes the drawing instructions of the first source-end component, the destination device loads the above-mentioned UI structure through the UI framework to construct (or layout) the multiple UI nodes, including creating the first destination-end component to obtain the first UI node, and drawing the image of the first destination-end component based on the drawing instructions of the first source-end component to obtain an image not rendered by the first source-end component.

[0153] The first destination component is a system pre-set component of the destination device, for example, the first destination component includes a first system pre-set component of the destination device. In a specific implementation, the first destination component includes one or more of a standard component, a special component, or components provided by other frameworks. The first destination component has the same or similar functions as the first source component mentioned above. The type of the first destination component can be the same as the type of the first source component, such as Figure 8 As shown, the first source component and the first destination component are both CustomPaint Node components. Of course, the type of the second component may be different from that of the first component.

[0154] In the case where the UI attribute data includes an image rendered by a first source-end component, the destination device loads the above-mentioned UI structure through a UI framework to construct the multiple UI nodes, which includes creating a second destination-end component to obtain a first UI node, and based on the image rendered by the first source-end component, setting the UI attributes of the second destination-end component so that the first UI node carries the image rendered by the first source-end component.

[0155] It should be understood that the image rendered by the first source-end component includes rendered pixels and may also include properties such as control size, animation, etc. The destination device sets the properties of the second destination-end component to set the rendered pixels, control size, animation, etc. of the first UI node into the first UI node, thereby obtaining the image rendered by the first source-end component.

[0156] The second destination component is also a system preset component of the destination device, for example, the second destination component includes a second system preset component of the destination device. In a specific implementation, the second destination component may also include one or more of a standard component, a special component, or components provided by other frameworks. The second destination component and the first source component may have different functions, and the type of the second destination component may be different from the type of the first source component, that is, the second destination component is different from the first source component. For example, the first source component is Figure 8 The CustomPaint Node component shown in the figure has a second destination component: Figure 8 The ImageNode component shown.

[0157] Next, the reconstruction method of the second type of nodes (corresponding to standard components) is introduced.

[0158] Taking the second UI node among the multiple UI nodes as an example, it can be seen from the above that the UI attribute data of the second UI node may include the drawing attribute data of the second UI node, such as the text, picture, animation, video, text attribute, cursor position, sliding position, color and other UI control status data of the second UI node. The destination device loads the above UI structure through the UI framework to construct the multiple UI nodes, including creating a third destination component to obtain the second UI node, and based on the drawing attribute data of the second UI node, setting the attributes of the third destination component through the UI framework to draw the corresponding image of the second UI node.

[0159] The third destination component is of the same type as the component used by the source device when drawing the second UI node (hereinafter referred to as the second source component). For example, the third destination component and the second source component are the same standard component. Of course, in some possible embodiments, the third destination component and the second source component may be of different types, as long as they have the same or similar functions.

[0160] Optionally, after constructing the UI structure of the first projection area, the destination device generates a unique projection area identifier for the first projection area, and the projection area identifier can be recorded as a destination profile identifier (DPID). The destination device records the correspondence between the SPID and the DPID of the first projection area, such as storing the SPID and the DPID of the first projection area in a page mapping table. Among them, in the case where the first projection area is the entire area of ​​the first page, the destination device generates a unique page identifier for the first page, that is, in this case, the above-mentioned projection area identifier is also called a page identifier.

[0161] Optionally, the destination device may also generate a unique node identifier for each of the multiple UI nodes constructed, and the node identifier may be recorded as a destination frame identifier (DFID). The destination device records the correspondence between the SFID and DFID of each UI node, such as storing the SFID and DFID of each UI node in a node mapping table.

[0162] The above describes the implementation process of projecting the first projection area of ​​the source device to the destination device through steps 701 to 705. Next, the projection process is described when the first projection area is updated, that is, the UI structure and / or UI attribute data of the first projection area is updated, and when the first projection area is switched to the second projection area.

[0163] When the UI structure and / or UI attribute data of the first projection area is updated, the source device sends the UI update data of the first projection area to the destination device, and the UI update data is used by the destination device to update the first projection area. The destination device receives the UI update data of the first projection area sent by the source device, and updates the first page based on the UI update data of the first projection area.

[0164] Among them, there are at least three situations in which the first projection area is updated. The first situation is that the destination device reversely controls the source device, the second situation is that the user operates the source device, and the third situation is that the source device automatically updates.

[0165] In the first case where the first projection area is updated (i.e., the destination device controls the source device), the destination device sends first control information to the source device in response to the destination user's first operation on the first projection area, and the first control information is used to instruct the source device to perform a second operation on the first projection area and update the UI structure and / or UI attribute data of the first projection area. The source device receives the first control information sent by the destination device, the first control information indicates the destination user's first operation on the first projection area, and the source device performs the second operation on the first projection area based on the first control information, and updates the UI structure and / or UI attribute data of the first projection area.

[0166] Among them, the second operation may be the same as or different from the first operation, and the objects of the second operation and the first operation may be the same or different. For example, in one example, the first projection area displayed by the destination device is playing a video, and the destination user long presses the right area of ​​the video screen to fast forward or rewind the video. After the long press operation of the destination user is reversed on the source device, the source device performs the fast forward or rewind operation, that is, in this example, the first operation is a long press operation, and the object of the first operation is the UI node corresponding to the right area of ​​the video screen. The second operation is a fast forward or rewind operation, and the object of the second operation is the UI node corresponding to the progress bar. In another example, the first projection area displayed by the destination device includes a pause button. The destination user clicks the pause button. After the click operation is reversed on the source device, the source device also performs the same click operation, that is, in this example, the first operation and the second operation are the same, both are click operations, and the objects of the first operation and the second operation are the same, both are UI nodes corresponding to the button.

[0167] Based on this, the first manipulation information sent by the destination device to the source device may include different information according to actual conditions. This will be introduced below.

[0168] As can be seen from the above, the UI structure of the first projection area includes multiple UI nodes, and the UI attribute data includes the identifiers of the multiple UI nodes. In some cases, the first control information includes the identifier of the third UI node and the control instruction corresponding to the first operation, the control instruction indicates the second operation, and the third UI node is one of the multiple UI nodes. One implementation method for the source device to perform the second operation on the first projection area based on the first control information is: based on the identifier of the third UI node, according to the control instruction, perform the second operation on the third UI node. Specifically, the source device searches for the third UI node from the multiple UI nodes based on the identifier of the third UI node, and performs the second operation on the found third UI node according to the control instruction.

[0169] Exemplarily, the destination user clicks on the first UI control (i.e., the third UI node) in the first projection area displayed on the destination device, triggering a click event regarding the first UI control. The destination device sends the identifier of the first UI control and a click instruction corresponding to the click event to the source device in response to the destination user's click operation. The source device searches for the first UI control from multiple UI nodes based on the identifier of the first UI control, and performs a click operation on the first UI control according to the click instruction.

[0170] In the case where the destination device records the correspondence between the SFID and the DFID of each UI node, after detecting the user's first operation on the third UI node, the destination device searches for the SFID of the third UI node from the correspondence between the SFID and the DFID based on the DFID of the third UI node, obtains the manipulation instruction corresponding to the first operation, and sends the SFID of the third UI node and the manipulation instruction corresponding to the first operation to the source device. The source device searches for the third UI node from multiple UI nodes based on the SFID of the third UI node.

[0171] In other cases, the first manipulation information includes an identifier of a fourth UI node and operation information of the first operation, and the fourth UI node is one of the multiple UI nodes. Another implementation method in which the source device performs the second operation on the first projection area based on the first manipulation information is: parsing the operation information of the first operation to determine the second operation, and performing the second operation on the fourth UI node based on the identifier of the fourth UI node. The source device can find the fourth UI node from multiple UI nodes based on the SFID of the fourth UI node.

[0172] Exemplarily, the destination user long presses the right area of ​​the video screen in the first projection area, triggering a drag operation on the progress bar (i.e., the fourth UI node). The destination device responds to the destination user's long press operation by sending the identifier of the fourth UI node and operation information of the drag operation to the source device. The source device parses the operation information of the drag operation to determine the fast forward or fast rewind operation, and performs the fast forward or fast rewind operation on the progress bar based on the identifier of the progress bar.

[0173] In the second case where the first projection area is updated (i.e., the user operates the source device), the source device updates the UI structure and / or UI attribute data of the first page in response to the source user's third operation on the first projection area.

[0174] It should be understood that the above-mentioned destination user refers to a user who operates the destination device, and the source user refers to a user who operates the source device. The destination user and the source user may be the same user or different users.

[0175] In the third case where the first projection area is updated (i.e., the source device is automatically updated), the source device may automatically update the first projection area based on the application code or application default settings, for example, the source device automatically switches the scrolling image in the first projection area.

[0176] The first case where the first projection area is updated is triggered by the destination device, while the second and third cases are triggered by the source device. Therefore, the second and third cases can be classified into one category, namely, the case where the source device triggers the update. In other words, the three cases can be considered as two categories, one is the case where the destination device triggers the update, and the other is the case where the source device triggers the update.

[0177] In order to reduce the amount of UI data transmitted from the source device to the destination device, reduce the lag of the destination device, and improve the screen projection effect, when the first projection area is updated, the source device can synchronize the changed part of the first projection area to the destination device, and the unchanged part will not be sent repeatedly. That is, the UI update data of the first projection area indicates the changed UI nodes and UI attribute data among the multiple UI nodes. The destination device updates the changed UI nodes in the first projection area based on the UI update data of the first projection area.

[0178] Exemplarily, the first UI node in the first projection area is a UI icon. After the user clicks the first UI node on the source device, the color or transparency of the first UI node changes, for example, the color changes from gray to blue. Then, the UI update data of the first projection area includes the identifier of the first UI node and the updated color information.

[0179] The source device may serialize the UI update data of the first projection area and obtain serialized page update data, and send it to the destination device. The destination device deserializes the received serialized page update data to obtain the UI update data of the first projection area. In addition, the source device may send a page update message to the destination device, and the page update message carries the serialized page update data.

[0180] Take the first projection area as the entire area of ​​the first page in the first application as an example. Fig.10 This is a screen projection flow chart for a page update provided in an embodiment of the present application. Fig.10 The above two situations of page update are shown.

[0181] See also Fig.10In the first case, the user operates the destination device, and the destination device responds to the user operation, determines the DFID of the UI node corresponding to the user operation, and finds the SFID of the UI node in the node mapping table based on the DFID of the UI node. The destination device obtains the control instruction corresponding to the control event of the UI node, and sends the SFID of the UI node and the corresponding control instruction to the source device. After the source device receives the SFID of the UI node and the corresponding control instruction, it finds the corresponding UI node based on the SFID of the UI node, and performs the corresponding operation on the UI node according to the control instruction, that is, injects the control event into the UI node. The source device updates the logic of the first application to determine whether a page switch occurs, that is, whether a new page is entered. If a new page is entered, the source device executes the UI construction process of the new page. If a new page is not entered, the source device obtains the update data of the UI structure of the current page, and obtains the update data of the UI attribute data of the current page, serializes the obtained update data to obtain serialized page update data, and sends a page update message to the destination device, and the page update message carries the above-mentioned serialized page update data. The destination device receives and parses the page update message to obtain serialized page update data, deserializes the serialized page update data to obtain the update data of the current page. The destination device updates the UI structure of the current page based on the update data of the current page, and updates the properties of the corresponding UI nodes, and then refreshes the displayed page.

[0182] In the second case, the user operates the source device, and the source device responds to the user operation, updates the logic of the first application, and determines whether a page switch has occurred, that is, whether a new page has been entered. Alternatively, the source device updates the logic of the first application according to the application code or the default setting of the application, and determines whether a page switch has occurred after the update. Then, the displayed page is finally refreshed on the destination device according to a similar process as in the first case.

[0183] When the projection area is switched from the first projection area to the second projection area (for example, the first page is switched to the second page), the source device constructs the UI structure of the second projection area and generates UI attribute data of the second projection area, and sends the UI structure and UI attribute data of the second projection area to the destination device. The above-mentioned application resources and the UI structure and UI attribute data of the second projection area are used by the destination device to reconstruct and render the second projection area through the UI framework. That is, the source device performs operations similar to the above-mentioned steps 702 and 703 to synchronize the UI structure and UI attribute data of the second projection area after switching to the destination device. The destination device receives the UI structure and UI attribute data of the second projection area, and based on the above-mentioned application resources and the UI structure and UI attribute data of the second projection area, reconstructs and renders the second projection area through the UI framework of the destination device. That is, the destination device performs operations similar to the above-mentioned steps 704 and 704 to switch the projection area.

[0184] Here, taking the switching from the first page to the second page as an example, the page switching includes page jump, page rollback, etc.

[0185] Next, please combine Figure 11 to Figure 12 The screen projection method provided in the embodiment of the present application is explained as an example. It should be noted that: Fig.11 The following introduction takes the projection area as the entire area of ​​the page as an example.

[0186] Fig.11 is a flowchart of another screen projection method provided by an embodiment of the present application. Fig.11 , the method comprises the following steps.

[0187] Step 1101: The source device executes the application logic portion of the first application.

[0188] Step 1102: The source device constructs a UI page, that is, constructs the page to be projected by loading its own system UI framework.

[0189] Step 1103: The source device obtains the UI structure of the constructed UI page. The UI structure may be a tree structure or a graph structure. The UI structure includes parent-child relationships, layout data, layer relationships, etc. between multiple UI nodes. The embodiment of the present application may obtain the UI structure of the page by directly obtaining the page structure information of the UI page or by obtaining the UI creation instruction. The source device may also generate and record the SPID of the constructed UI page.

[0190] Step 1104: The source device obtains UI attribute data of the constructed UI page. The UI attribute data includes the identification, type, status and other data of each UI node in the UI page. Among them, the status data includes the rendered image or drawing instruction / task corresponding to the text, picture, media, non-standard component, text attribute, cursor position, sliding position, text, color, control size, animation and other UI control states. The source device can also generate and record the SFID of each UI node.

[0191] Step 1105: The source device serializes the data to be sent (including the UI structure and UI attribute data of the page to be projected) to obtain serialized page data. Serialization may refer to assembling (or encoding) into a string or binary Buffer or file.

[0192] Step 1106: The source device sends a page reconstruction message to the destination device via the network, where the page reconstruction message carries the above serialized page data.

[0193] Step 1107: The destination device receives and parses the page reconstruction message from the source device to obtain serialized page data.

[0194] Step 1108: The destination device deserializes the serialized page data to obtain the UI structure and UI attribute data of the current page.

[0195] Step 1109: The destination device rebuilds the UI structure of the page through the system UI framework based on the UI structure of the current page or the UI creation instruction, and generates the DPID of the page. Optionally, the destination device records the DPID in the page mapping table.<SPID,DPID> .

[0196] Step 1110: The destination device sets the properties of each UI node in the current page based on the UI property data of the current page, including creating or configuring the UI node type and setting the UI property data to the UI node. For non-standard components, the destination device may draw the corresponding UI node based on the drawing instructions corresponding to the non-standard components, or set the corresponding rendered image, control size or animation on the corresponding UI node. The destination device may also generate and record the DFID of each UI node. Optionally, the destination device records the DFID in the node mapping table.<SFID,DFID> .

[0197] Step 1111: The destination device refreshes the displayed page, that is, renders the displayed page, such as displaying the page through a UI display module. The destination device can dynamically set the size of the display window, and the page can adapt to the window size when displayed in the display window.

[0198] It should be noted that if the projection mode of the first page in the first application is UI data stream mode or mixed mode, then the source device has sent the application resources of the first application to the destination device before projecting the second page. In this case, there is no need to repeatedly send the application resources of the first application to the destination device during the projection process after the first page. Fig.11 The embodiment shows the screen projection process of a page after the first page.

[0199] In an embodiment of the present application, after the screen projection is completed, for example, after the screen projection connection between the source device and the destination device is disconnected, the destination device may choose to delete all or part of the received data, for example, the destination device releases the UI display module. In other words, after the screen projection is completed, the source device may recycle or manage the resources of the other end. Optionally, the destination device determines whether to delete the received data based on the security level and / or user settings.

[0200] Exemplarily, after a screen projection connection is established between a source device and a destination device, a first prompt message is displayed on the source device, and the first prompt message is used to instruct the user to choose whether to trust the destination device. If the user chooses to trust the destination device by operating the source device, the destination device does not delete the received data after the screen projection is completed. If the user chooses not to trust the destination device by operating the source device, the destination device deletes the received data after the screen projection is completed. Alternatively, after the screen projection is completed, the source device displays a second prompt message, and the second prompt message indicates whether the user chooses to delete the data received by the destination device. If the user chooses to delete by operating the source device, the destination device deletes the received data after the screen projection is completed. If the user chooses not to delete by operating the source device, the destination device does not delete the received data after the screen projection is completed.

[0201] Fig.12 It is a schematic diagram of the core steps of a screen projection method provided in an embodiment of the present application. Fig.12 This scheme is combined with Figure 4 A schematic diagram of the hardware and software architecture of the electronic device shown. Fig.12 In one implementation of the screen projection method, the source device and the destination device both include the following five steps: negotiate the current screen projection mode by interacting with the software and hardware capabilities and the operating system and UI framework versions; synchronize application resources and configuration data; build and synchronize UI structures; update UI nodes / events; disconnect the screen projection connection, and recycle or manage the destination resources. Among them, the first and fifth steps can be implemented through the service management modules at both ends, and the second to fourth steps can be implemented through the UI frameworks (including UI proxy modules) and network synchronization modules at both ends.

[0202] In order to verify the performance of the UI data flow mode provided in the embodiment of the present application, a comparative test of the UI data flow mode and related technologies was conducted on the open source OpenHarmony system and ArkUI framework. The test results are shown in Table 1.

[0203] Table 1

[0204]

[0205]

[0206] Among them, the test indicators include network transmission data volume, reverse control delay, source power consumption and image quality. The mirror mode in the related technology needs to transmit video streams (also called media streams) through video encoding. I frames and P frames represent key frames and non-key frames in video encoding, respectively. Each I frame is about 50-100 kilobytes (KB), and each P frame is about 10-20KB. Based on the number of I frames and P frames that can be transmitted per second in mirror mode, the network transmission volume is about 1-2 megabytes per second (MB / s). The network transmission data volume of the UI data flow mode in this solution is about 10-100KB / page in the case of UI reconstruction, and the network transmission data volume in the case of page update is basically less than 1KB / page. In addition, the mirror mode needs to transmit every frame when the screen changes, while the UI data stream mode only transmits data when the UI attribute data of the page is updated or the page is switched. For example, if a page includes a dynamic image, and the screen of the page is constantly changing because of the dynamic image, then the mirror mode needs to transmit every frame, while the UI data stream mode only needs to transmit the UI attribute data of the dynamic image through the UI data stream after the UI node of the dynamic image is first created, and there is no need to transmit the UI-related data of the dynamic image later. It can be seen that the amount of network transmission data in the UI data stream mode is greatly reduced compared to the mirror mode.

[0207] The reverse control delay refers to the time from the target user operation to the target device updating the page. In the relevant technology, the reverse control delay in office collaboration application scenarios is about 110-120 milliseconds (ms), and the reverse control delay of the UI data stream mode in this solution is about 60-80ms. That is, the reverse control delay of this solution is shorter, the response speed of the target device is faster, and the jamming phenomenon is reduced.

[0208] The power consumption of the source end refers to the power consumption of the source device. In the related art, the power consumption of the source end in the shopping and office application scenario is about 110 milliamperes (mA), and in the video application scenario, the power consumption of the source end is about 220mA, that is, the power consumption of the source end in the related art is about 110-220mA. In this solution, the power consumption of the source end in the UI data stream mode is about 30-70mA.

[0209] When the screen resolution of the source device is 2K and the refresh rate is 60 Hz, the target device in the mirroring mode in the related technology cannot adapt to the resolution, and there will be freezes or screen distortion. However, this solution can achieve native 2K resolution and 60Hz refresh rate on the target device through screen resolution adaptation without freezes or screen distortion.

[0210] Fig.13 This is a schematic diagram of the screen projection area on the source device provided in an embodiment of the present application. Fig.14 This is a schematic diagram of the screen projection effect on the target device in the present solution provided in an embodiment of the present application. Fig.15 Schematic diagram of a screen projection effect of the mirror mode on the target device in the related technology provided by the embodiment of the present application. Figures 13 to 15 It can be clearly seen that the screen will not be stretched after projection in this solution, while the screen will be stretched in the related technology. This is because in this solution, the target device automatically layouts and renders each UI node in the display screen by means of UI reconstruction, and the reconstructed UI nodes are adaptive to the screen resolution of the target device.

[0211] Fig.16 This is another schematic diagram of the screen projection area of ​​the source device in the present solution provided in an embodiment of the present application. Fig.17 This is the solution provided in the embodiment of the present application. Fig.16 A schematic diagram of the projection area of ​​the corresponding target device. Fig.16 and Fig.17 It can be seen that in this solution, there may be some differences between the layout of the above-mentioned multiple UI nodes on the source device and the destination device, that is, the destination device can adaptively adjust the layout of the multiple UI nodes according to information such as screen resolution. For example, the display positions of some UI nodes on the source device and the destination device are different, such as being displayed in the center on the source device and displayed to the right on the destination device.

[0212] To summarize, in an embodiment of the present application, the source device sends the UI structure and UI attribute data of the projection area to the destination device, so that the destination device can rebuild and render the corresponding projection area based on the UI structure and UI attribute data at the UI level. This solution does not require the source device to perform video encoding and transmission after rendering the image, which reduces the power consumption of the source device. It also does not require the source device to send user-related data to the destination device, reducing the risk of user privacy data leakage. Moreover, this solution can also achieve screen projection when the destination device does not have an application that requires screen projection installed, thereby supporting multi-device collaboration. In addition, the destination device renders and displays the projection area by itself, which can adapt to the screen resolution of the destination device and improve the display effect.

[0213] Fig.18is a schematic diagram of a screen projection device provided in an embodiment of the present application. The screen projection device can be implemented as part or all of an electronic device by software, hardware, or a combination of both. The electronic device can be Figure 5 In the embodiment of the present application, the screen projection device is included in the source device. Fig.18 , the device includes: a sending module 1801.

[0214] The sending module 1801 is used to send the user interface UI structure and UI attribute data of the first projection area to the destination device, so as to project the first projection area to the destination device;

[0215] Among them, the UI structure indicates the layout relationship between multiple UI nodes in the first projection area, the UI attribute data indicates the rendering-related properties of each UI node in the multiple UI nodes, and the UI structure and UI attribute data are used to display the first projection area on the target device.

[0216] Optionally, the device further comprises:

[0217] The construction module is used to construct the UI structure and generate UI attribute data through the UI framework of the source device according to the application instruction.

[0218] Optionally, the multiple UI nodes include a first UI node corresponding to a first source-end component, the first source-end component belongs to a first-type component, and a corresponding image of the first-type component is drawn by a non-UI framework;

[0219] The UI property data includes a drawing instruction of the first source-end component, where the drawing instruction of the first source-end component is used to draw an image not rendered by the first source-end component through a non-UI framework; or, the UI property data includes an image rendered by the first source-end component.

[0220] Optionally, the multiple UI nodes include a second UI node corresponding to a second source-end component, the second source-end component belongs to a second-category component, and a corresponding image is drawn by the UI framework for the second-category component;

[0221] The UI attribute data includes drawing attribute data of the second source-end component, and the drawing attribute data of the second source-end component is used to draw a corresponding image of the second source-end component through the UI framework.

[0222] Optionally, the source device runs a first application, the first projection area is a projection area of ​​the first application, and the sending module 1801 is further used to:

[0223] The application resources of the first application are sent to the destination device, where the application resources include the resources required for rendering the multiple UI nodes, and the application resources and the UI attribute data are used together by the destination device to reconstruct and render the first projection area.

[0224] Optionally, the device further comprises:

[0225] A receiving module, configured to receive first manipulation information sent by a destination device, where the first manipulation information indicates a first operation performed by a destination user on a first projection area;

[0226] An updating module, configured to perform a second operation on the first projection area based on the first manipulation information, and update a UI structure and / or UI attribute data of the first projection area;

[0227] The sending module 1801 is also used to send UI update data of the first projection area to the destination device, and the UI update data is used by the destination device to update the first projection area through the UI framework of the destination device.

[0228] Optionally, the UI attribute data includes identifiers of multiple UI nodes, the first manipulation information includes an identifier of a third UI node and a manipulation instruction corresponding to the first operation, the manipulation instruction indicates the second operation, and the third UI node is one of the multiple UI nodes;

[0229] The update module is specifically used for: performing a second operation on the third UI node based on the identifier of the third UI node and in accordance with the manipulation instruction.

[0230] Optionally, the UI attribute data includes identifiers of the multiple UI nodes, the first manipulation information includes an identifier of a fourth UI node and operation information of the first operation, and the fourth UI node is one of the multiple UI nodes;

[0231] The update module is specifically used to:

[0232] Parsing the operation information of the first operation to determine the second operation;

[0233] Based on the identifier of the fourth UI node, a second operation is performed on the fourth UI node.

[0234] Optionally, the source device runs a first application, the first projection area is a projection area in a first page of the first application, and the sending module 1801 is further used to:

[0235] When the first application switches from the first page to the second page, the UI structure and UI attribute data of the second projection area are sent to the destination device to project the second projection area to the destination device. The second projection area is the projection area in the second page.

[0236] Optionally, the source device supports multiple screen projection modes; the apparatus further includes:

[0237] The display module is used to display the current projection mode, which is one of multiple projection modes.

[0238] Optionally, the source device supports multiple screen projection modes, and different screen projection modes are different in different screen projection areas; the apparatus further includes:

[0239] A determination module is used to determine the projection mode of the first projection area from multiple projection modes through a mode adaptation algorithm.

[0240] In an embodiment of the present application, the source device sends the UI structure and UI attribute data of the projection area to the destination device, so that the destination device can rebuild and render the corresponding projection area based on the UI structure and UI attribute data at the UI level. This solution does not require the source device to perform video encoding and transmission after rendering the image, which reduces the power consumption of the source device. It also does not require the source device to send user-related data to the destination device, reducing the risk of user privacy data leakage. Moreover, this solution can also achieve screen projection when the destination device does not have an application that requires screen projection installed, thereby supporting multi-device collaboration. In addition, the destination device renders and displays the projection area by itself, which can adapt to the screen resolution of the destination device and improve the display effect.

[0241] It should be noted that: the screen projection device provided in the above embodiment only uses the division of the above functional modules as an example when projecting the screen. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the screen projection device provided in the above embodiment and the screen projection method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0242] Fig.19 is a structural diagram of another screen projection device provided in an embodiment of the present application. The screen projection device can be implemented as part or all of an electronic device by software, hardware, or a combination of both. The electronic device can be Figure 5 In the embodiment of the present application, the device is included in the target device. Fig.19 , the device includes: a receiving module 1901 and a display module 1902.

[0243] Receiving module 1901, used to receive a user interface UI structure and UI attribute data of the first projection area sent by a source device, where the UI structure indicates a layout relationship between multiple UI nodes in the first projection area, and the UI attribute data indicates rendering-related attributes of each UI node in the multiple UI nodes;

[0244] Display module 1902 is used to display the first projection area based on the UI structure and UI attribute data.

[0245] Optionally, the display module 1902 is specifically used for:

[0246] Loading a UI structure through a UI framework of a destination device to layout a plurality of UI nodes in a first display area;

[0247] The attributes of each UI node in the multiple UI nodes are set based on the UI attribute data to reconstruct and render the first projection area.

[0248] Optionally, the multiple UI nodes include a first UI node corresponding to a first source-end component, the first source-end component belongs to a first-type component, and a corresponding image of the first-type component is drawn by a non-UI framework;

[0249] The UI property data includes a drawing instruction of the first source-end component, where the drawing instruction of the first source-end component is used to draw an image not rendered by the first source-end component through a non-UI framework; or, the UI property data includes an image rendered by the first source-end component.

[0250] Optionally, when the UI attribute data includes a drawing instruction of the first source-end component, the display module 1902 is specifically configured to:

[0251] Loading the UI structure through the UI framework to create a first destination component to obtain a first UI node;

[0252] Based on the drawing instruction of the first source-end component, the image of the first destination-end component is drawn through the non-UI framework to obtain an unrendered image of the first UI node.

[0253] Optionally, when the UI attribute data includes an image rendered by the first source-end component, the display module 1902 is specifically configured to:

[0254] Loading the UI structure through the UI framework to create a second destination component to obtain the first UI node, where the second destination component is different from the first source component;

[0255] Based on the image rendered by the first source-end component, by setting the UI property of the second destination-end component, the first UI node carries the image rendered by the first source-end component.

[0256] Optionally, the multiple UI nodes include a second UI node corresponding to a second source-end component, the second source-end component belongs to a second-category component, a corresponding image is drawn by a UI framework for the second-category component, and the UI attribute data includes drawing attribute data of the second source-end component;

[0257] The display module 1902 is specifically used for:

[0258] Loading the UI structure through the UI framework to create a third destination component to obtain a second UI node, where the third destination component is the same as the second source component;

[0259] Based on the drawing attribute data of the second source-end component, the attributes of the second source-end component are set through the UI framework to draw a corresponding image of the second UI node.

[0260] Optionally, the device further comprises:

[0261] A sending module, configured to send first manipulation information to a source device in response to a first operation of a destination user on a first projection area, wherein the first manipulation information is used to instruct the source device to perform a second operation on the first projection area and update a UI structure and / or UI attribute data of the first projection area;

[0262] The receiving module 1901 is further used to receive UI update data of the first projection area sent by the source device;

[0263] An update module is used to update the first projection area based on the UI update data of the first projection area.

[0264] Optionally, the source device supports multiple screen projection modes, and different screen projection modes are different in different screen projection areas; the apparatus further includes:

[0265] The sending module is used to send the real-time computing capability information of the destination device to the source device. The real-time computing capability information is used by the source device to determine the projection mode of the current projection area from multiple projection modes through a mode adaptation algorithm.

[0266] In an embodiment of the present application, the source device sends the UI structure and UI attribute data of the projection area to the destination device, so that the destination device can rebuild and render the corresponding projection area based on the UI structure and UI attribute data at the UI level. This solution does not require the source device to perform video encoding and transmission after rendering the image, which reduces the power consumption of the source device. It also does not require the source device to send user-related data to the destination device, reducing the risk of user privacy data leakage. Moreover, this solution can also achieve screen projection when the destination device does not have an application that requires screen projection installed, thereby supporting multi-device collaboration. In addition, the destination device renders and displays the projection area by itself, which can adapt to the screen resolution of the destination device and improve the display effect.

[0267] It should be noted that: the screen projection device provided in the above embodiment only uses the division of the above functional modules as an example when projecting the screen. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the screen projection device provided in the above embodiment and the screen projection method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0268] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiment of the present application may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0269] It should be understood that the "at least one" mentioned herein refers to one or more, and "multiple" refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.

[0270] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0271] The above-mentioned embodiments are provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A screen projection system, characterized in that: The system includes a source-end screen projection module and a destination-end screen projection module; The source-end screen projection module is used to send the user interface UI structure and UI attribute data of the first screen projection area to the destination-end screen projection module, so as to project the first screen projection area from the source device where the source-end screen projection module is located to the destination device where the destination-end screen projection module is located, wherein the UI structure indicates the layout relationship between multiple UI nodes in the first screen projection area, and the UI attribute data indicates the rendering-related attributes of each UI node in the multiple UI nodes; The target-end screen projection module is used to receive the UI structure and UI attribute data, and display the first screen projection area on the target device based on the UI structure and the UI attribute data.

2. The system according to claim 1, characterized in that The target end screen projection module is specifically used for: The UI structure is loaded through the UI framework of the target device to layout the multiple UI nodes in the first display area, and the properties of each UI node in the multiple UI nodes are set based on the UI property data to reconstruct and render the first projection area.

3. The system according to claim 1 or 2, characterized in that: The source-end screen projection module is also used to construct the UI structure and generate the UI attribute data through the UI framework of the source device according to application instructions.

4. The system according to any one of claims 1 to 3, characterized in that: The destination-end screen projection module is also used to send first control information to the source-end screen projection module, so as to update part or all of the UI nodes in the first screen projection area by reversely controlling the source device.

5. The system according to any one of claims 1 to 4, characterized in that: The source device runs a first application, and the first screen projection area is the screen projection area of ​​the first application.

6. The system according to claim 5, characterized in that The destination device does not have the first application installed.

7. The system according to any one of claims 1 to 6, characterized in that: The first projection area is part or all of the area of ​​the first page.

8. A screen projection method, characterized in that: Applied to a source device, the method comprises: Sending the user interface UI structure and UI attribute data of the first projection area to the destination device to project the first projection area to the destination device; Among them, the UI structure indicates the layout relationship between multiple UI nodes in the first projection area, the UI attribute data indicates the rendering related properties of each UI node in the multiple UI nodes, and the UI structure and the UI attribute data are used to display the first projection area on the destination device.

9. The method according to claim 8, characterized in that Before sending the UI structure and UI attribute data of the first projection area to the destination device, the method further includes: According to the application instruction, the UI structure is constructed and the UI attribute data is generated through the UI framework of the source device.

10. The method according to claim 8 or 9, characterized in that The multiple UI nodes include a first UI node corresponding to a first source-end component, the first source-end component belongs to a first type of component, and a corresponding image of the first type of component is drawn by a non-UI framework; The UI attribute data includes a drawing instruction of the first source-end component, where the drawing instruction of the first source-end component is used to draw an image not rendered by the first source-end component through the non-UI framework; or, The UI attribute data includes an image rendered by the first source-end component.

11. The method according to any one of claims 8 to 10, characterized in that: The multiple UI nodes include a second UI node corresponding to a second source-end component, the second source-end component belongs to a second type of component, and the second type of component has a corresponding image drawn by the UI framework; The UI attribute data includes drawing attribute data of the second source-end component, and the drawing attribute data of the second source-end component is used to draw a corresponding image of the second source-end component through the UI framework.

12. The method according to any one of claims 8 to 11, characterized in that: The source device runs a first application, the first screen projection area is a screen projection area of ​​the first application, and the method further includes: Send application resources of the first application to the destination device, the application resources including resources required for rendering the multiple UI nodes, and the application resources and the UI attribute data are used together by the destination device to reconstruct and render and display the first projection area.

13. The method according to any one of claims 8 to 12, characterized in that: The method further comprises: Receiving first manipulation information sent by the destination device, where the first manipulation information indicates a first operation of a destination user on the first projection area; Based on the first manipulation information, perform a second operation on the first projection area, and update the UI structure and / or UI attribute data of the first projection area; Send UI update data of the first screen projection area to the destination device, where the UI update data is used by the destination device to update the first screen projection area through the UI framework of the destination device.

14. The method according to claim 13, characterized in that The UI attribute data includes identifiers of the multiple UI nodes, the first manipulation information includes an identifier of a third UI node and a manipulation instruction corresponding to the first operation, the manipulation instruction indicates the second operation, and the third UI node is one of the multiple UI nodes; The performing a second operation on the first projection area based on the first manipulation information includes: Based on the identifier of the third UI node, and in accordance with the manipulation instruction, the second operation is performed on the third UI node.

15. The method according to claim 13, characterized in that The UI attribute data includes identifiers of the multiple UI nodes, the first manipulation information includes an identifier of a fourth UI node and operation information of the first operation, and the fourth UI node is one of the multiple UI nodes; The performing a second operation on the first projection area based on the first manipulation information includes: parsing the operation information of the first operation to determine the second operation; Based on the identifier of the fourth UI node, the second operation is performed on the fourth UI node.

16. The method according to any one of claims 8 to 15, characterized in that: The source device runs a first application, the first screen projection area is a screen projection area in a first page of the first application, and the method further includes: When the first application switches from the first page to the second page, the UI structure and UI attribute data of the second projection area are sent to the destination device to project the second projection area to the destination device, and the second projection area is the projection area in the second page.

17. The method according to any one of claims 8 to 16, characterized in that: The source device supports multiple screen projection modes; and the method further includes: Display the current screen projection mode, where the current screen projection mode is one of the multiple screen projection modes.

18. The method according to any one of claims 8 to 17, characterized in that: The source device supports multiple screen projection modes, and different screen projection areas have different screen projection modes; Before sending the user interface UI structure and UI attribute data of the first projection area to the destination device, the method further includes: The projection mode of the first projection area is determined from the multiple projection modes through a mode adaptation algorithm.

19. A screen projection method, characterized in that: Applied to a target device, the method comprises: A receiving source device sends a user interface UI structure and UI attribute data of a first projection area, wherein the UI structure indicates a layout relationship between multiple UI nodes in the first projection area, and the UI attribute data indicates rendering-related attributes of each UI node in the multiple UI nodes; The first projection area is displayed based on the UI structure and the UI attribute data.

20. The method of claim 19, wherein: The displaying the first projection area based on the UI structure and the UI attribute data includes: Loading the UI structure through the UI framework of the destination device to layout the multiple UI nodes in the first display area; The attributes of each UI node among the multiple UI nodes are set based on the UI attribute data to reconstruct and render the first projection area.

21. The method according to claim 19 or 20, characterized in that The multiple UI nodes include a first UI node corresponding to a first source-end component, the first source-end component belongs to a first type of component, and a corresponding image of the first type of component is drawn by a non-UI framework; The UI attribute data includes a drawing instruction of the first source-end component, where the drawing instruction of the first source-end component is used to draw an image not rendered by the first source-end component through the non-UI framework; or, The UI attribute data includes an image rendered by the first source-end component.

22. The method according to claim 21, characterized in that In a case where the UI attribute data includes a drawing instruction of the first source-end component, loading the UI structure through a UI framework of the destination device to layout the first UI node in a first display area includes: Loading the UI structure through the UI framework to create a first destination component to obtain the first UI node; Setting the attribute of the first UI node based on the UI attribute data includes: Based on the drawing instruction of the first source-end component, the image of the first destination-end component is drawn through the non-UI framework to obtain an unrendered image of the first UI node.

23. The method according to claim 19 or 20, characterized in that In a case where the UI attribute data includes an image rendered by the first source-end component, loading the UI structure through a UI framework of the destination device to layout the first UI node in the first display area includes: Loading the UI structure through the UI framework to create a second destination component to obtain the first UI node, where the second destination component is different from the first source component; Setting the attribute of the first UI node based on the UI attribute data includes: Based on the image rendered by the first source-end component, by setting the UI property of the second destination-end component, the first UI node carries the image rendered by the first source-end component.

24. The method according to any one of claims 19 to 23, characterized in that: The multiple UI nodes include a second UI node corresponding to a second source-end component, the second source-end component belongs to a second type of component, the second type of component has a corresponding image drawn by the UI framework, and the UI attribute data includes drawing attribute data of the second source-end component; Loading the UI structure through the UI framework of the target device to layout the second UI node in the first display area includes: Loading the UI structure through the UI framework to create a third destination component to obtain the second UI node, the third destination component being the same as the second source component; Setting the attribute of the second UI node based on the UI attribute data includes: Based on the drawing attribute data of the second source-end component, the attributes of the second source-end component are set through the UI framework to draw a corresponding image of the second UI node.

25. The method according to any one of claims 19 to 24, characterized in that The method further comprises: In response to a first operation of a destination user on the first projection area, sending first manipulation information to the source device, where the first manipulation information is used to instruct the source device to perform a second operation on the first projection area and update a UI structure and / or UI attribute data of the first projection area; Receiving UI update data of the first screen projection area sent by the source device; Update the first projection area based on the UI update data of the first projection area.

26. The method according to any one of claims 19 to 25, characterized in that The source device supports multiple screen projection modes, and different screen projection areas have different screen projection modes; the method further includes: The real-time computing capability information of the destination device is sent to the source device, where the real-time computing capability information is used by the source device to determine the projection mode of the current projection area from the multiple projection modes through a mode adaptation algorithm.

27. A screen projection device, characterized in that: Included in a source device, the apparatus comprises: A sending module, used to send the user interface UI structure and UI attribute data of the first projection area to the destination device, so as to project the first projection area to the destination device; Among them, the UI structure indicates the layout relationship between multiple UI nodes in the first projection area, the UI attribute data indicates the rendering related properties of each UI node in the multiple UI nodes, and the UI structure and the UI attribute data are used to display the first projection area on the destination device.

28. A screen projection device, characterized in that: Included in the destination device, the device includes: A receiving module, configured to receive a user interface UI structure and UI attribute data of a first projection area sent by a source device, wherein the UI structure indicates a layout relationship between a plurality of UI nodes in the first projection area, and the UI attribute data indicates a rendering-related attribute of each UI node in the plurality of UI nodes; A display module is used to display the first projection area based on the UI structure and the UI attribute data.

29. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the steps of any one of the methods of claims 8 to 18, or implements the steps of any one of the methods of claims 19 to 26.

30. A computer program product, characterized in that The computer program product stores computer instructions, and when the computer instructions are executed by a processor, the steps of the method described in any one of claims 8 to 18 are implemented, or the steps of the method described in any one of claims 19 to 26 are implemented.

Images