Cross-process rendering method

Abstract

The invention relates to a cross-process rendering method. The method comprises the steps that a first process sends a rendering request of a first rendering unit to a service process; the service process receives a rendering request of the first rendering unit, creates a first surface view and transmits surface control information of the first surface view to the first process; the first process receives surface control information of the first surface view from the service process, associates a first surface of the first surface view to a second surface created by the first rendering unit by using the surface control information of the first surface view, and sends rendering data of the first rendering unit to a rendering engine associated with the service process; and according to the rendering data of the first rendering unit, the service process uses a rendering engine associated with the service process to render the content of the first rendering unit on the first surface of the first surface view so as to enable a second surface associated with the first surface to display a rendering result. By utilizing the method disclosed by the embodiment of the invention, the high efficiency and the reliability of the cross-process rendering process can be ensured.

Description

Technical Field

[0001] The present disclosure relates to a cross-process rendering method, a computing device, a storage medium, a program product, and the like. Background Art

[0002] Traditional rendering methods are usually limited to completing graphics rendering tasks within a single process, especially on mobile platforms, such as applications using rendering engines in the Android system.

[0003] In traditional rendering architectures, rendering is typically done within a single process, such as a render thread within that process. The Unity engine performs tasks like scene drawing, resource management, and logic processing within that thread, with all rendering operations occurring within the same process.

[0004] Because rendering is performed within a single process, all rendering resources (such as textures, shaders, and models) are stored and managed within that process's memory. This approach makes resource management relatively simple, as the rendering engine accesses and operates on data within the same memory space. However, exceptions in one of these processes, limited memory resources, or memory leaks can make it difficult to continue rendering tasks. Summary of the Invention

[0005] Embodiments of the present disclosure provide cross-process rendering methods and corresponding computing devices, non-transitory machine-readable storage media, and computer program products for executing these methods.

[0006] According to the first aspect of an embodiment of the present disclosure, a cross-process rendering method is provided, including: a first process sends a rendering request of a first rendering unit to a service process; the service process receives the rendering request of the first rendering unit sent by the first process; the service process creates a first surface view and passes surface control information of the first surface view to the first process; the first process receives the surface control information of the first surface view from the service process; the first process uses the surface control information of the first surface view to associate the first surface of the first surface view to the second surface created by the first rendering unit; the first process sends rendering data of the first rendering unit to a rendering engine associated with the service process; and the service process uses the rendering engine associated with the service process to render the content of the first rendering unit on the first surface of the first surface view based on the rendering data of the first rendering unit received from the first process so that the second surface associated with the first surface displays the rendering result.

[0007] Optionally, the method also includes: the second process sends a rendering request for the second rendering unit to the service process; the service process receives the rendering request for the second rendering unit sent by the second process; the service process creates a second surface view and passes surface control information of the second surface view to the second process; the second process receives surface control information of the second surface view from the service process; the second process uses the surface control information of the second surface view to associate the third surface of the second surface view to the fourth surface created by the second rendering unit; the second process sends rendering data of the second rendering unit to the rendering engine associated with the service process; and the service process uses the rendering engine to render the content of the second rendering unit on the third surface of the second surface view based on the rendering data of the second rendering unit received from the second process so that the fourth surface associated with the third surface displays the rendering result, wherein the first surface and the third surface are respectively bound to the first screen and the second screen of the rendering engine, so that the content of the first screen and the second screen are rendered to the first surface and the third surface respectively, wherein the content of the first screen and the second screen are the content of the first rendering unit and the second rendering unit rendered by the rendering engine respectively.

[0008] Optionally, the method also includes: after the rendering of the first rendering unit is completed, the third process sends a rendering request of the third rendering unit to the service process; the service process receives the rendering request of the third rendering unit sent by the third process; the service process passes the surface control information of the first surface view to the third process; the third process receives the surface control information of the first surface view from the service process; the third process uses the surface control information of the first surface view to associate the first surface of the first surface view to the fifth surface created by the third rendering unit; the third process sends the rendering data of the third rendering unit to the rendering engine associated with the service process; and the service process uses the rendering engine to render the content of the third rendering unit on the first surface based on the rendering data of the third rendering unit received from the third process so that the fifth surface associated with the first surface displays the rendering result.

[0009] Optionally, the rendering engine supports rendering of multiple screens, and a correspondence between multiple rendering units of multiple processes and multiple screens of the rendering engine is preset in the service process. After receiving the rendering request of the first rendering unit from the first process, the service process determines that the first rendering unit is to use the first screen of the rendering engine based on the correspondence. When the service process determines that the first screen has not been enabled, the service process creates the first surface view, binds the first surface of the first surface view to the first screen, and passes the surface control information of the first surface view to the first process. When the service process determines that the first screen has been enabled, the service process passes the surface control information of the first surface view that has been created and bound to the first screen to the first process.

[0010] Optionally, the service process creates a first surface view, including: the service process creates a first surface control information view holder, and the service process adds a new surface view as the first surface view in the view tree of the first surface control information view holder; and the service process passes the surface control information of the first surface view to the first process, including: the service process returns the surface package of the first surface control information view holder to the first process, wherein the surface package contains the surface control information of the first surface view.

[0011] Optionally, the method also includes: the service process obtains information about the first surface of the first surface view in the create surface callback of the first surface view; and the service process calls the update screen internal method of the player of the rendering engine, and uses the obtained information about the first surface to bind the first surface to the first screen of the rendering engine.

[0012] Optionally, the first process uses the surface control information of the first surface view to associate the first surface of the first surface view to the second surface created by the first rendering unit, including: the first process reparents the surface control information of the first surface view to the surface control information of the first rendering unit, thereby setting the first surface as a child node of the second surface.

[0013] Optionally, the first rendering unit is the third surface view to be rendered by the first process, and the second surface is the surface of the third surface view; the first process receives surface control information of the first surface view from the service process, including: the first process receives a surface package of a first surface control information view holder from the service process, wherein the first surface view is a surface view in the view tree of the first surface control information view holder, and the surface package contains the surface control information of the first surface view; and the first process uses the surface control information of the first surface view to associate the first surface of the first surface view to the second surface created by the first rendering unit, including: the first process calls a set sub-surface package method for the third surface view and the surface package, thereby setting the first surface as a child node of the second surface.

[0014] Optionally, the first rendering unit is the third surface view to be rendered by the first process, including: the first process creates a subclass of a frame layout, the first process adds the subclass to the activity layout to be rendered as a child node of the content view, the first process creates a surface view inside the subclass as the third surface view, and the first process calls an add view method to add the third surface view to the subview of the subclass.

[0015] Optionally, the first rendering unit is the window to be rendered by the first process, and the second surface is the surface of the window; the first process receives surface control information of the first surface view created by the service process from the service process, including: the first process receives a surface encapsulation of a first surface control information view holder from the service process, wherein the first surface view is a surface view in the view tree of the first surface control information view holder, and the surface encapsulation contains the surface control information of the first surface view; and the first process uses the surface control information of the first surface view to associate the first surface of the first surface view to the second surface created by the first rendering unit, including: the first process obtains the surface control information of the window and the surface control information in the surface encapsulation, and the first process uses the method in the surface control transaction to reparent the surface control information in the surface encapsulation to the surface control information of the window, thereby setting the first surface as a child node of the surface of the window.

[0016] Optionally, the first process is a process corresponding to the wallpaper service, and the first process obtains the surface control information of the window and the surface control information in the surface package, including: the first process inherits the engine class of the wallpaper service to create an instance of the engine, the first process obtains the surface control member variable as the surface control information of the window through reflection in the surface creation callback of the engine instance, and the first process obtains the surface control information in the surface package by reflection using a method for obtaining the surface control information.

[0017] Optionally, the method also includes: in response to detecting that the first process ends abnormally, the service process terminates the association between the first surface and the second surface; in response to detecting that the first process is restarted, the service process re-passes the surface control information of the first surface view to the first process, so that the first process uses the surface control information of the first surface view to associate the first surface to the second surface recreated by the first rendering unit.

[0018] According to a second aspect of an embodiment of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method described in any one of the schemes in the first aspect above.

[0019] According to a third aspect of an embodiment of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in any of the solutions in the first aspect above.

[0020] According to a fourth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising an executable code. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described in any one of the solutions in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0022] Figure 1 The structure diagram of a mobile phone according to at least one embodiment of the present disclosure is exemplarily shown.

[0023] Figure 2 A schematic flowchart of a cross-process rendering method according to at least one embodiment of the present disclosure is exemplarily shown.

[0024] Figure 3A schematic timing diagram of a cross-process rendering method according to at least one embodiment of the present disclosure is exemplarily shown.

[0025] Figure 4 The present invention exemplarily shows the architecture of a cross-process rendering method according to at least one embodiment of the present disclosure, as well as a schematic diagram of some organizational structures and data flows generated when executing the method.

[0026] Figure 5 A schematic diagram exemplarily shows the organizational structure of a surface view (SurfaceView) created in a service process and its relationship with a screen in a cross-process rendering method according to at least one embodiment of the present disclosure.

[0027] Figure 6 A schematic diagram exemplarily shows the organizational structure of a surface view created in a client process in a cross-process rendering method according to at least one embodiment of the present disclosure.

[0028] Figure 7 The figure exemplarily shows a structural diagram of a computing device according to at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0030] It should be understood that, in this disclosure, the terms "first," "second," "third," "fourth," etc., are used to distinguish similar objects, rather than to describe a specific order or precedence, and do not have any additional limiting effect. Furthermore, in this disclosure, the term "plurality" or similar terms refers to two or more.

[0031] On computing devices such as mobile phones, car computers, wearable devices, or other devices, when a running process needs to complete graphics rendering tasks, it can have a rendering thread to perform graphics rendering tasks. This rendering thread can use a rendering engine such as Unity to perform tasks such as scene drawing, resource management, and logic processing in the main thread. However, single-process rendering may bring the following problems:

[0032] For example, as application scenarios become more complex, the load of single-process rendering gradually increases. Especially when processing large scenes or high-resolution images, rendering tasks may occupy a large amount of CPU and GPU resources, resulting in system performance degradation and even unstable frame rates.

[0033] For example, in single-process mode, if a rendering operation encounters an exception or memory leak, the entire application may crash. Because all critical tasks are within a single process, the normal operation of other application functions cannot be guaranteed after a crash.

[0034] For example, in single-process rendering, all resources are shared in the same memory space, which can easily lead to resource contention and conflicts. Resource operations between multiple modules or components may affect each other, making it difficult to ensure the stability of resource management.

[0035] For example, because rendering tasks are concentrated in a single process, it is difficult to expand or introduce new rendering features without affecting the main application. In particular, the flexibility and adaptability of the single-process model are poor in application scenarios that require high concurrency or multitasking.

[0036] In response to the aforementioned problems with the single-process rendering method, related technologies have proposed cross-process rendering solutions, which aim to solve various technical problems in the traditional single-process rendering method to improve performance, stability and scalability.

[0037] For example, existing solutions transfer the client process's Surface directly to the service process, which then passes it to a rendering engine like Unity for rendering. However, while Surfaces can be transferred across processes, they must be released after use. Furthermore, Surfaces are prone to leaks (e.g., hijacking by hooks) after being transferred across processes. These issues require well-defined inter-process handling logic, otherwise stability issues such as memory leaks and Surface layer leaks can easily occur.

[0038] In other cases, the client process's Surface is frequently destroyed and recreated. This is because the Surface passed to the rendering engine, such as Unity, must follow the client process's Surface lifecycle, causing it to be frequently destroyed and recreated. This is especially true when the client process is restarted due to, for example, an abnormal shutdown. The application must recreate the objects and processes required for the rendering task and then pass the recreated Surface to the server. During this time, the rendering task cannot proceed, resulting in longer response times and a certain impact on the user experience.

[0039] In other examples, because rendering engines such as Unity generally do not perform special processing on Surfaces transferred across processes, they still consider them to be instances of this process. Moreover, the Unity engine is a black box, and users cannot see its internal processing of Surfaces, which can easily lead to compatibility issues.

[0040] In view of this, the present disclosure proposes a new cross-process rendering method, which does not require the transfer of Surface between processes. Instead, the service process creates a Surface and passes the control information of the service process's own Surface to other processes, so that the service process calls the rendering engine to provide rendering services for other processes. Since there is no need to transfer the Surface between processes, the aforementioned stability and compatibility issues are avoided. In addition, since the rendering engine renders on the Surface of the service process, there is no need to frequently destroy and rebuild the Surface rendered by the Unity engine following the Surface life cycle of the client process, and the service process can always keep rendering in the background. Therefore, the method of the embodiment of the present disclosure ensures the efficiency and reliability of the rendering process, reduces system overhead, speeds up the response speed of the rendering engine, and is applicable to a variety of application scenarios. For example, it is particularly advantageous in complex scenarios where the Unity engine is used to provide rendering services for large-scale games or real-time simulation systems.

[0041] The cross-process rendering method provided by the embodiments of the present disclosure can be applied to various types of handheld devices (such as mobile phones, personal digital assistants (PDAs), etc.), various types of computers (such as tablets, laptops, ultra-mobile personal computers (UMPCs), netbooks, laptops, etc.), wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, and other terminal devices. It can also be applied to other systems that require rendering processing. The embodiments of the present disclosure do not impose any restrictions on the specific type of application devices such as terminal devices, as long as the application device has rendering requirements.

[0042] As an example and not a limitation, when the terminal device is a wearable device, the wearable device can also be a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as gloves, watches, AR head-mounted display devices, VR head-mounted display devices or MR (Mixed Reality) head-mounted display devices equipped with far-field communication modules and / or near-field communication modules.

[0043] In some embodiments, the terminal device may be a device having Figure 1 The exemplary hardware structure of the exemplary mobile phone 100 is shown as follows: Figure 1As shown, the mobile phone 100 may specifically include components such as a radio frequency (RF) circuit 110, a memory 120, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a short-range wireless communication module 170, a processor 180, and a power supply 190. Those skilled in the art will understand that Figure 1 The structure of the mobile phone 100 shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0044] The following combination Figure 1 The following describes the various components of the mobile phone 100:

[0045] The RF circuit 110 can be used to receive and send signals during information transmission or calls. In particular, after receiving downlink information from the base station, it is sent to the processor 180 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 110 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, and the wireless communication can include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), New Radio (NR), GNSS, FM, low-orbit satellite connection and / or IR technology, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS) and / or the Satellite Based Augmentation System (SBAS), etc.

[0046] The memory 120 can be used to store software programs and modules, and the processor 180 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 120. The memory 120 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as pictures, audio data, a phone book, etc.), etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Specifically, the memory 120 may store a program for implementing the cross-process rendering method according to the present disclosure as will be described in detail later, which may be an independent application, or a functional module embedded in one or more applications or an operating system.

[0047] The input unit 130 can be used to receive input digital or character information and generate key signal input related to the user settings and function control of the mobile phone 100. Specifically, the input unit 130 may include a touch panel 131 and other input devices 132. The touch panel 131, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using a finger (such as a single finger or multiple fingers), a stylus, or any other suitable object or accessory on or near the touch panel 131) and drive the corresponding connected device according to a pre-set program. Optionally, the touch panel 131 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch direction and detects the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 180. It can also receive commands sent by the processor 180 and execute them. In addition, the touch panel 131 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 131, the input unit 130 may further include other input devices 132. Specifically, the other input devices 132 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick.

[0048] The display unit 140 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. For example, the display unit 140 is configured to display the user interface of each application, and part or all of its display content can be rendered by the cross-process rendering method of the present disclosure. The display unit 140 may include a display panel 141. Optionally, the display panel 141 may be configured in the form of a liquid crystal display (LCD), a light emitting diode (LED), an organic light emitting diode (OLED), an active-matrix organic light emitting diode (AMOLED), etc. Further, the touch panel 131 may cover the display panel 141. When the touch panel 131 detects a touch operation on or near it, it is transmitted to the processor 180 to determine the type of touch event. The processor 180 then provides corresponding visual output on the display panel 141 according to the type of touch event. Although in Figure 1 In the embodiment, the touch panel 131 and the display panel 141 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 131 and the display panel 141 can be integrated to realize the input and output functions of the mobile phone.

[0049] The mobile phone 100 may also include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 141 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 141 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping, etc.); as for other sensors that the mobile phone can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.

[0050] Audio circuit 160, speaker 161, and microphone 162 provide an audio interface between the user and the phone. Audio circuit 160 converts received audio data into electrical signals and transmits them to speaker 161, which then converts them into sound signals for output. Microphone 162, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 160 and converted into audio data. The audio data is then processed by processor 180 and transmitted via RF circuit 110 to, for example, another electronic device, or to memory 120 for further processing.

[0051] Communication technologies such as Wi-Fi, Bluetooth, Near Field Communication (NFC), and Ultra Wide Band (UWB) are short-range wireless transmission technologies. The mobile phone can help users send and receive emails, browse web pages, and access streaming media through the short-range wireless module 170, which provides users with wireless broadband Internet access. The above-mentioned short-range wireless module 170 may include a Wi-Fi chip, a Bluetooth chip, an NFC chip, and a UWB chip. The Wi-Fi chip can realize the function of Wi-Fi Direct connection between the mobile phone 100 and other electronic devices, and can also enable the mobile phone 100 to operate in an AP mode (Access Point mode) that can provide wireless access services and allow other wireless devices to access, or in a STA mode (Station mode) that can connect to an AP but does not accept wireless devices to access, thereby establishing point-to-point communication between the mobile phone 100 and other Wi-Fi devices.

[0052] The processor 180 is the control center of the mobile phone. It uses various interfaces and lines to connect all parts of the mobile phone. By running or executing software programs and / or modules stored in the memory 120 and calling data stored in the memory 120, it performs various functions of the mobile phone and processes data, thereby monitoring the entire mobile phone. Optionally, the processor 180 may include one or more processing units. Optionally, the processor 180 may include, for example, an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Among them, different processing units can be independent devices or integrated into one or more processors.

[0053] The mobile phone 100 also includes a power supply 190 (such as a battery) for supplying power to various components. For example, the power supply can be logically connected to the processor 180 through a power management system, thereby managing functions such as charging, discharging, and power consumption through the power management system.

[0054] The following combination Figure 2 The overall solution according to at least one embodiment of the present disclosure is discussed in detail with reference to a flowchart of FIG.

[0055] Figure 2 A schematic flowchart of a cross-process rendering method according to at least one embodiment of the present disclosure is exemplarily shown.

[0056] The cross-process rendering method of the embodiment of the present disclosure is run on the same computing device (such as a mobile phone or a car-mounted device, etc.), wherein the service process can be created by a service on the computing device, and this service process can call the rendering engine recorded by the computing device to provide rendering services for one or more first processes (also referred to as client processes) installed on the computing device. The various client processes (such as the first process, the second process, or the third process, etc.) mentioned in the cross-process rendering method of the embodiment of the present disclosure can be processes corresponding to the native application of the computing device and the downloaded and installed third-party application, or processes corresponding to the native service of the computing device, such as the wallpaper service (WallpaperService), wherein the client process with rendering requirements can interact with the service process to allow the service process to perform rendering processing for itself. The cross-process rendering method of the present disclosure can reasonably divide the rendering tasks and bring performance optimization.

[0057] Figure 2 The figure shows the basic operations of interaction between a client process and a service process on a computing device. The "first" added before the process is only used to distinguish among multiple client processes, and is not used to specifically refer to a certain type of process. The first process can be the process corresponding to any application or service installed on the computing device that requires rendering processing.

[0058] like Figure 2 As shown, in step S210, the first process sends a rendering request of the first rendering unit to the service process. Correspondingly, in step S220, the service process receives the rendering request of the first rendering unit sent by the first process.

[0059] The rendering unit in the embodiment of the present disclosure has a separate surface (Surface), and the rendering unit may be, for example, a surface view (SurfaceView) or a window (Window).

[0060] In some embodiments, the first rendering unit can be a surface view to be rendered by the first process. For example, the first process can create the first rendering unit by: creating a subclass of FrameLayout, adding the subclass to the activity layout to be rendered as a child node of the content view, creating a surface view within the subclass as the first rendering unit, and calling the addView method to add the created surface view to the subview of the subclass.

[0061] In some embodiments, when the first process does not have a view tree to load the surface view, for example, when the first process is a process corresponding to a wallpaper service, the first rendering unit may be a window to be rendered by the first process.

[0062] Any cross-process communication method may be used to implement communication between the first process and the service process. For example, an AIDL (Android Interface Definition Language) interface may be used to implement cross-process communication.

[0063] Then, in step S230 , the service process creates a first surface view (SurfaceView) and passes surface control information (SurfaceControl) of the first surface view to the first process. Accordingly, in step S240 , the first process receives the surface control information of the first surface view from the service process.

[0064] Although the embodiment herein shows that step S230 is executed after step S220, it should be understood that in some examples step S230 may also be executed before step S220. For example, the service process may create a SurfaceView in advance and immediately pass its SurfaceControl after receiving the rendering request from the client process, thereby improving the response speed of the service. Alternatively, in some cases, a SurfaceView may be used by multiple client processes or by a client process multiple times. After the previous client process finishes rendering, the SurfaceView is idle, and the service process may pass its SurfaceControl to the current client process for use.

[0065] It can be understood that the computing device can run a client process alone, or run multiple client processes at the same time (such as the first process, the second process, and the third process described in this article). The service process can create multiple SurfaceViews and pass the surface control information of each SurfaceView to different client processes. This allows the computing device to use the rendering capabilities provided by a service process to achieve, for example, split-screen display, and each split screen can correspond to the display of a client process.

[0066] In some implementations, the rendering engine associated with the service process can support rendering to multiple displays and provide rendering services to multiple client processes simultaneously. The service process can create multiple SurfaceViews, where each SurfaceView's Surface is bound to a Display of the rendering engine, so that the rendering engine can render the content of each Display onto its bound Surface.

[0067] This multi-screen rendering capability allows you to distribute rendering tasks across multiple processes, fully utilizing the computing power of multi-core processors. This allows different processes to execute rendering tasks in parallel, significantly improving overall rendering efficiency. Furthermore, different processes can independently manage and execute different rendering tasks, such as main rendering, post-processing, and UI rendering, reducing the burden on a single process and avoiding performance bottlenecks.

[0068] In some examples, the correspondence between multiple rendering units of multiple processes and multiple screens of the rendering engine can be preset in the service process. This means that each rendering unit of each client process is pre-set to which screen to use for rendering. Depending on needs, a process can have one or more rendering units that require rendering from the service process, with each rendering unit pre-assigned a corresponding screen.

[0069] Accordingly, after the service process receives the rendering request of the first rendering unit from the first process in step S220, it can determine the first screen of the rendering engine that the first rendering unit is to use based on the corresponding relationship, and then determine whether the first screen has been enabled, that is, whether a surface with a surface view has been bound.

[0070] If the service process determines that the first screen has not been activated, it creates a first surface view, binds the first surface of the first surface view to the first screen, and passes surface control information of the first surface view to the first process. If the service process determines that the first screen has been activated, it passes the surface control information of the first surface view that has been created and bound to the first screen to the first process.

[0071] In some embodiments, the first surface of the first surface view can be bound to the first screen by calling a rendering engine-related method. For example, if the rendering engine is a Unity engine, a Unity engine-related method can be called to implement the binding. For example, the service process can obtain information about the first surface of the first surface view in the create surface (surfaceCreated) callback of the first surface view, and call the update screen internal (such as updateDisplayInternal) method of the rendering engine player (such as UnityPlayer) to use the obtained first surface information to bind the first surface to the first screen of the rendering engine (such as the Unity engine).

[0072] In some embodiments, the service process can use a class in the Android system, namely, a surface control information view holder (SurfaceControlViewHost), to create the first surface view. For example, the service process creates a first SurfaceControlViewHost and adds a new surface view as the first surface view in the view tree (ViewTree) of the first SurfaceControlViewHost. As a result, the service process can pass the surface control information of the first surface view to the first process by returning the surface package (SurfacePackage) of the first SurfaceControlViewHost to the first process, where the SurfacePackage contains the surface control information of the first surface view.

[0073] Then, in step S250, the first process uses the surface control information of the first surface view to associate the first surface of the first surface view to the second surface created by the first rendering unit. After the association is achieved, in step S260, the first process sends the rendering data of the first rendering unit to the rendering engine associated with the service process, and accordingly, in step S270, the service process receives the rendering data of the first rendering unit from the first process, and based on the rendering data, uses the rendering engine associated with the service process to render the content of the first rendering unit on the first surface of the first surface view so that the second surface associated with the first surface displays the rendering result. As mentioned above, each rendering unit (including surface view, window, etc.) has only a single surface, so the ordinal numbers "first", "second", etc. before the "surface" in this disclosure are only used to distinguish the surfaces of each rendering unit, rather than indicating that each rendering unit has multiple surfaces.

[0074] Since the surface of the first surface view has been associated with the surface of the first rendering unit, the first process can display the rendered content on the surface of the first surface view as the content of the first rendering unit.

[0075] This allows time-consuming rendering tasks to be assigned to independent processes, reducing the main process's blocking time, lowering rendering latency, and improving frame rate stability. Furthermore, through process isolation, even if a process experiences an exception or crashes, it will not affect the normal operation of other processes, thereby improving the overall stability of the application. Independent rendering processes can quickly restart and recover in the background after errors, ensuring that the system can continue to operate in extreme situations and reducing the negative impact of crashes on the user experience.

[0076] In some embodiments, the first process may reparent the surface control information of the first surface view to the surface control information of the first rendering unit, thereby setting the first surface as a child node of the second surface, thereby associating the surface of the first surface view to the surface of the first rendering unit.

[0077] In some embodiments, when the first rendering unit is a third surface view to be rendered by the first process, the aforementioned second surface is the surface of the third surface view. In addition, as described above, the first process can obtain the surface control information of the first surface view by receiving the SurfacePackage of the first SurfaceControlViewHost from the service process, where the first surface view is a surface view in the View Tree of the first SurfaceControlViewHost. Then, the first process can call the setChildSurfacePackage method for the third surface view and the received SurfacePackage, thereby setting the first surface as a child node of the second surface, thereby associating the first surface of the first surface view with the second surface of the first rendering unit.

[0078] In some embodiments, when the first rendering unit is a window to be rendered by the first process, the aforementioned second surface is the surface of the window. In addition, as previously described, the first process can obtain the surface control information of the first surface view by receiving the SurfacePackage of the first SurfaceControlViewHost from the service process, wherein the first surface view is a surface view in the View Tree of the first SurfaceControlViewHost. Then, the first process can obtain the surface control information of the window and the surface control information in the received SurfacePackage, and use the method in the surface control transaction (SurfaceControl.Transaction) to reparent the surface control information in the SurfacePackage to the surface control information of the window, thereby setting the first surface as a child node of the surface of the window, thereby associating the first surface of the first surface view with the second surface of the first rendering unit.

[0079] In the case where the first rendering unit is a window to be rendered by the first process, the first process can be the process corresponding to WallpaperService, and the surface control information of the window and the surface control information in the SurfacePackage can be obtained by the following operations: the engine class (WallpaperService.Engine) that inherits WallpaperService creates an instance of the engine (Engine), and in the surface creation (onSurfaceCreated) callback of the engine instance, the surface control member variable (mBbqSurfaceControl or mSurfaceControl) is obtained by reflection as the surface control information of the window, and the method of obtaining the surface control information (getSurfaceControl) is reflected to obtain the surface control information in the SurfacePackage.

[0080] After the association is achieved, the service process can receive the rendering data of the first rendering unit from the first process through, for example, an AIDL interface, and then communicate with the Unity engine as the rendering engine through, for example, the UnityPlayer.UnitySendMessage method, pass the rendering data to the rendering engine, and let the rendering engine perform corresponding rendering processing.

[0081] Therefore, the method of the embodiment of the present disclosure realizes cross-process rendering by transferring surface control information across processes. It does not need to transfer the surface across processes. Instead, the rendering engine renders on the surface of the service process, ensuring the efficiency and reliability of the rendering process.

[0082] In addition, in some embodiments, rendering efficiency can be improved through multi-core parallel processing, for example, by running multiple service processes on multiple cores to achieve parallel processing, thereby being able to take advantage of multi-core processors. As a result, cross-process rendering can handle more complex and demanding rendering scenarios, such as massively multiplayer online games, large open world games, and high-resolution real-time simulation applications, providing a smooth user experience. In addition, for applications that require real-time rendering and interaction (such as AR / VR applications), cross-process rendering can provide higher response speeds and lower latency, enhancing user immersion.

[0083] In addition, in some embodiments, the method of the present disclosure may further include:

[0084] In response to detecting that the first process terminates abnormally, the service process terminates the association between the first surface and the second surface;

[0085] In response to detecting that the first process is restarted, the service process re-delivers the surface control information of the first surface view to the first process, so that the first process uses the surface control information of the first surface view to associate the first surface with the second surface recreated by the first rendering unit.

[0086] This solves the problem in the prior art of running a rendering engine in an application process and requiring the rendering engine to be restarted when the application process crashes.

[0087] By using the method disclosed herein, a fault detection and recovery mechanism can be established by monitoring the status of each rendering process. Once a process crash is detected, the process can be quickly restarted and its rendering state restored to reduce the impact on the overall rendering.

[0088] Next, combine Figure 3 This section describes an example in which a service process provides rendering services to multiple client processes.

[0089] Figure 3 The schematic timing diagram of the cross-process rendering method executed between a service process and multiple client processes according to at least one embodiment of the present disclosure is exemplarily shown. It should be understood that Figure 3 The steps or operations in the figure are merely exemplary and non-restrictive, and are not intended to limit the scope of the present disclosure. For example, the number of screens supported by the rendering engine, the number of client processes supported by the service process, and the number of rendering units in each client process shown in the figure are all exemplary. Figure 3 The details or alternative methods of each step or operation can be referred to in conjunction with the above Figures 1 to 2 The content described will not be repeated here. Figure 3The various details in can also be combined with the previous Figures 1 to 2 The various embodiments or examples described are used in combination.

[0090] like Figure 3 As shown, the rendering engine associated with the service process supports rendering of multiple screens, and the correspondence between each rendering unit of each client process and the screen of the rendering engine is preset in the service process. For example, the first rendering unit of the first process, the second rendering unit of the second process, and the third rendering unit of the third process correspond to the first screen, the second screen, and the first screen, respectively.

[0091] Although Figure 3 The timing diagram shows that the rendering process of the second rendering unit of the second process is performed after the rendering of the first rendering unit of the first process is completed. However, because the first rendering unit and the second rendering unit correspond to different screens of the rendering engine, the rendering engine can perform rendering processing on these two rendering units simultaneously, and the service process can accordingly handle rendering requests from these two rendering units simultaneously. In other words, steps S321-S326 can be executed independently of steps S311-S316. The execution time of steps S321-S326 is not limited by the execution time of steps S311-S316 and can be executed before, in parallel, or partially in parallel.

[0092] In addition, although Figure 3 The timing diagram shows that the rendering process of the third rendering unit of the third process is performed after the rendering of the second rendering unit of the second process is completed. However, since the third rendering unit and the second rendering unit also correspond to different screens of the rendering engine, steps S331-S336 and steps S321-S326 can be executed independently. The execution time of steps S331-S336 is not limited by the execution time of steps S321-S326 and can be executed before, in parallel, or partially in parallel. However, since the third rendering unit of the third process and the first rendering unit of the first process use the same screen, namely the first screen, the rendering operation of the third rendering unit, namely steps S331-S336, must be performed after the rendering of the first rendering unit is completed. By better designing the correspondence between each rendering unit of each client process and the screen of the rendering engine, conflicts in use can be avoided, without the service process having to determine whether the screen to be used by the current rendering request is idle.

[0093] like Figure 3 As shown, steps S311-S316 are a process in which the service process provides rendering services for the first process.

[0094] In step S311 , the first process sends a rendering request of a first rendering unit to the service process.

[0095] Then, in step S312, the service process determines that the first rendering unit is to use the first screen of the rendering engine based on the aforementioned preset correspondence between the rendering unit and the screen, and determines that the first screen has not yet been enabled, thereby creating a first surface view and binding the first surface of the first surface view to the first screen.

[0096] Next, in step S313 , the service process transmits the surface control information of the first surface view to the first process.

[0097] Next, in step S314 , the first process associates the first surface of the first surface view with the second surface of the first rendering unit using the surface control information of the first surface view.

[0098] Next, in step S315, the first process sends the rendering data of the first rendering unit to the service process. In step S316, upon receipt, the service process transmits the rendering data to its associated rendering engine and instructs the rendering engine to render the first screen based on the rendering data. The rendering engine associated with the service process then renders the content of the first rendering unit based on the received rendering data of the first rendering unit and renders the content as the content of the first screen onto the first surface of the first surface view bound to the first screen.

[0099] It should be understood that after step S316, if the rendering data of the first rendering unit changes, that is, the first process wants to change the rendering content of the first rendering unit, then the above steps S315 and S316 can be repeated. After the first rendering unit completes its rendering, the first process can destroy the first rendering unit without performing any operations on the first surface view. At this time, the association between the first rendering unit and the first surface view is naturally released, and the first surface view remains in the background service process for subsequent use by other rendering units.

[0100] Steps S321 - S326 are a process in which the service process provides rendering services for the second process.

[0101] In step S321 , the second process sends a rendering request of a second rendering unit to the service process.

[0102] Then, in step S322, the service process determines that the second rendering unit is to use the second screen of the rendering engine based on the aforementioned preset correspondence between the rendering unit and the screen, and determines that the second screen has not yet been enabled, thereby creating a second surface view and binding the third surface of the second surface view to the second screen.

[0103] Next, in step S323 , the service process passes the surface control information of the second surface view to the second process.

[0104] Next, in step S324 , the second process associates the third surface of the second surface view with the fourth surface of the second rendering unit using the surface control information of the second surface view.

[0105] Next, in step S325, the second process sends the rendering data of the second rendering unit to the service process. In step S326, upon receipt, the service process transmits the rendering data to its associated rendering engine and instructs the rendering engine to render the second screen based on the rendering data. The rendering engine then renders the content of the second rendering unit based on the received rendering data, and renders the content as the content of the second screen onto the third surface of the second surface view bound to the second screen.

[0106] It should also be understood that after step S326, if the rendering data of the second rendering unit changes, that is, the second process wants to change the rendering content of the second rendering unit, then the above steps S325 and S326 can be repeated. After the second rendering unit completes its rendering, the second process can destroy the second rendering unit without performing any operations on the second surface view. At this time, the association between the second rendering unit and the second surface view is naturally released, and the second surface view remains in the background service process for subsequent use by other rendering units.

[0107] Steps S331 - S336 are a process in which the service process provides rendering services to the third process.

[0108] In step S331 , the third process sends a rendering request of the third rendering unit to the service process. At this time, the rendering of the first rendering unit has been completed, that is, the first screen and its bound first surface are both in an idle state.

[0109] Then, in step S332, the service process determines that the third rendering unit is to use the first screen of the rendering engine based on the aforementioned preset correspondence between the rendering unit and the screen, and determines that the first screen has been enabled. Therefore, in step S333, the service process directly passes the surface control information of the first surface view previously created to the third process.

[0110] Next, in step S334 , the third process associates the first surface of the first surface view with the fifth surface of the third rendering unit using the surface control information of the first surface view.

[0111] Next, in step S335, the third process sends rendering data of the third rendering unit to the service process. In step S336, upon receipt, the service process transmits the rendering data to its associated rendering engine, instructing the rendering engine to render the first screen based on the rendering data. The rendering engine then renders the content of the third rendering unit based on the received rendering data, and renders the content as the content of the first screen onto the first surface of the first surface view bound to the first screen.

[0112] It should also be understood that after step S336, if the rendering data of the third rendering unit changes, that is, the third process wants to change the rendering content of the third rendering unit, then the above steps S335 and S336 can be repeated. After the third rendering unit completes its rendering, the third process can destroy the third rendering unit without performing any operation on the first surface view. At this time, the association between the third rendering unit and the first surface view is naturally released, and the first surface view remains in the background service process for subsequent use by other rendering units.

[0113] Thus, the method of the disclosed embodiments can leverage the multi-screen mechanism of the rendering engine associated with the service process to provide rendering services to multiple client processes simultaneously, improving rendering efficiency. Furthermore, the surface bound to each screen can remain in the background service process, eliminating the need to destroy and recreate it in response to the destruction and reconstruction of the client process's rendering unit's surface. This reduces system overhead and speeds up the rendering engine's response time.

[0114] Furthermore, the cross-process rendering solution allows the rendering logic to be divided into multiple independent modules, each running in a different process. This modular design makes the system easier to expand and maintain, facilitating the introduction of new features and customized development. Because different functional modules run in independent processes, individual processes or modules can be updated or optimized independently without impacting the overall system operation. This improves the system's flexibility and maintainability.

[0115] Because the cross-process rendering solution enables independent updates and maintenance of features through a modular design, it is more easily compatible with different versions of Unity and Android systems, reducing issues caused by version upgrades. Furthermore, the cross-process rendering solution can be adjusted for different hardware configurations and system environments, ensuring optimized rendering effects on a variety of devices and operating systems.

[0116] Furthermore, cross-process resource sharing and management can avoid resource conflicts between different modules, while improving the security and stability of resource usage. Furthermore, resource allocation to each process can be dynamically adjusted based on demand, allowing the system to flexibly adapt to different scenarios and optimize resource utilization.

[0117] The following combination Figures 4 to 6 An example of implementing the cross-process rendering method on an Android system according to an embodiment of the present disclosure is described in more detail.

[0118] Figure 4 The present invention exemplarily shows the architecture of a cross-process rendering method according to at least one embodiment of the present disclosure, as well as a schematic diagram of some organizational structures and data flows generated when executing the method. Figure 5 A schematic diagram exemplarily shows the organizational structure of a surface view (SurfaceView) created in a service process and its relationship with a screen in a cross-process rendering method according to at least one embodiment of the present disclosure. Figure 6 A schematic diagram exemplarily shows the organizational structure of a SurfaceView created in a client process in a cross-process rendering method according to at least one embodiment of the present disclosure.

[0119] like Figure 4 As shown, the solution of the embodiment of the present disclosure can be divided into two parts: server and client. The method of the embodiment of the present disclosure can be implemented by a server application and multiple client applications installed on the Android system, wherein the server application is used to accept requests from client applications and provide rendering images, and the client applications send rendering data to the server application for rendering.

[0120] The server application consists of two parts: the Android functional part and the Unity engine as a rendering engine. The Android functional part serves as a container for the Unity engine to run, providing the Unity engine with the ability to access the Android environment, while the Unity engine provides rendering capabilities and has a multi-screen mechanism.

[0121] An SDK (Software Development Kit) can be encapsulated in each client application that has a need, and used to connect to each application to achieve cross-process rendering. The corresponding process of each client application can use its own encapsulated SDK. For example, Figure 4 The first process, the second process and the third process in the program respectively use SDK1, SDK2 and SDK3 encapsulated in their corresponding applications.

[0122] After launching the server application, a service process is created to prepare for executing the cross-process rendering method of the disclosed embodiment. This service process corresponds to the aforementioned Android functionality. After launching the client application, a corresponding client process is created. When cross-process rendering is required, the client process can access the SDK encapsulated therein to perform the corresponding cross-process rendering operation.

[0123] First, the operation of the service process in the cross-process rendering method of the embodiment of the present disclosure is discussed.

[0124] like Figure 4 As shown, the service process can create an AIDL interface and a corresponding service for IPC (Inter-Process Communication) between the service process and the client process, such as the service process passing SurfaceControlViewHost.SurfacePackage to the client process, and the client process sending data and instructions to the service process.

[0125] In addition, in some cases, the service process can preload a scene at startup to trigger the initialization of the Unity engine, thereby saving the time of initializing the Unity engine when the client process subsequently requests rendering, thereby speeding up the rendering speed.

[0126] In addition, as mentioned above, the service process can use the multi-screen mechanism of the Unity engine to render for multiple client processes simultaneously. Figure 4 As shown, the service process corresponding to the Android functional part may include a UnityDisplay management unit, which may preset the correspondence between each rendering unit of each client process and each screen (Display) of the Unity engine to ensure that there is no conflict.

[0127] In addition, when a Display is to be enabled, a SurfaceView needs to be created on the Android side to bind it to the Display for rendering by the Unity engine. To this end, the UnityDisplay Management Department can create a UnityDisplay class to manage the correspondence between the SurfaceView created by the Android functional part and the Display. In other words, when a client process requests to render a screen, the UnityDisplay Management Department finds that the corresponding Display has not been enabled, so it creates a UnityDisplay class and a SurfaceView, binding its Surface to the corresponding Display.

[0128] Figure 5The organizational structure of the SurfaceView on the Android side in the service process and its relationship with the Display of the Unity engine are exemplarily shown.

[0129] like Figure 5 As shown, the Android functional part of the service process can use Android's SurfaceControlViewHost class to create a background rendering View Tree. This View Tree fully supports Android's View control, so a SurfaceView can be created and loaded into the View Tree.

[0130] The service process can obtain the Surface object in the surfaceCreated callback of the SurfaceView, and then call the updateDisplayInternal method of UnityPlayer to bind the Surface to the corresponding Display. The multi-screen mechanism of the Unity engine will render the content on this Display to the bound Surface.

[0131] Back to Figure 4 The figure shows that the Unity engine supports two screens, namely Display0 and Display1. Correspondingly, two SurfaceControlViewHosts, namely SurfaceControlViewHost0 and SurfaceControlViewHost1, are created in the service process corresponding to the Android function part. The newly created SurfaceView0 and SurfaceView1 are loaded in their View Trees respectively, and the Surfaces of SurfaceView0 and SurfaceView1 are bound to Display0 and Display1 respectively for the Unity engine to render on the Surface.

[0132] The service process corresponding to the above-mentioned Android functional part can communicate with the Unity engine through the UnityPlayer.UnitySendMessage method, such as switching scenes / Displays and transmitting data.

[0133] In addition, the Unity engine can also communicate with the service process corresponding to the Android functional part by accessing the static methods provided by the service process corresponding to the Android functional part, thereby obtaining some information about the service process corresponding to the Android functional part and executing the methods provided by the service process corresponding to the Android functional part. For example, the Unity engine can set up the linkage between the time engine and the atmosphere light through the service process corresponding to the Android functional part.

[0134] Next, the operation of the client process created by each application in the cross-process rendering method of the embodiment of the present disclosure is discussed.

[0135] like Figure 4 As shown, each application process must access the corresponding SDK encapsulated therein when performing cross-process rendering. The SDK can implement the following operations to achieve cross-process rendering.

[0136] The SDK in each application can obtain the AIDL interface created by the service process, call the initialization method after the first process is connected, bind the server's service, and thus communicate with the service process.

[0137] In addition, for the client process (such as the first process and the second process) that has the View Tree capable of loading SurfaceView, the SDKs therein, such as SDK1 and SDK2, can be as follows Figure 6 Create a SurfaceView as shown for cross-process rendering.

[0138] Figure 6 A schematic diagram exemplarily shows the organizational structure of a SurfaceView created in a client process in a cross-process rendering method according to at least one embodiment of the present disclosure.

[0139] like Figure 6 As shown, the SDK creates a subclass of FrameLayout, names it RenderView, and adds it to the rendered Activity layout as a child of ContentView. After creating RenderView, create a SurfaceView within it and call the addView method to add it to the RenderView's child View. RenderView wraps the SurfaceView, providing a convenient interface.

[0140] Back to Figure 4Taking the first process as an example, SDK1 creates RenderView1, which wraps SurfaceView2. RenderView1 can also provide a changeScene method for requesting Unity's screen. After the first process calls the changeScene method, the corresponding rendering request can be sent to the service process through the AIDL interface. Then, the service process can determine to use Display0 as described above, and Display0 is not currently enabled, so it creates SurfaceControlViewHost0 and the SurfaceView0 therein, and binds the Surface of SurfaceView0 to Display0. The service process can then send instructions to the Unity engine to switch to Display0, and return the SurfaceControlViewHost.SurfacePackage corresponding to SurfaceControlViewHost0 to the first process.

[0141] After SDK1 obtains the SurfaceControlViewHost.SurfacePackage returned by the service process, it calls the setChildSurfacePackage method on SurfaceView2 and reparents the received SurfacePackage to the SurfaceControl of SurfaceView2, such as Figure 4 As shown by the dotted line with "reParent" in the SurfacePackage, the mounting operation of SurfaceControl is realized. This operation sets the Surface of SurfaceView in the SurfacePackage (that is, the Surface of SurfaceView0) as the child node of the Surface of SurfaceView2, so that the image of the Surface rendered by the service process (that is, the Surface of SurfaceView0) can be displayed in the window of the first process.

[0142] In addition, for a client process that does not have a View Tree and cannot load SurfaceView, such as a third process (such as the process corresponding to WallpaperService), the SDK therein, such as SDK3, can perform the mounting operation of SurfaceControl as follows.

[0143] Taking the process corresponding to WallpaperService as an example, first, you can develop WallpaperService, inherit WallpaperService.Engine to create an instance of Engine, and then obtain the Engine's mBbqSurfaceControl member variable or mSurfaceControl member variable through reflection in the onSurfaceCreated callback of the Engine instance. This member variable is the SurfaceControl used by WallpaperService to manage the Surface of its window.

[0144] In addition, similar to the first process mentioned above, SDK3 can send a rendering request to the service process through the AIDL interface, and obtain the SurfaceControlViewHost.SurfacePackage corresponding to the SurfaceControlViewHost0 bound to the corresponding Display0. Then, the getSurfaceControl method is reflected to obtain the SurfaceControl in the SurfacePackage (this SurfaceControl is used to control the Surface of SurfaceView0 rendered by the service process), and then the reparent method in SurfaceControl.Transaction is executed, and the show and apply methods in Transaction are executed to reparent the SurfaceControl in the SurfacePackage to the SurfaceControl of the WallpaperService window, such as Figure 4 As shown by the dotted line with "reParent" in . Thus, the mounting operation of SurfaceControl is realized, which sets the Surface of SurfaceView (i.e., Surface of SurfaceView0) in the SurfacePackage received from the service process as the child node of the Surface of the window of the third process, so that the image of the Surface rendered by the service process (i.e., Surface of SurfaceView0) can be displayed in the window of the third process.

[0145] As mentioned above, regardless of which client process, the SurfaceControl mounting operation is performed by the client process, and whoever uses it performs the operation, while the service process only focuses on providing rendering. The service process can continue to render content to the corresponding Surface in the background without worrying about changes in the rendering unit of the client process in the foreground.

[0146] After the SurfaceControl mounting operation is completed, each SDK can send instructions and data to the Unity engine associated with the service process through the AIDL interface.

[0147] Figure 7 A schematic structural diagram of a computing device that can be used to implement the above-mentioned cross-process rendering method according to at least one embodiment of the present disclosure is shown.

[0148] See also Figure 7 , the computing device 700 includes a memory 710 and a processor 720 .

[0149] The processor 720 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 720 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU) or a digital signal processor (DSP). In some embodiments, the processor 720 may be implemented using customized circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0150] The memory 710 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 720 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that retains stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory 710 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 710 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0151] The memory 710 stores executable code. When the executable code is processed by the processor 720 , the processor 720 can execute the cross-process rendering method mentioned above.

[0152] In addition, the method according to the present disclosure may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present disclosure.

[0153] Alternatively, the present disclosure may also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor is caused to perform the various steps of the above-mentioned method according to the present disclosure.

[0154] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.

[0155] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems and methods according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0156] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A cross-process rendering method, comprising: The first process sends a rendering request of the first rendering unit to the service process; The service process receives a rendering request of the first rendering unit sent by the first process; The service process creates a first surface view and passes surface control information of the first surface view to the first process; The first process receives surface control information of the first surface view from the service process; The first process associates the first surface of the first surface view with the second surface created by the first rendering unit using the surface control information of the first surface view; The first process sends rendering data of the first rendering unit to a rendering engine associated with the service process; as well as The service process renders the content of the first rendering unit on the first surface of the first surface view using the rendering engine associated with the service process according to the rendering data of the first rendering unit received from the first process, so that the second surface associated with the first surface displays the rendering result.

2. The method according to claim 1, further comprising: The second process sends a rendering request of the second rendering unit to the service process; The service process receives a rendering request from a second rendering unit sent by a second process; The service process creates a second surface view and passes surface control information of the second surface view to the second process; The second process receives the surface control information of the second surface view from the service process; The second process associates the third surface of the second surface view with the fourth surface created by the second rendering unit using the surface control information of the second surface view; The second process sends rendering data of the second rendering unit to the rendering engine associated with the service process; as well as The service process renders the content of the second rendering unit on the third surface of the second surface view using the rendering engine according to the rendering data of the second rendering unit received from the second process, so that the fourth surface associated with the third surface displays the rendering result. The first surface and the third surface are respectively bound to the first screen and the second screen of the rendering engine so that the contents of the first screen and the second screen are rendered onto the first surface and the third surface respectively, wherein the contents of the first screen and the second screen are the contents of the first rendering unit and the second rendering unit rendered by the rendering engine respectively.

3. The method according to claim 1, further comprising: After the rendering of the first rendering unit is completed, the third process sends a rendering request of the third rendering unit to the service process; The service process receives a rendering request from a third rendering unit sent by a third process; The service process transmits the surface control information of the first surface view to the third process; The third process receives the surface control information of the first surface view from the service process; A third process associates the first surface of the first surface view with the fifth surface created by the third rendering unit using the surface control information of the first surface view; The third process sends the rendering data of the third rendering unit to the rendering engine associated with the service process; as well as The service process uses the rendering engine to render the content of the third rendering unit on the first surface according to the rendering data of the third rendering unit received from the third process, so that the fifth surface associated with the first surface displays the rendering result.

4. The method according to claim 1, wherein The rendering engine supports rendering of multiple screens, and a correspondence between multiple rendering units of multiple processes and multiple screens of the rendering engine is preset in the service process. After receiving the rendering request of the first rendering unit from the first process, the service process determines, based on the corresponding relationship, that the first rendering unit is to use the first screen of the rendering engine. When determining that the first screen has not been enabled, the service process creates the first surface view, binds the first surface of the first surface view to the first screen, and passes surface control information of the first surface view to the first process. When determining that the first screen has been enabled, the service process transmits surface control information of the first surface view that has been created and bound to the first screen to the first process.

5. The method according to claim 1, wherein The service process creates the first surface view including: The service process creates the first surface control information view holder, and The service process adds a new surface view as the first surface view in the view tree of the first surface control information view holder; Furthermore, the service process passes the surface control information of the first surface view to the first process, including: the service process returns the surface package of the first surface control information view holder to the first process, wherein the surface package contains the surface control information of the first surface view.

6. The method according to claim 5, further comprising: The service process obtains information about the first surface of the first surface view in the create surface callback of the first surface view; as well as The service process calls the internal screen update method of the player of the rendering engine, and uses the acquired information of the first surface to bind the first surface to the first screen of the rendering engine.

7. The method according to claim 1, wherein The first process associates the first surface of the first surface view with the second surface created by the first rendering unit by using the surface control information of the first surface view, including: The first process reparents the surface control information of the first surface view to the surface control information of the first rendering unit, thereby setting the first surface as a child node of the second surface.

8. The method according to claim 1, wherein The first rendering unit is a third surface view to be rendered by the first process, and the second surface is a surface of the third surface view; The first process receives the surface control information of the first surface view from the service process, comprising: the first process receives a surface package of a first surface control information view holder from the service process, wherein the first surface view is a surface view in a view tree of the first surface control information view holder, and the surface package includes the surface control information of the first surface view; and The first process uses the surface control information of the first surface view to associate the first surface of the first surface view to the second surface created by the first rendering unit, including: the first process calls the set sub-surface encapsulation method for the third surface view and the surface encapsulation, thereby setting the first surface as a child node of the second surface.

9. The method according to claim 8, wherein The first rendering unit is a third surface view to be rendered by the first process, including: The first process creates a subclass of the frame layout, The first process adds the subclass to the activity layout to be rendered as a child node of the content view. The first process creates a surface view inside the subclass as the third surface view, and calls an add view method to add the third surface view to the subview of the subclass.

10. The method according to claim 1, wherein The first rendering unit is a window to be rendered by the first process, and the second surface is a surface of the window; The first process receives, from the service process, surface control information of a first surface view created by the service process, comprising: the first process receiving, from the service process, a surface package of a first surface control information view holder, wherein the first surface view is a surface view in a view tree of the first surface control information view holder, and the surface package includes the surface control information of the first surface view; and The first process associates the first surface of the first surface view with the second surface created by the first rendering unit by using the surface control information of the first surface view, including: The first process obtains the surface control information of the window and the surface control information in the surface package, and The first process uses a method in a transaction of a surface control to reparent the surface control information in the surface package to the surface control information of the window, thereby setting the first surface as a child node of the surface of the window.

11. The method according to claim 10, wherein: The first process is the process corresponding to the wallpaper service, and The first process obtains the surface control information of the window and the surface control information in the surface package, including: The first process inherits the engine class of the wallpaper service to create an instance of the engine. The first process obtains the surface control member variable as the surface control information of the window through reflection in the surface creation callback of the engine instance, and The first process reflects the method for obtaining surface control information, and obtains the surface control information in the surface package.

12. The method according to claim 1, further comprising: In response to detecting that the first process terminates abnormally, the service process terminates the association between the first surface and the second surface; In response to detecting that the first process is restarted, the service process re-delivers the surface control information of the first surface view to the first process, so that the first process uses the surface control information of the first surface view to associate the first surface with the second surface recreated by the first rendering unit.

13. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 12.

14. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 12.

Images