Cross-process rendering method

Abstract

The present disclosure relates to a cross-process rendering method, which comprises: a first process sending a rendering request of a first rendering unit to a service process; the service process receiving the rendering request of the first rendering unit, creating a first surface view and passing surface control information of the first surface view to the first process; the first process receiving the surface control information of the first surface view from the service process, associating a first surface of the first surface view to a second surface created by the first rendering unit by using the surface control information of the first surface view, and sending rendering data of the first rendering unit to a rendering engine associated with the service process; and the service process rendering content of the first rendering unit on the first surface of the first surface view by using the rendering engine associated with the service process according to the rendering data of the first rendering unit, so that the rendering result is displayed on the second surface associated with the first surface. By using the method of the present disclosure, the efficiency and reliability of the cross-process rendering process can be ensured.

Description

Technical Field

[0001] This disclosure relates to cross-process rendering methods, computing devices, storage media, and program products. Background Technology

[0002] Traditional rendering methods are typically limited to completing graphics rendering tasks within a single process, especially on mobile platforms, such as applications using rendering engines in the Android system.

[0003] In traditional rendering architectures, rendering work is typically completed within a single process, such as an in-process rendering thread. The Unity engine performs tasks such as scene drawing, resource management, and logic processing within this thread; all rendering operations occur within the same process.

[0004] Because rendering is performed within a single process, all rendering resources (such as textures, shaders, and models) are stored and managed in that process's memory. This approach simplifies resource management, as the rendering engine accesses and manipulates data within the same memory space. However, anomalies in the aforementioned process, limited memory resources, or memory leaks can all cause rendering tasks to become unsustainable. Summary of the Invention

[0005] This disclosure provides cross-process rendering methods and corresponding computing devices, non-transitory machine-readable storage media, and computer program products for performing these methods.

[0006] According to a first aspect of the present disclosure, a cross-process rendering method is provided, comprising: a first process sending a rendering request for a first rendering unit to a service process; the service process receiving the rendering request for the first rendering unit sent by the first process; the service process creating a first surface view and transmitting surface control information of the first surface view to the first process; the first process receiving the surface control information of the first surface view from the service process; the first process using the surface control information of the first surface view to associate a first surface of the first surface view with a second surface created by the first rendering unit; the first process sending rendering data of the first rendering unit to a rendering engine associated with the service process; and the service process rendering the content of the first rendering unit on a first surface of the first surface view using the rendering engine associated with the service process based on the rendering data of the first rendering unit received from the first process, so that the second surface associated with the first surface displays the rendering result.

[0007] Optionally, the method further includes: a second process sending a rendering request for a second rendering unit to a service process; the service process receiving the rendering request for the second rendering unit sent by the second process; the service process creating a second surface view and transmitting surface control information of the second surface view to the second process; the second process receiving the surface control information of the second surface view from the service process; the second process using the surface control information of the second surface view to associate a third surface of the second surface view with a fourth surface created by the second rendering unit; the second process sending rendering data of the second rendering unit to the rendering engine associated with the service process; and the service process, based on the rendering data of the second rendering unit received from the second process, using the rendering engine to render the content of the second rendering unit on the third surface of the second surface view so that the fourth surface associated with the third surface displays the rendering result, wherein the first surface and the third surface are respectively bound to a first screen and a second screen of the rendering engine so that the content of the first screen and the second screen is rendered onto the first surface and the third surface respectively, wherein the content of the first screen and the second screen are the content of the first rendering unit and the second rendering unit rendered by the rendering engine respectively.

[0008] Optionally, the method further includes: after the rendering of the first rendering unit is completed, the third process sends a rendering request for the third rendering unit to the service process; the service process receives the rendering request for the third rendering unit sent by the third process; the service process transmits the surface control information of the first surface view to the third process; the third process receives the surface control information of the first surface view from the service process; the third process uses the surface control information of the first surface view to associate the first surface of the first surface view with the fifth surface created by the third rendering unit; the third process sends the rendering data of the third rendering unit to the rendering engine associated with the service process; and the service process, based on the rendering data of the third rendering unit received from the third process, uses the rendering engine to render the content of the third rendering unit on the first surface so that the fifth surface associated with the first surface displays the rendering result.

[0009] Optionally, the rendering engine supports rendering multiple screens, and the service process has a pre-defined correspondence between multiple rendering units of multiple processes and multiple screens of the rendering engine. After receiving a rendering request from the first rendering unit from the first process, the service process determines the first screen of the rendering engine to be used by the first rendering unit according to the correspondence. If the service process determines that the first screen has not been enabled, it creates the first surface view, binds the first surface of the first surface view to the first screen, and passes the surface control information of the first surface view to the first process. If the service process determines that the first screen has been enabled, it passes the surface control information of the first surface view that has been created and bound to the first screen to the first process.

[0010] Optionally, the service process creating the first surface view includes: the service process creating a first surface control information view holder, and the service process adding a new surface view as the first surface view in the view tree of the first surface control information view holder; and the service process passing the surface control information of the first surface view to the first process includes: the service process returning the surface encapsulation of the first surface control information view holder to the first process, wherein the surface encapsulation contains the surface control information of the first surface view.

[0011] Optionally, the method further includes: the service process obtaining information about the first surface of the first surface view in the surface creation callback of the first surface view; and the service process calling the update screen internal method of the player of the rendering engine to bind the first surface to the first screen of the rendering engine using the obtained information about the first surface.

[0012] Optionally, the first process uses the surface control information of the first surface view to associate the first surface of the first surface view with the second surface created by the first rendering unit, including: the first process reparents the surface control information of the first surface view to the surface control information of the first rendering unit, thereby setting the first surface as a child node of the second surface.

[0013] Optionally, the first rendering unit is a third surface view to be rendered by the first process, and the second surface is a surface of the third surface view; the first process receives surface control information of the first surface view from the service process, including: the first process receives a surface encapsulation of a first surface control information view holder from the service process, wherein the first surface view is a surface view in the view tree of the first surface control information view holder, and the surface encapsulation contains the surface control information of the first surface view; and the first process uses the surface control information of the first surface view to associate a first surface of the first surface view with a second surface created by the first rendering unit, including: the first process calls a setting sub-surface encapsulation method for the third surface view and the surface encapsulation, thereby setting the first surface as a child node of the second surface.

[0014] Optionally, the first rendering unit is a third surface view to be rendered by the first process, comprising: the first process creating a subclass of a frame layout; the first process adding the subclass to the active layout to be rendered as a child node of the content view; the first process creating a surface view inside the subclass as the third surface view; and the first process calling the add view method to add the third surface view to the subview of the subclass.

[0015] Optionally, the first rendering unit is the window to be rendered by the first process, and the second surface is the surface of the window; the first process receives surface control information of a first surface view created by the service process from the service process, including: the first process receives a surface encapsulation of a first surface control information view holder from the service process, wherein the first surface view is a surface view in the view tree of the first surface control information view holder, and the surface encapsulation contains the surface control information of the first surface view; and the first process uses the surface control information of the first surface view to associate the first surface of the first surface view with the second surface created by the first rendering unit, including: the first process obtains the surface control information of the window and the surface control information in the surface encapsulation, and the first process uses a method in the surface control transaction to reparent the surface control information in the surface encapsulation to the surface control information of the window, thereby setting the first surface as a child node of the surface of the window.

[0016] Optionally, the first process is the process corresponding to the wallpaper service, and the first process obtains the surface control information of the window and the surface control information in the surface encapsulation, including: the first process inherits the engine class of the wallpaper service to create an instance of the engine; the first process obtains the surface control member variable as the surface control information of the window through reflection in the surface creation callback of the engine instance; and the first process obtains the surface control information in the surface encapsulation by reflecting the method of obtaining the surface control information.

[0017] Optionally, the method further includes: in response to detecting that the first process has terminated abnormally, the service process terminates the association between the first surface and the second surface; in response to detecting that the first process has restarted, the service process retransmits the surface control information of the first surface view to the first process, so that the first process uses the surface control information of the first surface view to associate the first surface with the second surface recreated by the first rendering unit.

[0018] According to a second aspect of the present disclosure, a computing device is provided, including: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any of the schemes of the first aspect above.

[0019] According to a third aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored, which, when executed by a processor of an electronic device, causes the processor to perform the method described in any of the schemes of the first aspect above.

[0020] According to a fourth aspect of the present disclosure, a computer program product is provided, including executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as described in any of the schemes of the first aspect above. Attached Figure Description

[0021] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, wherein like reference numerals generally denote like parts.

[0022] Figure 1 A schematic diagram of the structure of a mobile phone according to at least one embodiment of the present disclosure is shown as an example.

[0023] Figure 2 An illustrative flowchart of a cross-process rendering method according to at least one embodiment of the present disclosure is shown as an example.

[0024] Figure 3An illustrative timing diagram of a cross-process rendering method according to at least one embodiment of the present disclosure is shown as an example.

[0025] Figure 4 An exemplary diagram illustrates the architecture of a cross-process rendering method according to at least one embodiment of the present disclosure, as well as some organizational structures and data flows generated in performing the method.

[0026] Figure 5 An exemplary schematic diagram illustrates the organization of a SurfaceView created in a service process and its relationship to the screen in a cross-process rendering method according to at least one embodiment of the present disclosure.

[0027] Figure 6 A schematic diagram of the organization of a surface view created in a client process in a cross-process rendering method according to at least one embodiment of the present disclosure is shown as an example.

[0028] Figure 7 A schematic diagram of the structure of a computing device according to at least one embodiment of the present disclosure is shown as an example. Detailed Implementation

[0029] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0030] It should be understood that in this disclosure, the terms "first," "second," "third," "fourth," etc., are used to distinguish similar objects, not to describe a specific order or sequence, and do not have any additional limiting effect. Furthermore, in this disclosure, "multiple" or similar expressions refer to two or more.

[0031] In computing devices such as mobile phones, in-vehicle systems, wearable devices, or other devices, when a running process needs to complete a graphics rendering task, it can have a rendering thread to execute the graphics rendering task. This rendering thread can use a rendering engine such as Unity to perform tasks such as scene drawing, resource management, and logic processing within the main thread. However, single-process rendering may bring the following problems:

[0032] For example, as application scenarios become more complex, the load of single-process rendering gradually increases, especially when processing large scenes or high-resolution images. Rendering tasks may consume a lot of CPU and GPU resources, leading to a decrease in system performance or even unstable frame rates.

[0033] For example, in single-process mode, if a rendering operation encounters an exception or memory leak, it may cause the entire application to crash. Because all critical tasks reside within a single process, a crash cannot guarantee the continued normal operation of other application functions.

[0034] For example, in single-process rendering, all resources are shared in the same memory space, which can easily lead to resource contention and conflicts. Resource operations between multiple modules or components may affect each other, making it difficult to guarantee the stability of resource management.

[0035] For example, because rendering tasks are concentrated in a single process, it is difficult to extend or introduce new rendering features without affecting the main application. Especially in application scenarios that require high concurrency or multi-tasking, the single-process mode has poor flexibility and adaptability.

[0036] In response to the aforementioned problems with single-process rendering, cross-process rendering solutions have been proposed to address various technical issues in traditional single-process rendering methods, thereby improving performance, stability, and scalability.

[0037] For example, existing solutions pass the client process's Surface directly to the server process, which then passes this Surface to a rendering engine such as Unity for rendering. However, while Surface supports cross-process transfer, it must be released after use, and Surface is prone to leakage after cross-process transfer (e.g., being hijacked by hooks). These issues require proper handling logic between processes; otherwise, stability problems such as memory leaks and Surface layer leaks can easily occur.

[0038] In other cases, the client process's Surface is frequently destroyed and recreated because it follows the client process's Surface lifecycle. Consequently, the Surface passed to the rendering engine, such as Unity, is also frequently destroyed and recreated. This is especially problematic when the client process is restarted due to an abnormal shutdown, for example. The application needs to recreate the objects and processes required for the rendering task before passing the newly created Surface to the server. During this time, the rendering task cannot continue, resulting in longer response times and impacting the user experience.

[0039] In other cases, because rendering engines such as Unity generally do not perform special processing on Surfaces transferred from other processes, they still consider them to be instances of the local process. Moreover, the Unity engine is a black box, and users cannot see its internal processing of Surfaces, which can easily lead to compatibility issues.

[0040] In view of this, this disclosure proposes a novel cross-process rendering method that eliminates the need to pass Surfaces between processes. Instead, the service process creates the Surface and passes its own Surface control information to other processes, allowing the service process to invoke the rendering engine to provide rendering services to these other processes. Since there is no need to pass Surfaces between processes, the aforementioned stability and compatibility issues are avoided. Furthermore, because the rendering engine performs rendering on the service process's Surface, it avoids the need to frequently destroy and rebuild the Unity engine-rendered Surface following the client process's Surface lifecycle, and the service process can continuously maintain rendering in the background. Therefore, the method of this disclosure ensures high efficiency and reliability in the rendering process, reduces system overhead, and accelerates the rendering engine's response speed. It is suitable for various application scenarios, and is particularly advantageous in complex scenarios where the Unity engine is used to provide rendering services for large games or real-time simulation systems.

[0041] The cross-process rendering method provided in this disclosure can be applied to various types of handheld devices (such as mobile phones, personal digital assistants (PDAs), etc.), various types of computers (such as tablets, laptops, ultra-mobile personal computers (UMPCs), netbooks, laptops, etc.), wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, and other terminal devices. It can also be applied to other systems that require rendering processing. This disclosure does not impose any restrictions on the specific type of application device, such as terminal devices, as long as the application device has rendering requirements.

[0042] As an example and not a limitation, when the terminal device is a wearable device, the wearable device can also be a general term for devices that apply wearable technology to intelligently design and develop everyday wearables, such as gloves, watches, AR head-mounted displays, VR head-mounted displays, or MR (Mixed Reality) head-mounted displays equipped with far-field communication modules and / or near-field communication modules.

[0043] In some embodiments, the terminal device may be equipped with, for example, Figure 1 The exemplary mobile phone 100 with the exemplary hardware structure shown is an example of such a device. Figure 1As shown, the mobile phone 100 may specifically include: a radio frequency (RF) circuit 110, a memory 120, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a short-range wireless communication module 170, a processor 180, and a power supply 190, etc. Those skilled in the art will understand that... Figure 1 The structure of the mobile phone 100 shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0044] The following is combined Figure 1 A detailed introduction to each component of the mobile phone 100:

[0045] RF circuit 110 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 180; additionally, it transmits uplink data to the base station. Typically, RF circuitry includes, but is not limited to, antennas, at least one amplifier, transceiver, coupler, low-noise amplifier (LNA), duplexer, etc. Furthermore, RF circuit 110 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), New Radio (NR), GNSS, FM, low-Earth orbit satellite connections, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS), etc.

[0046] The memory 120 can be used to store software programs and modules. The processor 180 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 120. The memory 120 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as pictures, audio data, phonebook, etc.). In addition, the memory 120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Specifically, the memory 120 may store programs for implementing the cross-process rendering method according to this disclosure, as detailed later. These programs may be standalone applications or functional modules embedded in one or more applications or operating systems.

[0047] The input unit 130 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone 100. Specifically, the input unit 130 may include a touch panel 131 and other input devices 132. The touch panel 131, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using fingers (e.g., one or more fingers), styluses, or any suitable object or accessory on or near the touch panel 131), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 131 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 180, and can receive and execute commands sent by the processor 180. In addition, the touch panel 131 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 131, the input unit 130 may also include other input devices 132. Specifically, other input devices 132 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0048] Display unit 140 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. For example, display unit 140 is configured to display the user interface of various applications, and some or all of its displayed content can be rendered by the cross-process rendering method of this disclosure. Display unit 140 may include display panel 141, which can optionally be configured as a liquid crystal display (LCD), light-emitting diode (LED), organic light-emitting diode (OLED), active-matrix organic light-emitting diode (AMOLED), etc. Further, touch panel 131 may cover display panel 141. When touch panel 131 detects a touch operation on or near it, it transmits the information to processor 180 to determine the type of touch event. Subsequently, processor 180 provides corresponding visual output on display panel 141 according to the type of touch event. Although in Figure 1 In this embodiment, the touch panel 131 and the display panel 141 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 131 and the display panel 141 can be integrated to realize the input and output functions of the mobile phone.

[0049] The mobile phone 100 may also include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 141 according to the ambient light level, and the proximity sensor can turn off the display panel 141 and / or the backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), and vibration recognition functions (such as pedometers, taps, etc.). Other sensors that may be configured in the phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0050] Audio circuit 160, speaker 161, and microphone 162 provide an audio interface between the user and the mobile phone. Audio circuit 160 converts received audio data into electrical signals and transmits them to speaker 161, where speaker 161 converts them into sound signals for output. On the other hand, microphone 162 converts collected sound signals into electrical signals, which are received by audio circuit 160, converted into audio data, and then output to processor 180 for processing. The audio data is then transmitted via RF circuit 110 to, for example, another electronic device, or output to memory 120 for further processing.

[0051] Wi-Fi, Bluetooth, Near Field Communication (NFC), and Ultra Wide Band (UWB) are all short-range wireless transmission technologies. Mobile phones, through a short-range wireless module 170, can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. The aforementioned short-range wireless module 170 may include a Wi-Fi chip, a Bluetooth chip, an NFC chip, and a UWB chip. The Wi-Fi chip enables the mobile phone 100 to establish Wi-Fi Direct connections with other electronic devices. It can also enable the mobile phone 100 to operate in Access Point (AP) mode, which provides wireless access services and allows other wireless devices to connect, or in Station (STA) mode, which allows connection to an AP but does not accept wireless devices, thereby establishing point-to-point communication between the mobile phone 100 and other Wi-Fi devices.

[0052] The processor 180 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 120, and by calling data stored in the memory 120, thereby providing overall monitoring of the phone. Optionally, the processor 180 may include one or more processing units; optionally, the processor 180 may include, for example, an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. Different processing units can be independent devices or integrated into one or more processors.

[0053] The mobile phone 100 also includes a power supply 190 (such as a battery) that supplies power to various components. For example, the power supply can be logically connected to the processor 180 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0054] The following is combined Figure 2 The flowchart will be used to discuss in detail the overall scheme according to at least one embodiment of the present disclosure.

[0055] Figure 2 An illustrative flowchart of a cross-process rendering method according to at least one embodiment of the present disclosure is shown as an example.

[0056] The cross-process rendering method of this disclosure operates on the same computing device (e.g., a mobile phone or in-vehicle device). The service process can be created by a service on that computing device. This service process can call the rendering engine recorded on the computing device to provide rendering services to one or more first processes (also referred to as client processes) installed on that computing device. The client processes mentioned in the cross-process rendering method of this disclosure (e.g., first process, second process, or third process) can be processes corresponding to native applications of the computing device and downloaded and installed third-party applications, or processes corresponding to native services of the computing device such as a wallpaper service. Client processes with rendering needs can interact with the service process to allow the service process to perform rendering processing for them. Using the cross-process rendering method of this disclosure enables reasonable division of rendering tasks, leading to performance optimization.

[0057] Figure 2 This illustrates the basic operations of a client process interacting with a service process on a computing device. The word "first" before the process is only used to distinguish among multiple client processes, not to specifically refer to a certain type of process. The first process can be any application or service installed on the computing device that requires rendering.

[0058] like Figure 2 As shown, in step S210, the first process sends a rendering request for the first rendering unit to the service process, and correspondingly, in step S220, the service process receives the rendering request for the first rendering unit sent by the first process.

[0059] The rendering unit in this embodiment has a separate surface, which may be, for example, a surface view or a window.

[0060] In some implementations, the first rendering unit can be a surface view to be rendered by the first process. For example, the first process can create the first rendering unit by: creating a subclass of FrameLayout, adding the subclass to the Activity layout to be rendered as a child node of the ContentView, creating a surface view inside the subclass as the first rendering unit, and calling the addView method to add the created surface view to the subviews of the subclass.

[0061] In some implementations, where the first process does not have a view tree to load the surface view, for example, if the first process is the process corresponding to the Wallpaper Service, the first rendering unit can be the window that the first process wants to render.

[0062] Communication between the first process and the service process can be achieved using any cross-process communication method. For example, the AIDL (Android Interface Definition Language) interface can be used to achieve cross-process communication.

[0063] Then, in step S230, the service process creates a first surface view and passes the surface control information (SurfaceControl) of the first surface view to the first process. Correspondingly, in step S240, the first process receives the surface control information of the first surface view from the service process.

[0064] Although step S230 is shown to be executed after step S220 in this embodiment, it should be understood that in some examples step S230 may also be executed before step S220. For example, the service process may pre-create the SurfaceView and immediately pass its SurfaceControl after receiving a rendering request from a client process, thereby improving the service's response speed. Alternatively, in some cases, a SurfaceView can be used by multiple client processes or by a single client process multiple times. After the previous client process finishes rendering, the SurfaceView becomes idle, and the service process can then pass its SurfaceControl to the current client process for use.

[0065] It is understandable that a computing device can run a single client process or multiple client processes simultaneously (such as the first process, second process, and third process described in this article). The service process can create multiple SurfaceViews and pass the surface control information of each SurfaceView to different client processes. This allows the computing device to use the rendering capabilities provided by a service process to achieve, for example, split-screen display, where each split screen can correspond to the display of a client process.

[0066] In some implementations, the rendering engine associated with the service process can support rendering multiple displays, providing rendering services to multiple client processes simultaneously. The service process can create multiple SurfaceViews, where each SurfaceView's Surface is bound to a Display of the rendering engine, allowing the rendering engine to render the content of each Display onto its bound Surface.

[0067] This multi-screen rendering capability allows rendering tasks to be distributed across multiple processes, fully utilizing the computing power of multi-core processors. Different processes can execute rendering tasks in parallel, significantly improving overall rendering efficiency. Furthermore, different processes can independently manage and execute different rendering tasks, such as main rendering, post-processing, and UI rendering, reducing the burden on a single process and avoiding performance bottlenecks.

[0068] In some examples, the mapping between multiple rendering units of multiple processes and multiple screens of the rendering engine can be pre-defined within the service process. This means that each rendering unit of each client process is pre-set to use which screen for rendering. As needed, a process can have one or more rendering units that require rendering from the service process, with each rendering unit having a pre-defined corresponding screen.

[0069] Accordingly, after the service process receives the rendering request of the first rendering unit from the first process in step S220, it can determine the first screen of the rendering engine that the first rendering unit should use based on the correspondence, and then determine whether the first screen has been enabled, that is, whether it has been bound to a surface view.

[0070] If the service process determines that the first screen is not yet enabled, it creates a first surface view, binds the first surface of the first surface view to the first screen, and passes the surface control information of the first surface view to the first process. If the service process determines that the first screen is enabled, it passes the surface control information of the first surface view that has been created and bound to the first screen to the first process.

[0071] In some implementations, the first surface of a first surface view can be bound to the first screen by calling methods related to the rendering engine; for example, in the case of a Unity engine, methods related to the Unity engine can be called to achieve this binding. For instance, the service process can obtain information about the first surface of the first surface view in the surfaceCreated callback of the first surface view, and call the updateDisplayInternal method of the rendering engine's player (such as UnityPlayer) to bind the first surface to the first screen of the rendering engine (such as Unity engine) using the obtained information about the first surface.

[0072] In some implementations, the service process can utilize a class in the Android system, namely the SurfaceControlViewHost, to create a first surface view. For example, the service process creates a first SurfaceControlViewHost and adds a new surface view as the first surface view in the view tree of the first SurfaceControlViewHost. Thus, the service process can pass the surface control information of the first surface view to the first process by returning the surface package (SurfacePackage) of the first SurfaceControlViewHost to the first process, where the SurfacePackage contains the surface control information of the first surface view.

[0073] Then, in step S250, the first process uses the surface control information of the first surface view to associate the first surface of the first surface view with the second surface created by the first rendering unit. After the association is established, in step S260, the first process sends the rendering data of the first rendering unit to the rendering engine associated with the service process, and correspondingly, in step S270, the service process receives the rendering data of the first rendering unit from the first process, and renders the content of the first rendering unit on the first surface of the first surface view using the rendering engine associated with the service process, so that the second surface associated with the first surface displays the rendering result. As mentioned above, each rendering unit (including surface views, windows, etc.) has only one surface. Therefore, the ordinal numbers "first," "second," etc., before "surface" in this disclosure are only used to distinguish the surfaces of each rendering unit, and do not indicate that each rendering unit has multiple surfaces.

[0074] Since the surface of the first surface view has been associated with the surface of the first rendering unit, the first process can display the rendered content on the surface of the first surface view as the content of the first rendering unit.

[0075] This allows time-consuming rendering tasks to be allocated to independent processes, reducing the main process's blocking time, lowering rendering latency, and improving frame rate stability. Furthermore, process isolation ensures that even if one process malfunctions or crashes, it won't affect the normal operation of other processes, thus improving overall application stability. Independent rendering processes can quickly restart and recover in the background after an error, ensuring the system can continue running even in extreme situations and minimizing the negative impact of crashes on user experience.

[0076] In some implementations, the first process can reparent the surface control information of the first surface view to the surface control information of the first rendering unit, thereby setting the first surface as a child node of the second surface, thus realizing the association between the surface of the first surface view and the surface of the first rendering unit.

[0077] In some implementations, when the first rendering unit is the third surface view to be rendered by the first process, the aforementioned second surface is the surface of the third surface view. Additionally, as mentioned earlier, the first process can obtain surface control information for the first surface view by receiving the SurfacePackage from the first SurfaceControlViewHost from the service process, where the first surface view is a surface view in the View Tree of the first SurfaceControlViewHost. Then, the first process can call the `setChildSurfacePackage` method for the third surface view and the received SurfacePackage, thereby setting the first surface as a child node of the second surface, thus associating the first surface of the first surface view with the second surface of the first rendering unit.

[0078] In some implementations, when the first rendering unit is the window to be rendered by the first process, the aforementioned second surface is the surface of that window. Additionally, as mentioned earlier, the first process can obtain surface control information for a first surface view by receiving a SurfacePackage from a first SurfaceControlViewHost from a service process, where the first surface view is a surface view in the View Tree of the first SurfaceControlViewHost. Then, the first process can obtain the surface control information of the window and the surface control information in the received SurfacePackage, and use methods in the SurfaceControl.Transaction method to reparent the surface control information in the SurfacePackage to the surface control information of the window, thereby setting the first surface as a child node of the window's surface, thus associating the first surface of the first surface view with the second surface of the first rendering unit.

[0079] When the first rendering unit is the window to be rendered by the first process, the first process can be the process corresponding to WallpaperService, and the surface control information of the window and the surface control information in the SurfacePackage can be obtained through the following operations: Create an instance of Engine (Engine) by inheriting the engine class (WallpaperService.Engine) of WallpaperService, and in the surface creation (onSurfaceCreated) callback of the engine instance, obtain the surface control member variable (mBbqSurfaceControl or mSurfaceControl) as the surface control information of the window through reflection, and obtain the surface control information in the SurfacePackage through the method of obtaining surface control information (getSurfaceControl) by reflection.

[0080] After the association is established, the service process can receive rendering data from the first rendering unit from the first process through, for example, the AIDL interface, and then communicate with the Unity engine, which is the rendering engine, through, for example, the UnityPlayer.UnitySendMessage method, to pass the rendering data to the rendering engine and let the rendering engine perform the corresponding rendering processing.

[0081] Therefore, the method of this embodiment achieves cross-process rendering by passing surface control information across processes. Instead of passing the surface across processes, the rendering engine performs rendering on the surface of the serving process, ensuring the efficiency and reliability of the rendering process.

[0082] Furthermore, in some implementations, rendering efficiency can be improved through multi-core parallel processing. For example, parallel processing can be achieved by running multiple service processes on multiple cores, thus leveraging the advantages of multi-core processors. Therefore, cross-process rendering can handle more complex and demanding rendering scenarios, such as massively multiplayer online games, large open-world games, and high-resolution real-time simulation applications, providing a smooth user experience. Additionally, for applications requiring real-time rendering and interaction (such as AR / VR applications), cross-process rendering can offer higher response times and lower latency, enhancing user immersion.

[0083] In addition, in some embodiments, the method of this disclosure may further include:

[0084] In response to detecting that the first process has terminated abnormally, the service process terminates the association between the first surface and the second surface;

[0085] In response to detecting that the first process has restarted, the service process retransmits the surface control information of the first surface view to the first process, so that the first process uses the surface control information of the first surface view to associate the first surface with the second surface recreated by the first rendering unit.

[0086] This solves the problem in existing technologies where the rendering engine needs to be restarted when the application process crashes and is running within the application process.

[0087] Using the method disclosed herein, a fault detection and recovery mechanism can be established by monitoring the status of each rendering process. Once a process crash is detected, the process can be quickly restarted and its rendering state restored to minimize the impact on the overall rendering.

[0088] Next, the combination Figure 3 Describe an example of a service process providing rendering services to multiple client processes.

[0089] Figure 3 A schematic timing diagram is shown as an example of a cross-process rendering method executed between a service process and multiple client processes according to at least one embodiment of the present disclosure. It should be understood that... Figure 3 The steps or operations described herein are merely exemplary and not restrictive, and are not intended to limit the scope of this disclosure. For example, the number of screens supported by the rendering engine, the number of client processes supported by the service process, and the number of rendering units in each client process shown in the figures are all exemplary. Figure 3 For details of each step or operation, or alternative methods, please refer to the preceding text. Figures 1 to 2 The content described will not be repeated here. It will be described later. Figure 3The various details can also be combined with the previous ones. Figures 1 to 2 The various embodiments or examples described are used in combination.

[0090] like Figure 3 As shown, the rendering engine associated with the service process supports rendering multiple screens, and the service process has a pre-defined correspondence between each rendering unit of each client process and the screen of the rendering engine. For example, the first rendering unit of the first process, the second rendering unit of the second process, and the third rendering unit of the third process correspond to the first screen, the second screen, and the first screen, respectively.

[0091] Although Figure 3 The timing diagram shows that the rendering of the first rendering unit of the first process is completed before the rendering of the second rendering unit of the second process is performed. However, since the first and second rendering units correspond to different screens of the rendering engine, the rendering engine can perform the rendering of these two rendering units simultaneously. Correspondingly, the service process can handle the rendering requests of these two rendering units simultaneously. That is to say, steps S321-S326 and steps S311-S316 can be executed independently. The execution time of steps S321-S326 is not limited by the execution time of steps S311-S316, and can be executed before them, or in parallel or partially in parallel.

[0092] In addition, although Figure 3 The timing diagram shows that the rendering of the third rendering unit of the third process is performed after the rendering of the second rendering unit of the second process is completed. However, since the third rendering unit and the second rendering unit also correspond to different screens of the rendering engine, steps S331-S336 and steps S321-S326 can be executed independently. The execution time of steps S331-S336 is not limited by the execution time of steps S321-S326, and can be executed before, in parallel, or in partial parallel. However, since the third rendering unit of the third process and the first rendering unit of the first process use the same screen (the first screen), the rendering operation of the third rendering unit, i.e., steps S331-S336, must be performed after the rendering of the first rendering unit is completed. Conflicts can be avoided by better designing the correspondence between the rendering units of each client process and the screens of the rendering engine, without requiring the service process to determine whether the screen to be used by the current rendering request is idle.

[0093] like Figure 3 As shown, steps S311-S316 are the process by which the service process provides rendering services to the first process.

[0094] In step S311, the first process sends a rendering request for the first rendering unit to the service process.

[0095] Next, in step S312, the service process determines, based on the aforementioned preset correspondence between rendering units and screens, that the first rendering unit should use the first screen of the rendering engine, and determines that the first screen has not yet been enabled. Thus, the service process creates a first surface view and binds the first surface of the first surface view to the first screen.

[0096] Next, in step S313, the service process transmits the surface control information of the first surface view to the first process.

[0097] Next, in step S314, the first process uses the surface control information of the first surface view to associate the first surface of the first surface view with the second surface of the first rendering unit.

[0098] Next, in step S315, the first process sends the rendering data of the first rendering unit to the service process, and in step S316, upon receiving the data, the service process transmits the rendering data to its associated rendering engine and accordingly instructs the rendering engine to render the first screen based on the rendering data. Then, the rendering engine associated with the service process renders the content of the first rendering unit based on the received rendering data and renders it as the content of the first screen onto the first surface of the first surface view bound to the first screen.

[0099] It should be understood that after step S316, if the rendering data of the first rendering unit changes, i.e., the first process needs to change the rendering content of the first rendering unit, steps S315 and S316 can be repeated. After the rendering of the first rendering unit is completed, the first process can destroy the first rendering unit, but no operation needs to be performed on the first surface view. At this time, the association between the first rendering unit and the first surface view is naturally released, and the first surface view remains in the background service process for use by other rendering units later.

[0100] Steps S321-S326 are the process by which the service process provides rendering services to the second process.

[0101] In step S321, the second process sends a rendering request for the second rendering unit to the service process.

[0102] Next, in step S322, the service process determines, based on the aforementioned preset correspondence between rendering units and screens, that the second rendering unit should use the second screen of the rendering engine, and determines that the second screen has not yet been enabled. Thus, the service process creates a second surface view and binds the third surface of the second surface view to the second screen.

[0103] Next, in step S323, the service process transmits the surface control information of the second surface view to the second process.

[0104] Next, in step S324, the second process uses the surface control information of the second surface view to associate the third surface of the second surface view with the fourth surface of the second rendering unit.

[0105] Next, in step S325, the second process sends the rendering data of the second rendering unit to the service process, and in step S326, upon receiving the data, the service process transmits the rendering data to its associated rendering engine and accordingly instructs the rendering engine to render the second screen based on the rendering data. Then, the rendering engine renders the content of the second rendering unit based on the received rendering data and renders it as the content of the second screen onto the third surface of the second surface view bound to the second screen.

[0106] Similarly, it should be understood that after step S326, if the rendering data of the second rendering unit changes, i.e., the second process needs to change the rendering content of the second rendering unit, steps S325 and S326 can be repeated. After the rendering of the second rendering unit is completed, the second process can destroy the second rendering unit, but no operation needs to be performed on the second surface view. At this time, the association between the second rendering unit and the second surface view is naturally released, and the second surface view remains in the background service process for subsequent use by other rendering units.

[0107] Steps S331-S336 are the process by which the service process provides rendering services to the third process.

[0108] In step S331, the third process sends a rendering request for the third rendering unit to the service process. At this time, the rendering of the first rendering unit has been completed, that is, the first screen and its bound first surface are in an idle state.

[0109] Next, in step S332, the service process determines, based on the aforementioned preset correspondence between rendering units and screens, that the third rendering unit should use the first screen of the rendering engine, and determines that the first screen has been enabled. Thus, in step S333, the service process directly transmits the surface control information of the previously created first surface view to the third process.

[0110] Next, in step S334, the third process uses the surface control information of the first surface view to associate the first surface of the first surface view with the fifth surface of the third rendering unit.

[0111] Next, in step S335, the third process sends the rendering data of the third rendering unit to the service process, and in step S336, upon receiving the data, the service process transmits the rendering data to its associated rendering engine, and accordingly instructs the rendering engine to render the first screen based on the rendering data. Then, the rendering engine renders the content of the third rendering unit based on the received rendering data of the third rendering unit, and renders it as the content of the first screen onto the first surface of the first surface view bound to the first screen.

[0112] Similarly, it should be understood that after step S336, if the rendering data of the third rendering unit changes, i.e., the third process needs to change the rendering content of the third rendering unit, steps S335 and S336 can be repeated. After the rendering of the third rendering unit is completed, the third process can destroy the third rendering unit, but does not need to perform any operation on the first surface view. At this time, the association between the third rendering unit and the first surface view is naturally released, and the first surface view remains in the background service process for subsequent use by other rendering units.

[0113] Therefore, the method of this disclosure can provide rendering services to multiple client processes simultaneously using the multi-screen mechanism of the rendering engine associated with the service process, thereby improving rendering efficiency. Furthermore, the surface bound to each screen can be permanently retained in the background service process, without needing to be destroyed and rebuilt as the rendering unit surfaces of the client processes are destroyed and rebuilt, thus reducing system overhead and accelerating the rendering engine's response speed.

[0114] Furthermore, the aforementioned cross-process rendering scheme allows rendering logic to be divided into multiple independent modules, each running in a different process. This modular design makes the system easier to expand and maintain, and facilitates the introduction of new features or customized development. Because different functional modules run in independent processes, individual updates or optimizations can be performed on specific processes or modules without affecting the overall system operation. This improves the system's flexibility and maintainability.

[0115] Because cross-process rendering solutions can achieve independent updates and maintenance of functions through modular design, they are more easily compatible with different versions of Unity and Android systems, reducing problems caused by version upgrades. Furthermore, cross-process rendering solutions can be adjusted for different hardware configurations and system environments, ensuring optimized rendering results on multiple devices and operating systems.

[0116] Furthermore, by sharing and managing resources across processes, resource conflicts between different modules are avoided, while improving the security and stability of resource usage. Additionally, the system can dynamically adjust resource allocation for each process based on demand, enabling it to flexibly adapt to different scenario requirements and thus optimize resource utilization.

[0117] The following is combined Figures 4 to 6 An example of a cross-process rendering method implemented on an Android system according to embodiments of the present disclosure is described in more detail.

[0118] Figure 4 An exemplary diagram illustrates the architecture of a cross-process rendering method according to at least one embodiment of the present disclosure, as well as some organizational structures and data flows generated in performing the method. Figure 5 An exemplary schematic diagram illustrates the organization of a SurfaceView created in a service process and its relationship to the screen in a cross-process rendering method according to at least one embodiment of the present disclosure. Figure 6 A schematic diagram of the organization of a SurfaceView created in a client process in a cross-process rendering method according to at least one embodiment of the present disclosure is shown as an example.

[0119] like Figure 4 As shown, the solution of this embodiment can be divided into two parts: a server and a client. The method of this embodiment can be implemented using one server application and multiple client applications installed on an Android system. The server application accepts requests from client applications and provides rendered images, while the client applications send rendering data to the server application for rendering.

[0120] The server application consists of two parts: an Android functional part and a Unity engine as the rendering engine. The Android functional part acts as a container for the Unity engine to run, providing the Unity engine with the ability to access the Android environment, while the Unity engine provides rendering capabilities and has a multi-screen mechanism.

[0121] An SDK (Software Development Kit) can be packaged for each client application that requires it, allowing them to integrate and implement cross-process rendering. Each client application's corresponding process can use its own packaged SDK. For example, Figure 4 The first, second, and third processes in the application use SDK1, SDK2, and SDK3, respectively, which are encapsulated in their respective applications.

[0122] After starting the server application, a service process is created to prepare for executing the cross-process rendering method of this embodiment. This service process corresponds to the Android functional part described above. After starting the client application, a corresponding client process is created, which can access the encapsulated SDK to perform the corresponding cross-process rendering operation when there is a need for cross-process rendering.

[0123] First, we will discuss the operation of the service process in the cross-process rendering method of this disclosure embodiment.

[0124] like Figure 4 As shown, the service process can create AIDL interfaces and corresponding services for IPC (Inter-Process Communication) between the service process and the client process. For example, the service process can pass SurfaceControlViewHost.SurfacePackage to the client process, and the client process can send data and instructions to the service process.

[0125] Additionally, in some cases, the service process can preload a scene upon startup to trigger the initialization of the Unity engine, thereby saving the time of initializing the Unity engine when subsequent client processes request rendering, thus speeding up the rendering process.

[0126] Additionally, as mentioned earlier, the service process can leverage Unity's multi-screen mechanism to render simultaneously for multiple client processes. Therefore, as... Figure 4 As shown, the service process corresponding to the Android function part can include the UnityDisplay management unit, which can pre-determine the correspondence between each rendering unit of each client process and each screen (Display) of the Unity engine to ensure that there are no conflicts.

[0127] Additionally, when enabling a Display, the Android side needs to create a SurfaceView and bind it for rendering by the Unity engine. To do this, the UnityDisplay management department can create a UnityDisplay class to manage the mapping between SurfaceViews and Displays created by the Android functionalities. In other words, when a client process requests to render a screen, the UnityDisplay management department finds that the corresponding Display is not yet enabled, creates a UnityDisplay class, creates a SurfaceView, and binds its Surface to the corresponding Display.

[0128] Figure 5This example illustrates the organization of the SurfaceView on the Android side of the service process and its relationship with the Display in the Unity engine.

[0129] like Figure 5 As shown, the Android functionality portion of the service process can utilize the Android SurfaceControlViewHost class to create a background-rendered View Tree. This View Tree fully supports Android's View controls, so a SurfaceView can be created and loaded into this View Tree.

[0130] The service process can obtain the Surface object in the SurfaceView's surfaceCreated callback, and then call the UnityPlayer's updateDisplayInternal method to bind the Surface to the corresponding Display. Unity's multi-screen mechanism will then render the content on this Display onto the bound Surface.

[0131] Back Figure 4 The diagram illustrates that the Unity engine supports two screens, Display0 and Display1. Correspondingly, two SurfaceControlViewHosts, SurfaceControlViewHost0 and SurfaceControlViewHost1, are created in the service process corresponding to the Android functionality. Their View Trees load the newly created SurfaceView0 and SurfaceView1, respectively. The Surfaces of SurfaceView0 and SurfaceView1 are bound to Display0 and Display1, respectively, so that the Unity engine can render on the Surface.

[0132] The service processes corresponding to the aforementioned Android functionalities can communicate with the Unity engine through the UnityPlayer.UnitySendMessage method, for example, by switching scenes / displays and transmitting data.

[0133] In addition, the Unity engine may also communicate with the service process corresponding to the Android functional part by accessing a static method provided by the service process corresponding to the Android functional part, so as to obtain some information of the service process corresponding to the Android functional part and execute the method provided by the service process corresponding to the Android functional part, etc. For example, the Unity engine may set linkage between the time engine and the ambient light through the service process corresponding to the Android functional part.

[0134] The operations of the client process created by each application in the cross-process rendering method of the embodiments of the present disclosure will be discussed next.

[0135] As shown in Figure 4 , the process of each application needs to access the corresponding SDK encapsulated therein when performing cross-process rendering, and the SDK may perform the following operations to implement cross-process rendering.

[0136] The SDK in each application may obtain the AIDL interface created by the service process, call an initialization method after the first process accesses, and bind the service on the server side, so as to communicate with the service process.

[0137] In addition, for a client process (such as the first process and the second process) that has a View Tree capable of loading a SurfaceView, the SDK therein, such as SDK1 and SDK2, may Figure 6 create a SurfaceView for cross-process rendering as shown.

[0138] Figure 6 The diagram exemplarily shows a schematic diagram of the organizational structure of the SurfaceView created in the client process in the cross-process rendering method according to at least one embodiment of the present disclosure.

[0139] As shown in Figure 6 , the SDK may create a subclass of FrameLayout, name it RenderView, and add it to the Activity layout to be rendered as a child node of the ContentView. After the RenderView is created, a SurfaceView is created inside the RenderView, and after creation, the addView method is called to add it to the child Views of the RenderView. The RenderView is used to wrap the SurfaceView and provide a convenient interface for invocation.

[0140] Returning to Figure 4Taking the first process as an example, SDK1 creates RenderView1, which wraps SurfaceView2. RenderView1 can also provide the changeScene method to request the screen from Unity. When the first process calls the changeScene method, it can send the corresponding rendering request to the service process through the AIDL interface. Then, the service process can determine, as mentioned earlier, that Display0 should be used. Since Display0 is not currently enabled, it creates SurfaceControlViewHost0 and its SurfaceView0, and binds the Surface of SurfaceView0 to Display0. Then, the service process can send a command to the Unity engine to switch to Display0 and return the SurfaceControlViewHost.SurfacePackage corresponding to SurfaceControlViewHost0 to the first process.

[0141] After SDK1 receives the SurfaceControlViewHost.SurfacePackage returned by the service process, it calls the setChildSurfacePackage method on SurfaceView2 to reparent the received SurfacePackage to the SurfaceControl of SurfaceView2. Figure 4 The dotted line with "reParent" is shown in the diagram. This implements the mounting operation of the SurfaceControl, which sets the Surface of the SurfaceView in the SurfacePackage (i.e., the Surface of SurfaceView0) as a child node of the SurfaceView2's Surface, thus allowing the view of the Surface rendered by the service process (i.e., the SurfaceView0's Surface) to be displayed in the window of the first process.

[0142] Additionally, for client processes that do not have a View Tree and therefore cannot load SurfaceView, such as third-party processes (e.g., the process corresponding to WallpaperService), their SDKs, such as SDK3, can perform SurfaceControl mounting operations as follows.

[0143] Taking the process corresponding to WallpaperService as an example, firstly, you can develop WallpaperService, inherit WallpaperService.Engine to create an instance of Engine, and then in the onSurfaceCreated callback of this Engine instance, obtain the mBbqSurfaceControl member variable or mSurfaceControl member variable of Engine through reflection. This member variable is the SurfaceControl used by WallpaperService to manage the Surface of its window.

[0144] In addition, similar to the first process mentioned above, SDK3 can send rendering requests to the service process through the AIDL interface and obtain the SurfaceControlViewHost.SurfacePackage corresponding to the SurfaceControlViewHost0 bound to Display0. Then, the getSurfaceControl method of reflection is used to obtain the SurfaceControl in this SurfacePackage (this SurfaceControl is used to control the SurfaceView0 rendered by the service process), and then the reparent method in SurfaceControl.Transaction is executed, along with the show and apply methods in Transaction, thereby reparenting the SurfaceControl in SurfacePackage to the SurfaceControl of the WallpaperService window, as shown below. Figure 4 The dotted line with "reParent" is shown in the diagram. This implements the mounting operation of SurfaceControl, which sets the Surface of SurfaceView (i.e., the Surface of SurfaceView0) in the SurfacePackage received from the service process as a child node of the Surface of the third process's window, thus allowing the Surface rendered by the service process (i.e., the Surface of SurfaceView0) to be displayed in the window of the third process.

[0145] As mentioned above, regardless of which client process is involved, the SurfaceControl mounting operation is performed by the client process itself; whoever uses it performs the operation, while the service process only focuses on providing rendering. The service process can continuously render content to the corresponding Surface in the background without being concerned about changes in the rendering units of the foreground client process.

[0146] After the SurfaceControl mounting operation is completed, each SDK can send instructions and data to the Unity engine associated with the service process through the AIDL interface.

[0147] Figure 7 A schematic diagram of the structure of a computing device that can be used to implement the above-described cross-process rendering method according to at least one embodiment of the present disclosure is shown.

[0148] See Figure 7 The computing device 700 includes a memory 710 and a processor 720.

[0149] Processor 720 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 720 may include a general-purpose main processor and one or more special-purpose coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, processor 720 may be implemented using custom circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0150] Memory 710 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 720 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 710 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 710 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0151] The memory 710 stores executable code, which, when processed by the processor 720, enables the processor 720 to execute the cross-process rendering method described above.

[0152] Furthermore, the method according to this disclosure can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing the steps defined in the above-described method of this disclosure.

[0153] Alternatively, this disclosure may be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) that, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the steps of the method described above according to this disclosure.

[0154] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0156] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A cross-process rendering method, comprising: The first process sends a rendering request for the first rendering unit to the service process; The service process receives the rendering request from the first rendering unit sent by the first process; The service process creates a first surface view and transmits the surface control information of the first surface view to the first process; The first process receives surface control information of the first surface view from the service process; The first process uses the surface control information of the first surface view to associate the first surface view with the second surface created by the first rendering unit; The first process sends the rendering data of the first rendering unit to the rendering engine associated with the service process; as well as The service process renders the content of the first rendering unit on the first surface of the first surface view using the rendering engine associated with the service process based on the rendering data of the first rendering unit received from the first process, so that the second surface associated with the first surface displays the rendering result. The first process associates the first surface of the first surface view with the second surface created by the first rendering unit using the surface control information of the first surface view, including: the first process reparents the surface control information of the first surface view to the surface control information of the first rendering unit, thereby setting the first surface as a child node of the second surface.

2. The method according to claim 1, further comprising: The second process sends a rendering request for the second rendering unit to the service process; The service process receives a rendering request for the second rendering unit sent by the second process; The service process creates a second surface view and passes the surface control information of the second surface view to the second process; The second process receives surface control information of the second surface view from the service process; The second process uses the surface control information of the second surface view to associate the third surface of the second surface view with the fourth surface created by the second rendering unit; The second process sends the rendering data of the second rendering unit to the rendering engine associated with the service process; as well as The service process, based on the rendering data of the second rendering unit received from the second process, uses the rendering engine to render the content of the second rendering unit on the third surface of the second surface view so that the fourth surface associated with the third surface displays the rendering result. The first surface and the third surface are respectively bound to the first screen and the second screen of the rendering engine so that the contents of the first screen and the second screen are rendered onto the first surface and the third surface, respectively, wherein the contents of the first screen and the second screen are the contents of the first rendering unit and the second rendering unit rendered by the rendering engine.

3. The method according to claim 1, further comprising: After the first rendering unit completes its rendering, the third process sends a rendering request for the third rendering unit to the service process. The service process receives a rendering request from the third rendering unit sent by the third process; The service process transmits the surface control information of the first surface view to the third process; The third process receives surface control information of the first surface view from the service process; The third process uses the surface control information of the first surface view to associate the first surface of the first surface view with the fifth surface created by the third rendering unit; The third process sends the rendering data of the third rendering unit to the rendering engine associated with the service process; as well as The service process renders the content of the third rendering unit on the first surface using the rendering engine based on the rendering data of the third rendering unit received from the third process, so that the fifth surface associated with the first surface displays the rendering result.

4. The method according to claim 1, wherein, The rendering engine supports rendering for multiple screens, and the service process pre-defines the correspondence between multiple rendering units of multiple processes and multiple screens of the rendering engine. After receiving the rendering request from the first rendering unit from the first process, the service process determines, based on the correspondence, the first screen that the first rendering unit should use the rendering engine for. If the service process determines that the first screen is not yet enabled, it creates the first surface view, binds the first surface of the first surface view to the first screen, and passes the surface control information of the first surface view to the first process. When the service process determines that the first screen has been enabled, it transmits the surface control information of the first surface view that has been created and bound to the first screen to the first process.

5. The method according to claim 1, wherein, The service process creates the first surface view, including: The service process creates the first surface control information view holder, and The service process adds a new surface view as the first surface view in the view tree of the first surface control information view holder; Furthermore, the service process transmits the surface control information of the first surface view to the first process, including: the service process returning the surface encapsulation of the first surface control information view holder to the first process, wherein the surface encapsulation contains the surface control information of the first surface view.

6. The method according to claim 5, further comprising: The service process obtains information about the first surface of the first surface view in the surface creation callback of the first surface view; as well as The service process calls the player's update screen method of the rendering engine, and uses the information of the first surface to bind the first surface to the first screen of the rendering engine.

7. The method according to claim 1, wherein, The first rendering unit is the third surface view to be rendered by the first process, and the second surface is the surface of the third surface view; The first process receives surface control information of the first surface view from the service process, including: the first process receiving a surface encapsulation of a first surface control information view holder from the service process, wherein the first surface view is a surface view in the view tree of the first surface control information view holder, and the surface encapsulation contains the surface control information of the first surface view; and The first process uses the surface control information of the first surface view to associate the first surface of the first surface view with the second surface created by the first rendering unit, including: the first process calls the set sub-surface encapsulation method for the third surface view and the surface encapsulation, thereby setting the first surface as a child node of the second surface.

8. The method according to claim 7, wherein, The first rendering unit is the third surface view to be rendered by the first process, including: The first process creates a subclass of FrameLayout. The first process adds the subclass to the activity layout to be rendered as a child node of the content view. The first process creates a surface view inside the subclass as the third surface view, and the first process calls the add view method to add the third surface view to the subview of the subclass.

9. The method according to claim 1, wherein, The first rendering unit is the window to be rendered by the first process, and the second surface is the surface of the window; The first process receives surface control information of a first surface view created by the service process from the service process, including: the first process receiving a surface encapsulation of a first surface control information view holder from the service process, wherein the first surface view is a surface view in the view tree of the first surface control information view holder, and the surface encapsulation contains the surface control information of the first surface view; and The first process uses the surface control information of the first surface view to associate the first surface of the first surface view with the second surface created by the first rendering unit, including: The first process obtains the surface control information of the window and the surface control information in the surface encapsulation, and The first process uses a method in the surface control transaction to reparent the surface control information in the surface encapsulation to the surface control information of the window, thereby setting the first surface as a child node of the window's surface.

10. The method according to claim 9, wherein, The first process is the process corresponding to the wallpaper service, and The first process obtains the surface control information of the window and the surface control information in the surface encapsulation, including: The first process inherits the engine class of the wallpaper service and creates an instance of the engine. In the surface creation callback of the engine instance, the first process obtains surface control member variables via reflection as the surface control information of the window, and The method of obtaining surface control information by first-process reflection is used to obtain surface control information in the surface package.

11. The method according to claim 1, further comprising: In response to detecting that the first process has terminated abnormally, the service process terminates the association between the first surface and the second surface; In response to detecting that the first process has restarted, the service process retransmits the surface control information of the first surface view to the first process, so that the first process uses the surface control information of the first surface view to associate the first surface with the second surface recreated by the first rendering unit.

12. A computing device, comprising: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-11.

13. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-11.

Images