Rendering method, engine, electronic device, and storage medium

Abstract

The application provides a rendering method, an engine, an electronic device and a storage medium, the method comprises the following steps: obtaining a plurality of to-be-drawn data; applying a window based on a drawing request generated according to each to-be-drawn data; applying a layer interface of each to-be-drawn data; creating a frame buffer according to each drawing request; performing three-dimensional rendering on the frame buffer; and obtaining display content of each layer interface. The method can be adapted to an android platform, can realize the effect of fusion rendering across processes, and meets the user demand.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a rendering method, engine, electronic device, and storage medium. Background Technology

[0002] With the development of intelligent connected vehicles, in-vehicle infotainment systems based on the Android operating system are becoming increasingly popular. Android's support for real-time 3D rendering is very basic. Currently, Android uses two rendering methods: one is Skia (a 2D vector graphics library) from Google. As the official graphics library for Android, Skia is very compatible with the Android system, but it is only a 2D vector graphics library; the other is using the cross-platform graphics API OpenGL ES (specificating embedded devices). However, OpenGL's API is too primitive and basic, resulting in very low development efficiency.

[0003] While it's possible to use mature commercial engines for 3D development, such as Unity (based on the GNOME desktop environment) or Unreal Engine (a cross-platform rendering engine), and then separately adapt and package them for the Android platform, in reality, commercial engines need to support cross-operating system platform compatibility and adaptation, as well as meet the engine requirements of each platform. Therefore, they often suffer from insufficient adaptation to the specific Android platform, and none of them can achieve cross-process fusion rendering effects, failing to meet user needs. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the present invention provides a rendering method, engine, electronic device and storage medium to solve the above technical problems.

[0005] This invention provides a rendering engine that runs in the system runtime library layer of the Android system. The rendering engine includes:

[0006] The interface layer includes multiple drawing interfaces for acquiring multiple data to be drawn, wherein the multiple data to be drawn includes at least one of the following: data to be drawn from multiple applications, or multiple data to be drawn from one application.

[0007] The client is used to generate drawing requests based on the data to be drawn.

[0008] The core layer is used to request a window based on the drawing request, and to request layer interfaces for each of the data to be drawn, and to create a frame buffer according to each drawing request;

[0009] The platform layer is used to perform 3D rendering on the frame buffer to obtain the display content of each layer interface.

[0010] In one embodiment of the present invention, the core layer includes a rendering pipeline, which includes a management thread and a rendering thread, wherein:

[0011] The management thread is used to generate rendering instructions based on the frame buffer, and store the rendering instructions in a preset circular queue for the rendering thread to retrieve.

[0012] The rendering thread is used to detect the presence of the rendering instructions queued in the preset circular queue, obtain the rendering instructions, and perform 3D rendering by calling the platform layer to access the frame buffer of the rendering instructions.

[0013] In one embodiment of the present invention, the management thread includes at least one of the following:

[0014] A framework diagram example, used to create a rendering layer based on preset business requirements;

[0015] The culling module is used to monitor rendering operations and optimize the drawing logic;

[0016] A command generation module is used to generate rendering instructions based on the frame buffer in response to the next frame preparation message of the rendering thread, wherein the frame buffer is the next frame;

[0017] A pre-defined circular queue is used to store the rendering instructions for the rendering thread to retrieve.

[0018] The notification module is used to generate notification messages and send them to the rendering thread. The notification messages include notifications for switching rendering contexts, stopping the current frame drawing, and preparing to draw the next frame.

[0019] In one embodiment of the present invention, the rendering thread includes at least one of the following:

[0020] The resource preparation module is used to switch the rendering context, stop drawing the current frame, and prepare the rendering resources of the frame buffer in response to the notification message.

[0021] The monitoring module is used to monitor the rendering commands in the preset circular queue;

[0022] The instruction acquisition module is used to acquire the rendering instruction and perform 3D rendering by calling the frame buffer of the rendering instruction in the platform layer;

[0023] The completion prompt module is used to send a next frame preparation message to the command generation module to trigger the command generation module to generate rendering instructions for the next frame of the frame buffer.

[0024] In one embodiment of the present invention, the core layer further includes a display control module. If the display device is a multi-screen interactive device including multiple displays, the display control module is used to determine the current display screen of each display screen according to the acquired memory descriptor and the display attributes of the multi-screen interactive device. The current display screen includes at least a portion of the display content of at least one layer interface.

[0025] In one embodiment of the present invention, the display control module is further configured to:

[0026] In response to a display screen movement command, the current display screen of each of the aforementioned displays is moved so that at least one of the aforementioned displays displays the display content of at least two layer interfaces. The display screen movement command includes movement object start information, movement distance information, and movement direction information.

[0027] In one embodiment of the present invention, if the display device includes a display screen, the display content of each layer interface is displayed by directly sending the memory descriptor to the screen.

[0028] In one embodiment of the present invention, the platform layer includes at least one of the following:

[0029] The rendering module is used to perform 3D rendering on the frame buffer to obtain the display content of each layer interface;

[0030] The display content cache space is used to store the display content of each layer interface, wherein the display content of each layer interface is stored in a contiguous shared memory block in the display content cache space;

[0031] The memory descriptor sending module is used to send the memory descriptor of the displayed content to the core layer.

[0032] This invention also provides a rendering method applied to the system runtime library layer of an Android system, the rendering method comprising:

[0033] Acquire multiple data to be drawn, wherein the multiple data to be drawn includes at least one of the following: multiple data to be drawn from multiple applications, or multiple data to be drawn from one application;

[0034] A drawing request is generated based on the data to be drawn;

[0035] Based on the drawing request, a window is requested, and layer interfaces for each of the data to be drawn are requested. A frame buffer is created according to each drawing request.

[0036] The frame buffer is rendered in 3D to obtain the display content of each layer interface.

[0037] In one embodiment of the present invention, before performing 3D rendering on the frame buffer, the rendering method further includes:

[0038] In response to the acquired next frame preparation message, a rendering instruction is generated based on the frame buffer, and the rendering instruction is stored in a preset circular queue for the rendering thread to retrieve;

[0039] Send notification messages, including a rendering thread switching rendering context notification, a stop drawing of the current frame notification, and a preparation for drawing the next frame notification;

[0040] In response to the notification message, switch the rendering context, stop drawing the current frame, and prepare the rendering resources for the frame buffer;

[0041] The system listens to rendering instructions in the preset circular queue, detects that there are rendering instructions queued in the preset circular queue, obtains the rendering instructions, and triggers 3D rendering of the frame buffer of the rendering instructions.

[0042] In one embodiment of the present invention, after obtaining the rendering instruction, the rendering method further includes:

[0043] The prepared frame buffer rendering resources are assembled into the preset resource cache space;

[0044] Activate the shading module and set the display content cache space to trigger 3D rendering of the frame cache of the rendering command;

[0045] If the 3D rendering is complete, a next frame preparation message is sent to trigger the rendering instruction for generating the next frame of the frame buffer.

[0046] In one embodiment of the present invention, the plurality of data to be drawn includes data to be drawn from a plurality of applications, and before performing 3D rendering on the frame buffer, the rendering method further includes:

[0047] Each application requests a temporary window and a temporary layer interface to generate a layer instance for the transition animation;

[0048] Animation instances are generated based on the coating instances of each layer interface, and each animation instance is bound to the scene layer to realize the animation effects of 3D rendering of the frame buffer.

[0049] In one embodiment of the present invention, after applying for temporary layer interfaces, the rendering method further includes releasing each of the temporary windows.

[0050] The present invention provides an electronic device, the electronic device comprising:

[0051] One or more processors;

[0052] A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to implement the rendering method as described in any of the preceding embodiments.

[0053] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform the rendering method described in any of the above embodiments.

[0054] The beneficial effects of the present invention are as follows: The rendering method, engine, electronic device and storage medium of the present invention acquire multiple data to be drawn, generate drawing requests based on each data to be drawn, apply for a window based on the drawing requests, apply for layer interfaces for each data to be drawn, create a frame buffer based on each drawing request, perform 3D rendering on the frame buffer, and obtain the display content of each layer interface. It can be adapted to the Android platform and can achieve the effect of cross-process (multiple applications) fusion rendering, thus meeting user needs.

[0055] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0057] Figure 1 This is a schematic diagram illustrating the implementation environment of a rendering system as shown in an exemplary embodiment of this application;

[0058] Figure 2 This is a block diagram illustrating a rendering engine in an exemplary embodiment of this application;

[0059] Figure 3 This is a flowchart illustrating a rendering pipeline in an exemplary embodiment of this application;

[0060] Figure 4 This is a flowchart illustrating a multi-screen rendering function as an exemplary embodiment of this application;

[0061] Figure 5 This is an exemplary embodiment of the present application illustrating a cross-process animation processing flowchart;

[0062] Figure 6 This is an exemplary embodiment of the present application illustrating the architecture of a specific rendering engine;

[0063] Figure 7 This is a flowchart illustrating a rendering method in an exemplary embodiment of this application;

[0064] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0065] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0066] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0067] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0068] Figure 1 This is a schematic diagram illustrating the implementation environment of a rendering system as shown in an exemplary embodiment of this application. Figure 1 As shown, the rendering system 100 includes multiple applications (APP1, APP2, ..., APPN), a rendering engine, and a display device. When one or more applications need to be rendered, the rendering engine obtains the data to be drawn from each application, renders the data required by each application, and outputs the values ​​to the display device for display. The applications can be various applications known to those skilled in the art, and the display device can include one or more displays.

[0069] With the development of intelligent connected vehicles, in-vehicle infotainment systems based on the Android operating system are becoming increasingly popular. Android's support for real-time 3D rendering is very basic. Currently, Android uses two rendering methods: one is Skia, released by Google. As the official graphics library for Android, Skia is very compatible with Android, but it is only a 2D vector graphics processing library; the other is using the cross-platform graphics API OpenGL ES. However, OpenGL's API is too primitive and basic, resulting in very low development efficiency.

[0070] While it's possible to use mature commercial engines for 3D development, such as cross-platform rendering engines like Unity or Unreal Engine, and then adapt and package them separately for the Android platform, the reality is that commercial engines need to support cross-operating system platform compatibility and adaptation, as well as meet the engine requirements of various platforms. Therefore, they often suffer from insufficient adaptation to specific Android platforms, and none of them can achieve cross-process fusion rendering effects, failing to meet user needs. Furthermore, all rendering engines on the market are integrated using NDK packaging, which is insufficient to meet the highly customized rendering needs of car manufacturers, such as cross-application animation effects and multi-screen rendering.

[0071] To address these issues, embodiments of this application propose a rendering method, a rendering engine, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail below.

[0072] Please see Figure 2 , Figure 2 This is a block diagram illustrating a rendering engine in an exemplary embodiment of this application. This rendering engine can be applied to... Figure 1 The implementation environment is shown, and the method is specifically executed by the rendering engine in that implementation environment. It should be understood that this method can also be applied to other exemplary implementation environments and executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0073] like Figure 2 As shown, the exemplary rendering engine 200 includes:

[0074] Interface layer 201 includes multiple drawing interfaces for obtaining multiple data to be drawn. The multiple data to be drawn includes at least one of the following: data to be drawn from multiple applications, or multiple data to be drawn from one application.

[0075] Client 202 is used to generate drawing requests based on the data to be drawn;

[0076] Core layer 203 is used to request a window based on a drawing request, as well as to request layer interfaces for each data to be drawn, and to create a frame buffer based on each drawing request;

[0077] Platform layer 204 is used for 3D rendering of the frame buffer to obtain the display content of each layer interface.

[0078] The interface layer obtains the data to be drawn from each application and / or multiple data points from a single application through various provided drawing interfaces. For example, if two applications have rendering requirements, the interface layer will obtain the data to be drawn from both applications. As another example, if an application has two rendering requirements, such as a map application that needs to display a map page on one hand and add ground information such as gas stations and weather markers on the other, rendering can be performed based on the two data points from the same application.

[0079] In one embodiment, the rendering engine can be adapted to a GPU platform using OpenGL ES 3.x.

[0080] In one embodiment, the data to be drawn includes, but is not limited to, at least one of camera data (primary view orientation data), renderable data (position data, color data), ransform data (interaction data), light data (lifecycle), skybox data (skybox, data stream), and surface instances to be drawn (window instances, animation instances).

[0081] The client can generate drawing requests based on the data to be drawn in a manner known to those skilled in the art. The drawing requests include, but are not limited to, at least one of window data, drawing data, interactive data, and lifecycle data.

[0082] In this embodiment, the rendering engine requests only one window Surface for the data to be drawn from multiple applications, and requests multiple layer interfaces based on this window. In this way, even if an application X is not an upper-level application, the 3D animation of application X can still be displayed on the display panel, or the 3D animation of application X can be displayed on any other application interface.

[0083] In one embodiment, the core layer includes a rendering pipeline, which includes a management thread and a rendering thread, wherein:

[0084] The management thread is used to generate rendering instructions based on the frame buffer and store the rendering instructions in a preset circular queue for the rendering thread to retrieve.

[0085] The rendering thread is used to detect rendering commands queued in a preset circular queue, obtain the rendering commands, and perform 3D rendering by calling the platform layer to buffer the rendering commands.

[0086] By splitting the rendering pipeline into management threads and rendering threads, and adopting the ECS (Entity Component System) design philosophy to separate resources and logic, rendering efficiency can be improved. The core layer also includes a thread management module. This module dynamically configures the number of threads for different hardware, especially on multi-core hardware where work-stealing is used for parallel task scheduling, greatly enhancing rendering efficiency.

[0087] By using a pre-defined circular queue to store rendering commands, the frequent use of locks can be reduced, thus improving rendering execution efficiency.

[0088] In one embodiment, the management thread includes at least one of the following:

[0089] A framework diagram example, used to create a rendering layer based on preset business requirements;

[0090] The culling module is used to monitor rendering operations and optimize the drawing logic;

[0091] The command generation module is used to generate rendering instructions based on the frame buffer in response to the next frame preparation message of the rendering thread, with the frame buffer being the next frame;

[0092] A pre-defined circular queue is used to store rendering commands for the rendering thread to retrieve.

[0093] The notification module is used to generate notification messages and send them to the rendering thread. Notification messages include notifications for switching rendering contexts, stopping the current frame drawing, and preparing for the next frame drawing.

[0094] The management thread, which can be considered the main thread, starts two threads (the management thread and the rendering thread) during rendering engine initialization. These threads perform a rendering loop according to a dynamically set frame rate (preset frame rate). When the client management layer receives a new request (a new drawing request), the first thread (the main thread) starts executing from the update step and loops until the request is stopped or encounters an error. Before starting to draw a frame, the main thread performs some preparatory work, such as notifying the context (rendering context) to switch, and notifying the driver (rendering execution end) to stop drawing the previous frame and release the corresponding resources to prepare for drawing the current frame. The second thread (the rendering thread) is the actual rendering thread. Upon receiving the notification message from the main thread, it begins preparatory work, including retrieving the corresponding target (rendering instruction) from the rendering buffer pool, switching the context, setting window-related parameters, and preparing resources for the next frame; or stopping drawing the current frame and releasing the current frame's resources.

[0095] In one embodiment, a FrameGraph instance creates multiple rendering passes (rendering layers) according to pre-defined business requirements, such as setting MSAA (MultiSampling Anti-Aliasing), and then defines input or output resource data structures for each rendering pass, including resources such as textures and targets.

[0096] Optimizing the drawing logic includes, but is not limited to, optimizing the drawing logic itself, such as invalid culling.

[0097] In one embodiment, the rendering thread includes at least one of the following:

[0098] The resource preparation module is used to switch rendering contexts, stop drawing the current frame, and prepare rendering resources for the frame buffer in response to notification messages.

[0099] The listening module is used to listen for rendering commands in a preset circular queue.

[0100] The instruction acquisition module is used to acquire rendering instructions and perform 3D rendering by calling the frame buffer of the rendering instructions in the platform layer.

[0101] The completion prompt module is used to send a next frame preparation message to the command generation module, so as to trigger the command generation module to generate the rendering instructions for the next frame of the frame buffer.

[0102] Here, the current frame is the frame X currently being rendered by the rendering module, the next frame is the frame buffer M, and the relationship between the next frame Y of the frame buffer is X < M < Y. In other words, X is the frame before M, and Y is the frame after M.

[0103] In one embodiment, the platform layer includes an OpenGL interface. 3D rendering can be performed by calling the frame buffer of the rendering instructions in the platform layer, which can also be done by calling the corresponding OpenGL interface.

[0104] Please see Figure 3 , Figure 3 This is a flowchart illustrating a rendering pipeline in an exemplary embodiment of this application. Taking the rendering process of one frame as an example, as follows... Figure 3 As shown, the process of this exemplary rendering pipeline includes:

[0105] During rendering engine initialization, two threads (the management thread and the rendering thread) are started to perform a rendering loop according to a dynamically set frame rate. When the client management layer receives a new request (drawing request), the first thread (the main thread, the management thread) will loop from update (start rendering) until the request is stopped or an exception occurs (end rendering). Before the main thread starts drawing a frame, it performs some preparatory work, such as switching the context, notifying the driver to stop drawing the previous frame and release the corresponding resources to prepare for drawing the current frame (notifying the driver to prepare to receive rendering commands). The second thread (the rendering thread) is the actual rendering thread. Upon receiving notification from the main thread, it begins preparatory work, including obtaining the corresponding target from the rendering buffer pool, switching the context, setting window-related parameters, preparing resources for the next frame, etc.; or stopping drawing the current frame and releasing the current frame's resources, etc.

[0106] During the rendering phase, designed according to ECS principles, a FrameGraph instance is first created. Then, multiple rendering passes are created based on business requirements (adding renderpasses), such as setting MSAA (msaa). Next, input or output resource data structures are defined for each rendering pass, including textures, targets, and other resources. Operations on resources and rendering operations are monitored (compile), while the drawing logic is optimized, such as invalid culling. Finally, commands are generated (execute) and sent to the commandstream circular queue (Command Queue) to await acquisition by the rendering thread (Driver). The circular queue is primarily used to reduce the frequent use of locks and improve rendering execution efficiency.

[0107] When the rendering thread detects an instruction queue in the circular queue, it immediately retrieves the instruction, loads the resources into the corresponding buffer (loading the prepared rendering resources of the frame buffer into the preset resource buffer space), activates the shader program (shading module), sets the corresponding buffer (display content buffer space), and then calls the corresponding OpenGL interface for rendering. Once the rendering drawcall is submitted, it notifies the main thread to prepare the command for the next frame (sending a next frame preparation message). The rendering driver calls Platform, VertexBuffer, IndexBuffer, Primitive, Program, Texture, Render Target, Fence, and Window. These modules organize data according to the rendering requirements of the corresponding graphics library (OpenGL in this engine), set the shaders, import the corresponding textures, set the rendering target, and set the window before finally submitting it to the GPU for rendering. The Platform module is mainly responsible for shielding various graphics interface calls, such as OpenGL and Vulkan. The VertexBuffer and IndexBuffer are mainly responsible for organizing and storing vertex data blocks. The Primitive type is mainly used to abstract the drawing and setting of points, lines, and triangles. The Program is the carrier program for the shader program, executed on the GPU. It primarily manages attached shaders and handles data transfer between the CPU and GPU. The Texture module is mainly responsible for managing texture information and organizing texture data. The Render Target is primarily used to store information such as base color, normals, and ambient occlusion. The Fence is mainly used for synchronous rendering operations. The Window is primarily used to shield the different usage methods of operating system windows.

[0108] Once the main thread receives the completion message, it releases resources whose lifecycles have ended, marks the end, and notifies the Driver to terminate. This concludes the rendering process for one frame, and the loop begins executing the rendering process for the next frame.

[0109] In terms of management, this rendering pipeline adopts the ECS design philosophy to separate resources and logic. Regarding efficiency, it constructs a framegraph to dynamically optimize the logical links, ensuring that each frame only executes the necessary logic during actual execution. Furthermore, it relies on the core layer's thread management module to dynamically configure the number of threads for different hardware, especially using work-stealing for parallel task scheduling on multi-core hardware, greatly improving rendering efficiency.

[0110] In one embodiment, the core layer further includes a display control module. If the display device is a multi-screen interactive device including multiple displays, the display control module is used to determine the current display screen of each display screen based on the acquired memory descriptor and the display attributes of the multi-screen interactive device. The current display screen includes at least a portion of the display content of at least one layer interface.

[0111] The memory descriptor (fd) identifies the storage address of the displayed content. Since different hardware platforms implement multi-screen interaction differently, the display control module itself will also adapt. If the hardware platform does not support multi-screen interaction, the display control module will pass the fd to the screen driver for direct screen display; if the hardware platform supports multi-screen interaction, the display control module will adapt to the platform's solution. This means that the content rendered in the above embodiment will be displayed in a split-screen manner.

[0112] In one embodiment, the display control module is further configured to:

[0113] In response to a display screen movement command, the current display screen of each display screen is moved so that at least one display screen displays the display content of at least two layer interfaces. The display screen movement command includes the starting information of the moving object, the movement distance information, and the movement direction information.

[0114] The display screen movement command can be implemented by the user interacting with the rendering engine through voice, gestures, etc., so that the rendering engine knows the starting information, movement distance information, and movement direction information of the moving object, and then determines the starting movement point. Based on the movement distance information and movement direction information, the current display screen is moved. During the movement, the display content obtained from one set of data to be drawn can be directly displayed on a display screen, or the movement process of the display content can be displayed, and a portion of the display content of at least two sets of data to be drawn can be displayed on a display screen. That is, the display screen displayed on a certain display screen is a portion of the content of two sets of display content.

[0115] In one embodiment, if the display device includes a single display screen, the display content of each layer interface is displayed by directly sending the acquired memory descriptor to the screen. In this case, the display content can be displayed using methods known to those skilled in the art.

[0116] Using the rendering engine provided in this embodiment, multi-screen rendering functionality can be achieved. Please refer to [link / reference]. Figure 4 , Figure 4 This is a flowchart illustrating a multi-screen rendering function in an exemplary embodiment of this application. Taking two displays as the display devices and the data to be rendered from two applications (APP1 and APP2) as an example, as follows... Figure 4 As shown, this exemplary multi-screen rendering function includes:

[0117] Multi-screen rendering involves displaying different rendered content (the display content of each piece of data to be drawn) on different screens. For example, in map rendering, it's necessary to ensure that navigation information displayed on the main screen is not obstructed, while on the secondary screen, users can add waypoints or display other information. To implement this functionality, the 3D data to be rendered for both the main and secondary screens needs to be prepared. Two Layer instances (one canvas) are generated using ClientManagerService, and a framebuffer (display content cache space) is created. Offline rendering is then performed via the rendering pipeline. The difference this time is that the content is directly rendered to two contiguous shared memory segments, storing the display content of Layer 1 and Layer 2 respectively. Then, the memory descriptor (fd) is returned, and the fd is passed to the Display module (display control module). The main responsibility of this display control module is to handle multi-screen interaction. Since the implementation schemes for multi-screen interaction differ across hardware platforms, this module itself also needs to be adapted. If the hardware platform does not support multi-screen interaction, then Display will pass the file descriptor (fd) to the screen driver and implement it by directly sending the screen; if the hardware platform supports multi-screen interaction, then Display will adapt to the solution of that platform.

[0118] In one embodiment, the platform layer includes at least one of the following:

[0119] The rendering module is used to perform 3D rendering on the frame buffer to obtain the display content of each layer interface. The rendering module can be an OpenGL interface or other modules known to those skilled in the art.

[0120] The display content cache space is used to store the display content of each layer interface. The display content of each layer interface is stored in a contiguous shared memory block in the display content cache space.

[0121] The memory descriptor sending module is used to send memory descriptors of the displayed content to the core layer.

[0122] The platform layer may also include other third-party libraries, EGL (native platform interface), and other objects that those skilled in the art may add as needed.

[0123] The platform layer needs to be related to the operating system and hardware platform to support cross-platform capabilities as well as the hardware capabilities of the Android system on a specific platform.

[0124] Please see Figure 5 , Figure 5 This is an exemplary embodiment of the cross-process animation processing flowchart illustrating this application. Taking the processes of two applications (APP1 and APP2) as an example, as follows... Figure 5As shown, this exemplary cross-process animation processing includes:

[0125] Figure 5 The process shown is located in the client-side management layer of the entire rendering engine. This section mainly describes the processing flow of cross-process animation.

[0126] Taking multiple data sets to be drawn, including data from multiple applications, as an example, before 3D rendering of the frame buffer, each application (APP1 and APP2) sends the starting and ending display Surface instances of the transition animation to the ClientManager. The SurfaceManager requests temporary windows (Surface1 and Surface2) and temporary layer interfaces (Layer1 and Layer2). After receiving these, the SurfaceManager generates two Layer instances (Layer1 and Layer2) through the Layer module. Then, it passes the animation parameters of Layer1 and Layer2 to the Animation module to generate corresponding animation instances (generating the layer instances of the transition animation), and then sends them together to the Layer module and binds them to the Scene module. Finally, the drawing content data (RenderData and Transfer) passed by the application (APP) is imported into the rendering pipeline for rendering to achieve the animation effect of 3D rendering of the frame buffer (data to be drawn). When generating the Layer, at least one of the original temporary windows Surface1 and Surface2 can be released, so when APP1 is closed, the transition animation process is completely unaffected by the application's lifecycle.

[0127] For example, when implementing weather effects on Android, they are usually implemented only within the application. However, in the automotive industry, it is often required that weather effects appear in any application interface. Because the Android operating system is process-isolated, effects within an application cannot escape its own lifecycle. To achieve this type of cross-process animation, it is necessary to rely on Android's native WindowManager or TransitionManager. However, there is currently no ready-made solution to achieve 3D animation effects in this way. Since the rendering engine provided in this embodiment runs in the native layer of the Android operating system, it can bypass the Android framework and directly request a Surface (a new canvas (window), different from Surface1 and Surface2 in the previous embodiment) from SurfaceFlinger. This Surface can be defined with the highest display layer. Then, the Surface is passed to the Layer module to request a Layer. Then, based on the drawing data and animation requirements provided by the upper-layer application, a specific animation instance is requested from the Animation module. Finally, the drawing data and animation data are bound to the scene module, and real-time 3D rendering is performed through the rendering pipeline. Since the display layer is not affected by the upper-layer application, a global 3D weather animation can be achieved.

[0128] For example, to provide another highly customized animation experience, there is currently no readily available solution for 3D transition effects in a specified area between two independent applications. However, the rendering engine provided in this embodiment can achieve this in the following way:

[0129] First, the two applications (APP) send the starting and ending display Surface instances of the transition animation to the ClientManager, respectively. The SurfaceManager receives these and generates two Layer instances using the Layer module. Then, it passes the animation parameters of Layer1 and Layer2 to the Animation module to generate corresponding animation instances, which are then sent to the Layer module and bound to the scene module. Finally, the drawing data passed from the APP is imported into the rendering pipeline for rendering. Since the original Surface can be released when the Layer is generated, the transition animation process is completely unaffected by the application's lifecycle when APP1 is closed.

[0130] Please see Figure 6 , Figure 6 This is an exemplary embodiment of the present application illustrating the architecture of a specific rendering engine. For example... Figure 6As shown, the architecture of this exemplary rendering engine is mainly divided into four layers, from top to bottom: the interface layer, the client management layer, the core layer, and the platform layer. Its deployment method differs from traditional engines; this engine runs as an independent process in the Android native layer and interacts with the application that needs to be rendered through a cross-process approach.

[0131] The interface layer is mainly responsible for providing the drawing interface. The application organizes and provides Camera data (camera angle, human eye, orientation of the main viewpoint), Renderable data (position and color), Transform data (interaction), Light data (lifecycle), Skybox data (skybox, individual images, data stream), and Surface instances (canvas, windows) that need to be drawn.

[0132] The ClientManager, the client-side management layer, is primarily responsible for the unified management of drawing requests from different applications, including window data, drawing data, interaction data, and lifecycle.

[0133] The core layer primarily focuses on the rendering process, including traditional scene management (Scene), layer management (Layer), particle system (Partical), rendering pipeline (Pipline), animation system (Animation), asset management (Resource), display control module (Display), memory management, surface management, thread management, event control, audio, and file management (FILE IO).

[0134] The platform layer is primarily related to the operating system and hardware platform, supporting cross-platform capabilities as well as the hardware capabilities of the Android system specific to that platform. It can include OpenGL ES, shared memory, EGL, and other third-party libraries.

[0135] The rendering engine provided in this embodiment runs in the system runtime library layer (native layer) of the Android system. It includes an interface layer for obtaining multiple data to be drawn, a client for generating drawing requests based on each data to be drawn, a window for requesting a drawing request, a layer interface for requesting each data to be drawn, a core layer for creating a frame buffer based on each drawing request, and a platform layer for performing 3D rendering on the frame buffer to obtain the display content of each layer interface. It can be adapted to the Android platform and can achieve cross-process fusion rendering effect to meet user needs.

[0136] Meanwhile, this Android-based rendering engine can improve the efficiency of developing real-time 3D rendering applications for Android, while also supporting the highly customized rendering requirements of the Android system. It can be used by Android developers to develop high-quality 3D rendering applications and is compatible with OpenGL ES 3.x GPU platforms.

[0137] The rendering engine provided in this embodiment can effectively improve development efficiency, allowing developers to focus only on the functionality of the application itself, instead of using the original Android system-supported APIs to develop 3D real-time rendering applications or those large commercial rendering engines.

[0138] It can support highly customized functions for different applications, such as cross-application animation effects, multi-screen rendering, and other customized functions that other rendering engines cannot achieve. At the same time, with the support of specific Android platform hardware, it can achieve greater improvement in running efficiency.

[0139] Please see Figure 7 , Figure 7 This is a flowchart illustrating a rendering method in an exemplary embodiment of this application. This method can be applied to... Figure 1 The implementation environment is shown, and the method is specifically executed by the rendering engine in that implementation environment. It should be understood that this method can also be applied to other exemplary implementation environments and executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0140] like Figure 7 As shown, in an exemplary embodiment, the service activation management method includes at least steps S701 to S704, which are described in detail below:

[0141] Step S701: Obtain multiple data to be drawn, wherein the multiple data to be drawn includes at least one of the following: multiple application data to be drawn, or multiple application data to be drawn.

[0142] Step S702: Draw requests generated based on the data to be drawn.

[0143] Step S703: Request a window based on the drawing request, and request layer interfaces for each data to be drawn, and create a frame buffer according to each drawing request.

[0144] Step S704: Perform 3D rendering on the frame buffer to obtain the display content of each layer interface.

[0145] In one embodiment, before performing 3D rendering on the frame buffer, the rendering method further includes:

[0146] In response to the acquired next frame preparation message, a rendering instruction is generated based on the frame buffer and stored in a preset circular queue for the rendering thread to retrieve.

[0147] Send notification messages, including notifications for switching rendering contexts on the rendering thread, stopping the current frame of drawing, and preparing to draw the next frame;

[0148] In response to notification messages, switch rendering contexts, stop drawing the current frame, and prepare rendering resources for the frame buffer;

[0149] Listen for rendering commands in a preset circular queue. If a rendering command is detected in the preset circular queue, the rendering command is retrieved to trigger 3D rendering of the frame buffer of the rendering command.

[0150] In one embodiment, after obtaining the rendering instructions, the rendering method further includes:

[0151] Mount the prepared framebuffer rendering resources into the preset resource cache space;

[0152] Activate the shading module and set the display content cache space to trigger 3D rendering of the frame buffer for rendering instructions;

[0153] If the 3D rendering is complete, send a next frame preparation message to trigger the rendering instruction for the next frame of the generated frame buffer.

[0154] In one embodiment, the multiple data to be drawn includes data to be drawn from multiple applications. Before performing 3D rendering on the frame buffer, the rendering method further includes:

[0155] Each application requests a temporary window and a temporary layer interface to generate layer instances for the transition animation;

[0156] Animation instances are generated based on the coating instances of each layer interface, and each animation instance is bound to the scene layer to achieve 3D rendering animation effects through frame caching.

[0157] In one embodiment, after requesting temporary layer interfaces respectively, the rendering method further includes releasing each temporary window.

[0158] It should be noted that the rendering method provided in the above embodiments is different from that in the above embodiments. Figures 2-6 The provided rendering engines belong to the same concept, and the specific methods of each step have been described in detail in the rendering engine embodiments, and will not be repeated here. In practical applications, the rendering method provided in the above embodiments can be assigned to different functional modules as needed, that is, the steps of the rendering method can be divided into different functional modules according to the internal structure of the rendering engine to complete all or part of the functions described above, and this is not a limitation.

[0159] The rendering method for the system runtime library layer of the Android system provided in this embodiment obtains multiple data to be drawn, generates drawing requests based on each data to be drawn, applies for a window based on the drawing requests, applies for layer interfaces for each data to be drawn, creates a frame buffer based on each drawing request, performs 3D rendering on the frame buffer, and obtains the display content of each layer interface. It can be adapted to the Android platform and can achieve the effect of cross-process fusion rendering, meeting user needs.

[0160] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the rendering methods provided in the above embodiments.

[0161] Figure 8 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 8 The computer system 800 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0162] like Figure 8 As shown, the computer system 800 includes a Central Processing Unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 802 or programs loaded from storage portion 808 into Random Access Memory (RAM) 803, such as performing the methods described in the above embodiments. The RAM 803 also stores various programs and data required for system operation. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An Input / Output (I / O) interface 805 is also connected to the bus 804.

[0163] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0164] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs various functions defined in the system of this application.

[0165] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0166] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0167] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0168] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the rendering method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0169] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the rendering methods provided in the various embodiments described above.

[0170] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A rendering engine, characterized in that, The rendering engine runs in the system runtime library layer of the Android system, and the rendering engine includes: The interface layer includes multiple drawing interfaces for obtaining multiple data to be drawn, which include: data to be drawn from multiple applications, or multiple data to be drawn from two applications. The client is used to generate drawing requests based on the data to be drawn. The core layer is used to request a window from SurfaceFlinger based on the drawing request, and to request layer interfaces for each of the data to be drawn based on the window, and to create a frame buffer according to each drawing request; The platform layer is used to perform 3D rendering on the frame buffer to obtain the display content of each layer interface.

2. The rendering engine as described in claim 1, characterized in that, The core layer includes a rendering pipeline, which comprises a management thread and a rendering thread, wherein: The management thread is used to generate rendering instructions based on the frame buffer, and store the rendering instructions in a preset circular queue for the rendering thread to retrieve. The rendering thread is used to detect the presence of the rendering instructions queued in the preset circular queue, obtain the rendering instructions, and perform 3D rendering by calling the platform layer to access the frame buffer of the rendering instructions.

3. The rendering engine as described in claim 2, characterized in that, The management thread includes at least one of the following: A framework diagram example, used to create a rendering layer based on preset business requirements; The culling module is used to monitor rendering operations and optimize the drawing logic; A command generation module is used to generate rendering instructions based on the frame buffer in response to the next frame preparation message of the rendering thread, wherein the frame buffer is the next frame; A pre-defined circular queue is used to store the rendering instructions for the rendering thread to retrieve. The notification module is used to generate notification messages and send them to the rendering thread. The notification messages include notifications for switching rendering contexts, stopping the current frame drawing, and preparing to draw the next frame.

4. The rendering engine as described in claim 3, characterized in that, The rendering thread includes at least one of the following: The resource preparation module is used to switch the rendering context, stop drawing the current frame, and prepare the rendering resources of the frame buffer in response to the notification message. The monitoring module is used to monitor the rendering commands in the preset circular queue; The instruction acquisition module is used to acquire the rendering instruction and perform 3D rendering by calling the frame buffer of the rendering instruction in the platform layer; The completion prompt module is used to send a next frame preparation message to the command generation module to trigger the command generation module to generate rendering instructions for the next frame of the frame buffer.

5. The rendering engine as described in any one of claims 1-4, characterized in that, The core layer also includes a display control module. If the display device is a multi-screen interactive device including multiple displays, the display control module is used to determine the current display screen of each display screen according to the acquired memory descriptor and the display attributes of the multi-screen interactive device. The current display screen includes at least a portion of the display content of at least one layer interface.

6. The rendering engine as described in claim 5, characterized in that, The display control module is also used for: In response to a display screen movement command, the current display screen of each of the aforementioned displays is moved so that at least one of the aforementioned displays displays the display content of at least two layer interfaces. The display screen movement command includes movement object start information, movement distance information, and movement direction information.

7. The rendering engine as described in claim 5, characterized in that, If the display device includes a single display screen, the display content of each layer interface is displayed by directly sending the memory descriptor to the screen.

8. The rendering engine as described in any one of claims 1-4, characterized in that, The platform layer includes at least one of the following: The rendering module is used to perform 3D rendering on the frame buffer to obtain the display content of each layer interface; The display content cache space is used to store the display content of each layer interface, wherein the display content of each layer interface is stored in a contiguous shared memory block in the display content cache space; The memory descriptor sending module is used to send the memory descriptor of the displayed content to the core layer.

9. A rendering method, characterized in that, The rendering method is applied to the system runtime library layer of the Android system, and the rendering method includes: Acquire multiple data to be drawn, including: data to be drawn from multiple applications, or multiple data to be drawn from two applications; A drawing request is generated based on the data to be drawn; Based on the drawing request, a window is requested from SurfaceFlinger, and layer interfaces for each of the data to be drawn are requested based on the window. A frame buffer is created according to each drawing request. The frame buffer is rendered in 3D to obtain the display content of each layer interface.

10. The rendering method as described in claim 9, characterized in that, Before performing 3D rendering on the frame buffer, the rendering method further includes: In response to the acquired next frame preparation message, a rendering instruction is generated based on the frame buffer, and the rendering instruction is stored in a preset circular queue for the rendering thread to retrieve; Send notification messages, including a rendering thread switching rendering context notification, a stop drawing of the current frame notification, and a preparation for drawing the next frame notification; In response to the notification message, switch the rendering context, stop drawing the current frame, and prepare the rendering resources for the frame buffer; The system listens to rendering instructions in the preset circular queue, detects that there are rendering instructions queued in the preset circular queue, obtains the rendering instructions, and triggers 3D rendering of the frame buffer of the rendering instructions.

11. The rendering method as described in claim 10, characterized in that, After obtaining the rendering instructions, the rendering method further includes: The prepared frame buffer rendering resources are assembled into the preset resource cache space; Activate the shading module and set the display content cache space to trigger 3D rendering of the frame cache of the rendering command; If the 3D rendering is complete, a next frame preparation message is sent to trigger the rendering instruction for generating the next frame of the frame buffer.

12. The rendering method as described in claim 9, characterized in that, The plurality of data to be rendered includes data to be rendered from multiple applications. Before performing 3D rendering on the frame buffer, the rendering method further includes: Each application requests a temporary window and a temporary layer interface to generate a layer instance for the transition animation; Animation instances are generated based on the coating instances of each layer interface, and each animation instance is bound to the scene layer to realize the animation effects of 3D rendering of the frame buffer.

13. The rendering method as described in claim 9, characterized in that, After applying for temporary layer interfaces, the rendering method also includes releasing each of the temporary windows.

14. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the rendering method as described in any one of claims 9 to 13.

15. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the rendering method according to any one of claims 9 to 13.

Images