Image rendering method and related device

Abstract

The present application provides an image rendering method and related equipment, which relate to the field of image processing. According to the method, the application sends a first rendering instruction stream, and the first rendering instruction stream is used to instruct the electronic device to perform a rendering operation of the first frame image; the application sends a second rendering instruction stream, and the second rendering instruction stream is used to instruct the electronic device to perform a rendering operation of the second frame image; the electronic device determines a third rendering result based on the first rendering result and the second rendering result; the first rendering result is the rendering result corresponding to the first main scene, and the second rendering result is the rendering result corresponding to the second main scene; the electronic device mixes the third rendering result and the fourth rendering result to determine a predicted frame image; the fourth rendering result is the rendering result corresponding to the first dynamic translucent object. The present application renders the dynamic translucent objects in each frame separately from the main scene and then mixes them as needed, thereby avoiding jitter problems.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image rendering method and related equipment. Background Art

[0002] With the development of electronic devices, the content of displayed images is becoming more and more abundant. In some images, semi-transparent objects such as water columns, bubbles, glass, transparent shells, etc. may be included.

[0003] However, when the semi-transparent object is in motion, the rendering effect of these image frames is not satisfactory. For example, when the semi-transparent object is in motion and the motion state is inconsistent with the motion trend (or motion direction) of the object superimposed on it, the relevant technical processing will cause the corresponding picture of the semi-transparent object to jitter.

[0004] Therefore, there is an urgent need for an image rendering method that can improve the rendering effect of semi-transparent dynamic objects. Summary of the invention

[0005] The present application provides an image rendering method and related devices thereof. According to the method, semi-transparent objects are rendered separately in a moving state (or dynamic), and the corresponding picture quality can be effectively improved without jitter.

[0006] In a first aspect, the present application provides an image rendering method, which is applied to an electronic device installed with an application program, and the method may include:

[0007] The application sends a first rendering instruction stream, where the first rendering instruction stream is used to instruct the electronic device to perform a rendering operation on a first frame of image, where the first frame of image includes a first main scene and a first dynamic semi-transparent object;

[0008] The application sends a second rendering instruction stream, where the second rendering instruction stream is used to instruct the electronic device to perform a rendering operation on a second frame of image, where the second frame of image includes a second main scene and a second dynamic semi-transparent object;

[0009] The electronic device determines a third rendering result according to the first rendering result and the second rendering result; the first rendering result is a rendering result corresponding to the first main scene, and the second rendering result is a rendering result corresponding to the second main scene;

[0010] The electronic device mixes the third rendering result and the fourth rendering result to determine a predicted frame image; the fourth rendering result is a rendering result corresponding to the first dynamic translucent object.

[0011] Through the image rendering method provided by the embodiment of the present application, for the rendering instruction stream corresponding to each frame of the image, a separately drawn dynamic semi-transparent object texture and a main scene pipeline texture can be obtained; then, the main scene rendering result of the predicted frame is determined based on the rendering results corresponding to the two frames of the main scene pipeline, and the dynamic semi-transparent object rendering result of the predicted frame is determined based on the rendering result of one frame of the dynamic semi-transparent object, and the predicted frame is obtained by combining the two. Here, since the dynamic semi-transparent object is drawn separately, it will not be affected by other superimposed objects. Therefore, the drawing quality of the dynamic semi-transparent object can be improved, and the display effect of the dynamic semi-transparent object in the predicted frame can be improved.

[0012] In some possible implementations, the first rendering instruction stream includes a first instruction stream and a second instruction stream, the first instruction stream is used to instruct the electronic device to render the first main scene to obtain the first rendering result, and the second instruction stream is used to instruct the electronic device to render the first dynamic semi-transparent object to obtain the fourth rendering result;

[0013] Before the electronic device mixes the third rendering result and the fourth rendering result, the method further includes:

[0014] The electronic device performs rendering according to the first instruction stream to obtain the first rendering result, and stores the first rendering result in a first frame buffer, wherein the first instruction stream includes instructions pointing to the first frame buffer;

[0015] The electronic device performs rendering according to the second instruction stream to obtain the fourth rendering result, and stores the fourth rendering result in a second frame buffer, wherein the second instruction stream includes instructions pointing to the second frame buffer, and the second frame buffer is different from the first frame buffer.

[0016] In the implementation method, a moving translucent object is first identified, and after the moving translucent object is identified, the frame buffer is switched, and the semi-transparent object is rendered separately using the switched frame buffer to obtain a separately drawn semi-transparent object texture; then, the separately drawn semi-transparent object texture is mixed with the main scene texture to obtain the final image frame. In this process, since the moving semi-transparent object is drawn separately using the newly created frame buffer, the influence of the superimposed object on the movement trend of the semi-transparent object during drawing can be avoided, thereby avoiding the jitter of the picture when the semi-transparent object moves, and thus achieving the purpose of improving the picture quality corresponding to the semi-transparent object.

[0017] In some possible implementations, before the electronic device performs rendering according to the second instruction stream to obtain the fourth rendering result, the method further includes: the electronic device creates the second frame buffer.

[0018] In the implementation, the rendertarget is changed between the dynamic translucent object and the non-dynamic translucent object, and different frame buffers are used for corresponding drawing, such as using the second frame buffer to draw the dynamic translucent object.

[0019] In some possible implementations, when the electronic device performs rendering, it determines that a semi-transparent object is to be drawn based on the preset first instruction and second instruction in the second instruction stream.

[0020] Exemplarily, the preset first instruction includes at least one of the following: glEnable instruction, glEnablei instruction. The preset second instruction includes at least one of the following: glDisable instruction, glDisablei instruction, glDiscardFramebufferEXT() instruction. Based on this solution, a specific example of the first instruction and the second instruction is provided. In this implementation, the judgment of the translucent object can be realized.

[0021] In some possible implementations, when the electronic device performs rendering, it determines that what is being drawn is a dynamic object based on the third instruction, the fourth instruction, and the fifth instruction included in the second instruction stream.

[0022] Exemplarily, the third instruction may be a glBindBuffer instruction; the fourth instruction may be a glBufferSubData instruction; and the fifth instruction may be a glBindBufferRange instruction.

[0023] In this implementation, dynamic objects can be judged; combined with the judgment of translucent objects, dynamic translucent objects can be judged.

[0024] In some possible implementations, the method further includes:

[0025] The electronic device mixes the third rendering result and the fifth rendering result to determine the predicted frame image; the fifth rendering result is the rendering result corresponding to the second dynamic semi-transparent object.

[0026] In this implementation, the predicted frame image can be mixed in another way.

[0027] In some possible implementations, after determining the third rendering result, the method further includes:

[0028] The electronic device determines a seventh rendering result according to the fifth rendering result and the sixth rendering result; the fifth rendering result is a rendering result corresponding to the first dynamic semi-transparent object, and the sixth rendering result is a rendering result corresponding to the second dynamic semi-transparent object;

[0029] The electronic device mixes the third rendering result and the seventh rendering result to determine the predicted frame image.

[0030] In this implementation, the predicted frame image can be mixed in another way.

[0031] In some possible implementations, the first frame image is the Nth frame image, the second frame image is the N+2th frame image, and the predicted frame image is the N+1th frame image, or;

[0032] The first frame image is the Nth frame image, the second frame image is the N+1th frame image, and the predicted frame image is the N+2th frame image.

[0033] In this implementation, the middle frame can be obtained through the two preceding and following frames as the predicted frame image; or, the third frame can be obtained through two consecutive frames as the predicted frame image.

[0034] In a second aspect, the present application provides an image rendering device, the device comprising a unit for executing the method in the first aspect. The device may correspond to executing the method described in the first aspect, and the relevant description of the unit in the device can refer to the description of the first aspect, which will not be repeated here for the sake of brevity.

[0035] The method described in the first aspect above can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions.

[0036] In a third aspect, the present application provides an electronic device, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute a method involved in the first electronic device or the second electronic device in the first aspect or any one of the implementations of the first aspect.

[0037] In a fourth aspect, the present application provides a computer storage medium, which stores a computer program. When the computer program is executed by an electronic device, the electronic device executes the method involved in the first aspect or any one of the implementations of the first aspect.

[0038] In the fifth aspect, an embodiment of the present application provides a chip system, which can be applied to an electronic device. The chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method described in the first aspect or any one of the implementations of the first aspect.

[0039] In a sixth aspect, an embodiment of the present application provides a computer program product, comprising: a computer program code, when the computer program code is executed by an electronic device, the electronic device executes any one of the methods in the first aspect.

[0040] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of an image frame including a semi-transparent object provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of another image frame of a semi-transparent object provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of a rendering process provided in an embodiment of the present application;

[0046] Figure 6 A rendering instruction stream provided in an embodiment of the present application;

[0047] Figure 7 The data related to the world coordinates in RenderDoc provided in the embodiment of the present application;

[0048] Figure 8 A schematic diagram of a flow chart of an image rendering method provided in an embodiment of the present application;

[0049] Fig. 9 A schematic diagram of the rendering pipeline sequence provided for an embodiment of the present application;

[0050] Fig.10 Another schematic diagram of a flow chart of an image rendering method provided in an embodiment of the present application;

[0051] Fig.11 A schematic diagram of another process flow of the image rendering method provided in an embodiment of the present application;

[0052] Fig.12 A schematic diagram of another process flow of the image rendering method provided in an embodiment of the present application;

[0053] Fig.13A schematic diagram of another process flow of the image rendering method provided in an embodiment of the present application;

[0054] Fig.14 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0056] It should be understood that the terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0057] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0058] To facilitate understanding, the nouns involved in the embodiments of the present application are first explained.

[0059] 1. Rendering: The process of generating an image from a model using software. A model is a description of a three-dimensional object using a strictly defined language or data structure, which includes geometry, viewpoint, texture, and lighting information. The process of projecting a model in a three-dimensional scene into a digital image in two dimensions according to the set environment, lighting, material, and rendering parameters.

[0060] 2. Game scene: represents the running stage in the game application, such as lobby scene, sports scene, slight change scene, distant scene, etc.

[0061] 3. Frame prediction technology (also called interpolation algorithm): Use the image information of two frames to predict the image information of another frame. For example, the image information of two adjacent frames can be used to predict the image information of the next frame, or the image information of two interval frames can be used to predict the image information of the middle frame. For example, in the embodiment of the present application, the image information of the Nth frame and the N+2th frame can be used to predict the image information of the N+1th frame.

[0062] 4. Translucent objects: can appear translucent in an image. For example, taking the image displayed by an electronic device as an image in a game scene, in a shooting game, the image may include translucent objects such as smoke, gunfire, and waves. Electronic devices can increase the realism of scene display and enhance user experience by adding rendering effects of translucent objects in the image. In the embodiment of the present application, translucent can include all transparency levels except opaque, including completely transparent.

[0063] 5. Engine: refers to the core components of developing programs or systems on electronic platforms. Using the engine, developers can quickly build and lay out the functions required by the program, or use it to assist the operation of the program. Generally speaking, the engine is the supporting part of a program or a system. Common program engines include game engines, search engines, etc.

[0064] 6. World space: The absolute coordinate space of the world (scene) simulated by electronic devices. World coordinates (world position) are used to indicate the position in the simulated world.

[0065] As the performance of electronic equipment hardware platforms improves, games, videos and other display content are gradually moving towards high frame rates and high image quality. High frame rates can bring users a smoother experience, but they also bring high power consumption issues. In order to reduce rendering overhead and power consumption, frame prediction technology came into being.

[0066] Taking the game scene as an example, in the frame prediction technology, the historical frames can be processed using the interpolation algorithm to determine the predicted frame; then, the determined predicted frame is used to replace the original real frame for rendering, thereby achieving the purpose of reducing power consumption. It can be understood that a lot of image information between adjacent frames can be reused, and frame prediction can reduce the consumption of this reusable image information.

[0067] However, when the displayed content includes semi-transparent objects, the predicted frames generated by the interpolation algorithm may not necessarily achieve the same image quality as the real frames; especially when the semi-transparent objects are in motion, the rendering effect of the predicted frames is even less satisfactory.

[0068] From the analysis, it can be seen that the semi-transparent object itself has no depth, and the final rendered color and depth are related to the superimposed objects, and are greatly affected by the superimposed objects. Therefore, when the frame prediction technology provided by the relevant technology is processed, the semi-transparent object is usually processed together with the superimposed objects (or background objects). However, when the semi-transparent object is in motion (or dynamic), and the motion trend is inconsistent with the superimposed object, if the frame prediction technology provided by the relevant technology is continued to be used for processing, the part of the picture corresponding to the semi-transparent object will cause jitter, affecting the user's visual experience. It should be noted that dynamic semi-transparent objects will have jitter, while static semi-transparent objects will not.

[0069] For example, Figure 1 FIG. 1 shows a schematic diagram of an image frame including a semi-transparent object provided by an embodiment of the present application. Figure 1 As shown, in the game scene, when the game character throws a game item, the throwing line (with a certain width and height) of the game item will be displayed on the screen. Here, the throwing line can be used as a translucent object, and the content covered by the throwing line is the superimposed object corresponding to the translucent object.

[0070] When the viewing angle corresponding to the game character moves to the left, the throwing line should move with the direction of the game character's viewing angle, such as moving to the left, while the superimposed object is moving to the right. It can be seen that the movement trend of the translucent object is different from that of the superimposed object, and can even be considered to be completely opposite. At this time, if the frame prediction technology provided by the relevant technology is used, the translucent objects in multiple predicted frames may be affected by the superimposed objects and show a movement trend to the right, while the translucent objects in the non-predicted frames show a movement trend to the left. When the two parts of the frame are displayed after being sorted according to the frame sequence, the movement trend of the translucent object will show a situation of constantly alternating left and right; for the user, during the game, the translucent object (throwing line) seen will be jittery.

[0071] For example, Figure 2 FIG. 2 shows another image frame including a semi-transparent object provided by an embodiment of the present application. Figure 2 As shown, in this game scene, the game character drives a "Bobo car" to move, and the shell of the "Bobo car" displayed in the screen is a translucent object, and the content covered by the shell is the superimposed object corresponding to the translucent object.

[0072] When the game character drives the Bobo car forward, the car shell should move in the direction of the game character, such as moving inward (in the direction perpendicular to the display screen), while the superimposed object is moving outward. It can be seen that the movement trend of the translucent object is different from that of the superimposed object, and can even be considered to be completely opposite. At this time, if the frame prediction technology provided by the relevant technology is used, the translucent objects in multiple predicted frames may be affected by the superimposed objects and show an outward movement trend, while the translucent objects in the non-predicted frames show an inward movement trend. When the two parts of the frames are displayed after being sorted according to the frame sequence, the movement trend of the translucent object will show a situation of constantly alternating inside and outside; for the user, during the game, the translucent object (car shell) seen will be jittery.

[0073] In view of this, the present application provides a new image rendering method. For the rendering instruction stream, a moving translucent object is first identified. After the moving translucent object is identified, the frame buffer is switched, and the switched frame buffer is used to render the translucent object separately to obtain a separately drawn translucent object texture; then, the separately drawn translucent object texture is mixed with the main scene texture to obtain the final image frame. In this process, since the moving translucent object is drawn separately using the newly created frame buffer, the influence of the superimposed object on the movement trend of the translucent object during drawing can be avoided, thereby avoiding the jitter of the picture when the translucent object moves, and thus achieving the purpose of improving the picture quality corresponding to the translucent object.

[0074] The solution provided in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0075] It should be noted that the image rendering method provided in the embodiment of the present application can be applied to electronic devices with display functions. The electronic device can be an electronic device such as a mobile phone, a tablet computer, or a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, and a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device (such as a smart watch, a smart bracelet, etc.), a vehicle-mounted device, a smart home device (such as a smart TV, a large screen, etc.) and / or a smart city device. The embodiment of the present application does not impose any special restrictions on the specific type of the electronic device.

[0076] For example, in some embodiments, from the perspective of hardware composition, Figure 3 As shown, the electronic device involved in the embodiment of the present application may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0077] It should be noted that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the modules illustrated in the embodiments of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device.

[0078] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. A memory may also be provided in the processor 110 for storing instructions and data.

[0079] The electronic device implements the display function through a GPU, a display screen 140, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 140 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information. The display screen 140 is used to display images, videos, etc. In some embodiments, the electronic device may include 1 or N display screens 140, where N is a positive integer greater than 1.

[0080] In the embodiment of the present application, the function of the electronic device to display the aforementioned game interface can be implemented through a GPU, a display screen 140, and an application processor, etc.

[0081] The electronic device can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor. Camera 150 is used to capture static images or videos. ISP is used to process data fed back by camera 150. Light is transmitted to the camera photosensitive element through the lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. The electronic device may include 1 or N cameras 150, where N is a positive integer greater than 1. Video codec is used to compress or decompress digital video. Electronic devices can support one or more video codecs. In this way, electronic devices can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0082] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data. The non-volatile memory can also store executable programs and store user and application data, etc., and can be loaded into the random access memory in advance for direct reading and writing by the processor 110.

[0083] The external memory interface 120 may be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device.

[0084] The electronic device can implement audio functions through the audio module 130, the speaker 130A, the receiver 130B, the microphone 130C, the headphone interface 130D, and the application processor.

[0085] In other embodiments, the electronic device involved in the embodiments of the present application may also have software partitioning. For example, an electronic device running an Android operating system may have hierarchical software partitioning.

[0086] For example, Figure 4 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application.

[0087] like Figure 4 As shown, the electronic device may include: an application (APP) program layer, an application framework (Framework) layer, an Android runtime (Android runtime) and a system library, and a kernel layer.

[0088] like Figure 4 As shown, the application layer may include a series of application packages, which may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0089] In an embodiment of the present application, the application package may also include an application that needs to display images or videos to users by rendering images. Among them, the video can be understood as the continuous playback of multiple frames of images, and the frame image may include a moving translucent object. Exemplarily, the application may include a game application.

[0090] The application framework layer provides an application programming interface (API) and a programming framework for the applications of the application layer. The application framework layer includes some predefined functions. Exemplarily, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, etc.

[0091] The window manager is used to provide window management services (WMS). The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0092] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0093] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0094] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0095] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, the terminal device vibrates, the indicator light flashes, etc.

[0096] The activity manager can provide acquisition management services (activity manager service, AMS), which can be used to start, switch, and schedule system components (such as activities, services, content providers, and broadcast receivers) as well as manage and schedule application processes.

[0097] The input manager can provide input management services (IMS). The input manager can be used to manage the input of the system, such as touch screen input, key input, sensor input, etc. The input manager takes events from the device node and distributes the events to the appropriate window through interaction with the WMS.

[0098] In an embodiment of the present application, one or more functional modules may be provided in the application framework layer to implement the image rendering method provided in the present application.

[0099] Exemplarily, the application framework layer may be provided with an identification module, an interception module, a creation module, a mixing module, etc. Among them, the identification module is used for offline pipeline analysis, for example, determining the main scene pipeline by offline analyzing the main scene features, and analyzing dynamic semi-transparent object recognition features.

[0100] The interception module may be used to intercept real-time related instructions. For example, the related instructions may refer to instructions issued by an application program to instruct rendering of a semi-transparent object.

[0101] The frame prediction module can be used to perform related algorithm processing and use the image information of two frames to predict the image information of another frame. For example, the image information of the N+1 frame can be predicted based on the image information of the Nth frame and the N+2th frame.

[0102] The creation module can be used to create a new storage space, for example, to create a new framebuffer (FB), the texture corresponding to the framebuffer, etc.

[0103] The blending module can be used to instruct the electronic device to synthesize the rendering result of the semi-transparent object stored in the newly created frame buffer with other rendering results, so as to obtain the rendering result corresponding to the frame image. Among them, the other rendering results can include static semi-transparent objects and / or non-semi-transparent objects, etc.

[0104] The system library may include a graphics library. In different implementations, the graphics library may include at least one of the following: an open graphics library (OpenGL), an open graphics library for embedded systems (OpenGL ES), and a 2D graphics engine (e.g., skia graphics library (SGL)). In some embodiments, the system library may also include other modules, such as a surface manager, a media library, and the like.

[0105] Among them, the surface manager is used to manage the display subsystem and provide the fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications. The media library supports the playback and recording of multiple audio formats, multiple video formats and static image files. The media library can support multiple audio and video coding formats, such as: moving pictures experts group 4 (MPEG4), H.264, moving picture experts group audio layer III (MP3), advanced audio coding (AAC), adaptive multi-rate (AMR), joint photographic experts group (JPG) and portable network graphics (PNG). The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0106] like Figure 4 As shown, the electronic device may also include a hardware layer. The hardware layer may include a central processing unit (CPU), a graphics processing unit (GPU), and a memory with a storage function. In some implementations, the CPU may be used to control each module in the framework layer to implement its respective functions, and the GPU may be used to perform corresponding rendering processing according to an API in a graphics library (such as OpenGL ES) called by instructions processed by each module in the application framework layer.

[0107] For ease of understanding, the following will be combined Figure 4 The software division shown takes the application program as a game application as an example to explain in detail the solution provided in the embodiment of the present application.

[0108] For electronic devices, in order to obtain image data for display, the electronic devices can render images according to the rendering instruction stream issued by the application (such as a game application), and then obtain image data including moving translucent objects for display.

[0109] For example, Figure 5 A schematic diagram of the rendering process, combined with Figure 4 and Figure 5When rendering a certain frame of image, the game application can send a rendering instruction stream; the CPU can call the interface in the graphics library according to the rendering instruction stream to instruct the GPU to perform the corresponding rendering operation; the rendering result of the GPU performing the rendering operation can be stored in the electronic device, and after the subsequent rendering corresponding to the rendering instruction stream, the display data can be obtained. Then, the electronic device can display the frame of image on the display screen according to the display data.

[0110] The frame image may include a moving semi-transparent object, so the rendering instruction stream sent by the game application may include an instruction segment for instructing to render the main scene, an instruction segment for instructing to render the semi-transparent object, etc. It can be understood that the semi-transparent object is drawn in the main scene.

[0111] Among them, the main scene may correspond to the scene with the highest rendering load executed by the electronic device during the rendering process of the current frame image. Exemplarily, in some embodiments, the main scene may correspond to the rendering pipeline (render pass) with the largest number of rendering commands (drawcall). In the rendering process of a frame of image, multiple render passes may be included. The rendering results of each render pass can be stored in a frame buffer. The rendering of each render pass can include multiple draw calls. The more draw calls are executed, the richer the content of the map obtained after the corresponding render pass is completed. In other embodiments, the main scene may include multiple rendering targets (color attachment) or color buffers (color buffer); or, the main scene may also be a scene with a rendering command greater than a preset threshold.

[0112] When drawing multiple objects including translucent objects in the main scene, the blending equations involved in the subsequent synthesis are related to the drawing order. Therefore, when drawing multiple objects, it is usually necessary to sort the objects. For example, first draw the opaque objects, and then process the translucent objects from back to front on top of the opaque objects, otherwise serious distortion will occur. In this drawing process, game applications usually draw translucent objects through a command segment that starts with a specific beginning command (such as the glEnable command) and ends with a specific ending command (such as the glDisable command).

[0113] It should be understood that when drawing an object on the screen, the alpha value is used to describe the degree of transparency of the object at a given pixel. Therefore, in order to render different transparency levels (i.e., different alpha values) and distinguish between opaque and transparent objects of different degrees, the color mixing state needs to be enabled during rendering. In an embodiment of the present application, the game application can enable the color mixing state through the glEnable instruction, that is, instruct the electronic device to start rendering a semi-transparent object through the glEnable instruction.

[0114] For example, Figure 6 A rendering instruction flow provided by an embodiment of the present application is shown. Figure 6 As shown in the figure, the instruction with ID 1567 sent by the game application is glEnable, that is, the glEnable instruction can instruct the electronic device to enable the color mixing state. Then, in the subsequent instructions, the game application can start to instruct the electronic device to perform the corresponding operation of semi-transparent object rendering through different instructions. For example, the game application can set the mixing factor through the glBlendFuncSeparate instruction with ID 1568.

[0115] It should also be understood that if before the main scene drawing process is completed, only the glEnable instruction is issued to indicate the start of the corresponding operation of semi-transparent object rendering, but the glDisable instruction is not enabled to indicate the end of the semi-transparent object drawing, then it indicates that the currently drawn object is still a semi-transparent object; that is, from glEnable until the currently drawn object, all that is drawn are semi-transparent objects.

[0116] In addition, each object in the game scene has a corresponding world coordinate, including static objects and dynamic objects. And, by analyzing the implementation characteristics of the game engine (such as the Unreal engine), it can be seen that in the process of rendering each frame, dynamic objects need to continuously update the world coordinates, while static objects do not need to be updated or the world coordinates corresponding to static objects remain unchanged. When objects in motion continuously update their world coordinates, this data will be stored in a uniform buffer object (UBO). Therefore, when rendering, each frame needs to call glBufferSubData to update the UBO data first; in contrast, static objects only need to call glBufferData once because their world coordinates remain unchanged; therefore, the UBO storing the world coordinates can be identified by matching the buffer size, and whether the object is a dynamic object can be determined based on whether the currently drawn object uses the corresponding UBO data.

[0117] It should be noted that the size of the buffer is used to indicate the size of the UBO used to describe the world coordinate information in the Shader, and the size is in bytes. For example, the UBO describing the world coordinate information can be called VB1, and the corresponding size (Byte Range) is 0 to 352. Therefore, when identifying, you can determine whether the buffer is a UBO (or VB1) that stores world coordinates by matching whether the buffer size is 352.

[0118] It can be understood that UBO is a buffer object that stores uniform type variables in GLSL, through which data sharing between different shaders can be achieved.

[0119] For example, Figure 7 1 shows the relevant data of the world coordinates in the RenderDoc provided by the embodiment of the present application. Figure 7 As shown, it indicates the information of Vertex UBO1 captured from RenderDoc. Among them, Primitive_ActorWorldPositionvb1 represents the position of the object in the world space, that is, the world coordinates corresponding to the object are (53326.00,54811.00,1429.21997). When the object is in motion, the value will also change, so the corresponding bound buffer will complete the update of the corresponding data before the object is drawn in each frame.

[0120] Through the above analysis, it can be known that the main scene, the semi-transparent object, and the motion state can all be identified based on the corresponding conditions. In combination with the analysis, the solution of the present application is introduced below.

[0121] Figure 8 A schematic flow chart of an image rendering method provided in an embodiment of the present application is shown.

[0122] like Figure 8 As shown, the image rendering method provided by the embodiment of the present application may include a first stage, a second stage and a third stage, the first stage may include the following S10 to S20, the second stage may include the following S30 to S60, and the third stage may include the following S70 and S80. The first stage may also be called a pre-stage, and the third stage may also be called an output stage.

[0123] Phase 1:

[0124] S10, the recognition module determines whether it is the main scene; if so, execute S20; if not, draw normally.

[0125] Optionally, the identification module may determine whether it is the main scene by identifying whether the current rendering pipeline is a main scene pipeline (main pass).

[0126] The main scene pipeline is the pass for drawing the main scene. Generally, the main scene features can be obtained through offline analysis. Then, when the game draws the current frame, whether it is the main scene can be determined by analyzing whether the features are the main scene features.

[0127] The main scene feature refers to the aforementioned features such as the largest number of rendering commands, multiple rendering targets, or multiple color buffers, or the rendering command being greater than a preset threshold. Of course, the main scene feature can also be other features, which are not limited in this application. It is understood that translucent objects are drawn in the main pass.

[0128] For example, Fig. 9 As shown, taking a game application as an example, rendering can usually include a depth pipeline (depthpass), a main scene pipeline and a UI pipeline (UI pass). When the main scene pipeline is executed, it can be drawn in combination with the depth information provided by the depth pipeline, and the UI pipeline can draw and generate a separate UI texture. Since there are multiple pipelines during rendering, it is necessary to identify the current rendering pipeline; only when the current rendering pipeline is the main scene pipeline, continue with the subsequent solutions of this application.

[0129] Optionally, the process of determining the main scene pipeline may be performed before starting to render the current frame image.

[0130] S20. The creation module creates a frame buffer and a texture corresponding to the semi-transparent object.

[0131] It is understandable that the main scene may include translucent objects. In order to draw the translucent objects separately later, the present application needs to first create a new frame buffer and texture to provide for the translucent objects.

[0132] For example, the following is an instruction flow for creating a frame buffer and a texture corresponding to a semi-transparent object provided by an embodiment of the present application. For example:

[0133] unsigned int blendFB;

[0134] glGenFramebuffers(1,&blendFB); / / Create frame buffer (blendFB);

[0135] glBindFramebuffer(GL_FRAMEBUFFER,blendFB); / / Bind framebuffer;

[0136] unsigned int textureColorbuffer;

[0137] glGenTextures(1,&textureColorbuffer); / / Create texture map (textureColorbuffer);

[0138] glBindTexture(GL_TEXTURE_2D,textureColorbuffer); / / Bind texture map;

[0139] glTexImage2D(GL_TEXTURE_2D,0,GL_RGB,SCR_WIDTH,SCR_HEIGHT,0,GL_RGB,GL_UNSIGNED_BYTE,NULL);

[0140] glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MIN_FILTER,GL_LINEAR);

[0141] glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MAG_FILTER,GL_LINEAR);

[0142] glFramebufferTexture2D(GL_FRAMEBUFFER,GL_COLOR_ATTACHMENT0,GL_TEXTURE_2D,textureColorbuffer,0); / / Bind textureColorbuffer to FB(blendFB).

[0143] It should be understood that based on the above instructions, a frame buffer (blendFB) and a texture (attachment texture) corresponding to the semi-transparent object can be created. The creation of the frame buffer and the texture corresponding to the semi-transparent object is performed before starting the rendering of the current frame image.

[0144] Phase 2:

[0145] S30, the interception module intercepts the rendering instruction stream and determines whether the object being drawn is a semi-transparent object. If so, continue to execute S40; if not, execute S60.

[0146] Optionally, the game application sends a rendering instruction stream, and the interception module intercepts a first instruction stream to determine the drawing of a semi-transparent object. The first instruction stream may be an instruction instructing the electronic device to render a semi-transparent object.

[0147] Exemplarily, during the rendering process of the Nth frame image, the interception module can monitor whether a preset beginning instruction appears in the rendering instruction stream issued by the game application. For example, the beginning instruction can be a glEnable() instruction. The interception module can determine to start drawing translucent objects after detecting the glEnable() instruction. The interception module can also monitor whether a preset ending instruction appears in the rendering instruction stream issued by the game. For example, the ending instruction can be a glDisable() instruction. The interception module can determine to stop drawing translucent objects after detecting the glDisable() instruction, and then stop intercepting instructions. Therefore, the intercepted instructions can be called the first instruction stream.

[0148] Optionally, the first instruction stream may include a preset start instruction and an end instruction, as well as other instructions before the start instruction and the end instruction; or, the first instruction stream may only include a preset start instruction and other instructions after the start instruction. The start instruction may also be referred to as the first instruction, and the end instruction may also be referred to as the second instruction.

[0149] S40, the interception module intercepts the rendering instruction stream and determines whether the semi-transparent object is moving. If so, continue to execute S50; if not, execute S60.

[0150] Optionally, the interception module may determine whether the buffer bound to the translucent object is updated by intercepting the second instruction stream, and further determine whether the translucent object is moving. If updated, it can be considered that the translucent object is moving.

[0151] The second instruction stream may be instructions for instructing the electronic device to use updated UBO data for rendering.

[0152] As a way to achieve this, Fig.10 As shown, the above S40 may include the following S401 to S403.

[0153] S401. The interception module intercepts the third instruction and determines the identifier (ID) of the currently bound UBO.

[0154] For example, the third instruction may be glBindBuffer, and the glBindBuffer function is used to bind a buffer with an ID to an object. Therefore, the interception module intercepts glBindBuffer, identifies the target as GL_UNIFORM_BUFFER, and can determine the ID of the currently bound UBO. For example, if the ID is 1, it can be determined that the currently bound UBO is UBO1.

[0155] S402: The interception module intercepts the fourth instruction, and when it is determined that the currently bound UBO stores world coordinates, stores the UBO in the corresponding Map.

[0156] For example, the fourth instruction may be glBufferSubData, and the glBufferSubData function is used to update the data in the bound buffer. The interception module intercepts glBufferSubData and identifies the size of the buffer to determine whether the currently bound UBO1 is VB1. For example, when the buffer size is equal to the preset threshold (the difference between the maximum and minimum values ​​of the byte range), such as when the preset threshold is 352 and the buffer size is 352, the buffer size is the same as the preset threshold, thereby determining that the buffer is VB1. VB1 indicates a code structure that stores information related to the world coordinates of an object. The 1 of VB1 is an index value.

[0157] At this time, if it is determined to be VB1, it means that the current UBO stores updated world coordinate information. After determining to be VB1, the currently bound UBO (UBO1) needs to be stored in the corresponding Map, such as VB1 Map. If it is determined not to be VB1, it can be considered that the current command drawing is not dynamic and will not be stored.

[0158] S403: The interception module intercepts the fifth instruction and determines whether the object being drawn is a dynamic object.

[0159] For example, the fifth instruction may be glBindBufferRange, and the glBindBufferRange function is used to bind the buffer to the binding point pointed to by the index specified by the target. The interception module intercepts glBindBufferRange, and when it recognizes that the index value is 1 (or when it recognizes VB1), it determines whether the uniform buffer bound to index 1 is in the VB1 Map. If it is, it means that the current command is drawing a dynamic object; if not, it means that the current command is not drawing a dynamic object. It should be understood that since there may be multiple other buffers bound to index1, and the buffer may not exist in the VB1 Map, it is necessary to query and compare in the VB1 Map.

[0160] Optionally, the second instruction stream may include the third instruction, the fourth instruction and the fifth instruction.

[0161] It should be understood that the above is the process of determining whether the motion state is dynamic. Combined with the above, if the object is a semi-transparent object, it can be determined that the object currently being drawn is a dynamic semi-transparent object.

[0162] Optionally, the determination order of the above S30 and S40 can be reversed, or can be performed simultaneously. The present application does not impose any limitation on this. As long as the interception module intercepts the rendering instruction stream, identifies the rendering instruction for drawing dynamic translucent objects from the main pass, and determines to draw dynamic translucent objects, it can be done.

[0163] S50, switching the frame buffer (blendFB) corresponding to the semi-transparent object and drawing it.

[0164] Optionally, when dynamic semi-transparent objects in the main scene pipeline are determined for the rendering instruction stream, the rendering target can be changed first, and glBindFramebuffer(GL_FRAMEBUFFER, blendFB) can be executed to bind the semi-transparent objects to the newly created frame buffer and draw them. In this way, the rendering results of the dynamic semi-transparent objects can be stored in the frame buffer (blendFB).

[0165] S60: Rebind the frame buffer (mainFB) corresponding to the main scene pipeline and perform drawing.

[0166] It should be understood that after changing the blendFB, if you find that you need to draw other objects besides dynamic semi-transparent objects, such as dynamic semi-transparent objects, dynamic non-transparent objects, etc., you need to continue to change the render target and execute glBindFramebuffer(GL_FRAMEBUFFER, mainFB) to change the blendFB to mainFB; then, draw with mainFB. In this way, the rendering results of other objects besides dynamic semi-transparent objects can be stored in the frame buffer (mainFB).

[0167] S70, obtaining a separately drawn dynamic semi-transparent object texture.

[0168] It should be understood that the dynamic semi-transparent object is drawn on the blendFB, so that a separately drawn dynamic semi-transparent object texture can be obtained.

[0169] S80: Get the main scene pipeline texture.

[0170] It should be understood that objects other than the dynamic semi-transparent objects are drawn on the mainFB, and the main scene pipeline texture including the objects other than the dynamic semi-transparent objects can be obtained.

[0171] It should be understood that when it is determined that it is not a translucent object after S30 and when it is determined that it is not a dynamic object after S40, if the binding has not been changed before, it is drawn on mainFB; if the binding has been changed before, it is necessary to switch from blendFB to mainFB and then draw.

[0172] In the embodiment of the present application, it is possible to analyze whether the object being drawn is a dynamic translucent object according to preset conditions, change the render target between a dynamic translucent object and a non-dynamic translucent object, and use different frame buffers for corresponding drawing until the main Pass drawing is completed; finally, a separately drawn dynamic translucent object texture and a main scene pipeline texture can be obtained; it should be understood that the main scene pipeline texture does not include a dynamic translucent object texture. Here, since the dynamic translucent object is drawn separately, it will not be affected by other superimposed objects, so the drawing quality of the dynamic translucent object can be improved, and the display effect of the dynamic translucent object in the corresponding frame can be improved.

[0173] If the binding is not changed, it is equivalent to superimposing other objects in the process of drawing the dynamic semi-transparent object, which will cause problems in the prior art and the drawing effect will be poor.

[0174] It should be understood that the second stage and the third stage mentioned above may be processing processes for a certain image frame.

[0175] Combining the above method with frame prediction technology, Fig.11 FIG. 1 is a flow chart of an image rendering method provided by an embodiment of the present application. Fig.11 As shown, the image rendering method provided in the embodiment of the present application may include the following S110 to S160, which are introduced below.

[0176] S110 . For the Nth image frame, determine a corresponding first dynamic semi-transparent object texture.

[0177] Wherein, N is an integer greater than or equal to 1.

[0178] S120 . Determine a first main scene pipeline texture for the Nth image frame.

[0179] Optionally, you can Figure 8 , Fig.10The method shown determines whether the rendering instruction stream corresponding to the Nth image frame includes a semi-transparent object and whether the world coordinates are updated; when the conditions are met, the frame buffer corresponding to the semi-transparent object is switched and drawn to obtain a frame of dynamic semi-transparent object texture drawn separately; when the conditions are not met, the frame buffer corresponding to the main scene is switched and drawn to obtain a frame of main scene texture. In this way, by switching the frame buffer for the Nth frame, the dynamic semi-transparent object texture and other textures can be drawn separately, thereby improving the drawing quality of the dynamic semi-transparent object included in the Nth frame.

[0180] S130 . Determine a corresponding second dynamic semi-transparent object texture for the (N+2)th image frame.

[0181] S140 . Determine a second main scene texture for the (N+2)th image frame.

[0182] Optionally, you can use 8, Fig.10 The method shown determines whether the rendering instruction stream corresponding to the N+2th frame image frame includes a semi-transparent object and whether the world coordinates are updated; when the conditions are met, the frame buffer corresponding to the semi-transparent object is switched and drawn to obtain a frame of dynamic semi-transparent object texture drawn separately; when the conditions are not met, the frame buffer corresponding to the main scene is switched and drawn to obtain a frame of main scene pipeline texture. In this way, by switching the frame buffer for the N+2th frame, the dynamic semi-transparent object texture and other textures can be drawn separately, thereby improving the drawing quality of the dynamic semi-transparent object included in the N+2th frame.

[0183] S150. For the N+1th image frame, the frame prediction module determines a predicted frame main scene pipeline texture (or called a third main scene pipeline texture) according to the first main scene pipeline texture and the second main scene pipeline texture.

[0184] Here, the predicted frame is the N+1th image frame.

[0185] It should be understood that the main scenes of different frames can be the same or different. For example, the main scene of the Nth image frame can be called the first main scene, and the main scene of the N+2th image frame can be called the second main scene. The first main scene and the second main scene can be the same or different. The frame buffer mainFB corresponding to the first main scene is different from the frame buffer mainFB corresponding to the second main scene.

[0186] It should also be understood that the dynamic semi-transparent objects in different frames of images may be the same or different. For example, the dynamic semi-transparent object in the Nth frame of the image may be the first dynamic semi-transparent object, and the dynamic semi-transparent object in the N+2th frame of the image may be the second dynamic semi-transparent object. The first dynamic semi-transparent object and the second dynamic semi-transparent object may be the same or different. The frame buffer blendFB corresponding to the first dynamic semi-transparent object is different from the frame buffer blendFB corresponding to the second dynamic semi-transparent object.

[0187] The frame prediction module can use the prediction method provided by the relevant technology to generate a new frame of texture according to the correlation of the input texture, that is, determine the pipeline texture of the main scene of the predicted frame, which is not limited in the embodiment of the present application. It should be understood that since the data input to the frame prediction module does not include dynamic semi-transparent objects, the predicted generated data also does not include dynamic semi-transparent objects.

[0188] S160: For the N+1th image frame, the mixing module mixes the main scene pipeline texture of the prediction frame and the first dynamic semi-transparent object texture to obtain a prediction frame. The prediction frame is the N+1th image frame.

[0189] Here, it is equivalent to reusing the first dynamic semi-transparent object texture corresponding to the Nth frame. When determining the N+1th frame, the dynamic semi-transparent object texture corresponding to the Nth frame and the main scene texture determined by the N+1th frame can be directly mixed. While reducing power consumption and improving processing efficiency, the drawing quality of dynamic semi-transparent objects can also be guaranteed.

[0190] Optionally, as another possible implementation manner, for the N+1th image frame, the mixing module may also mix the main scene pipeline texture of the prediction frame and the second dynamic semi-transparent object texture to obtain a prediction frame.

[0191] Here, the frame buffer blendFB corresponding to the first dynamic translucent object and the frame buffer blendFB corresponding to the second dynamic translucent object can be the same. In this way, when drawing the second dynamic translucent object, it is equivalent to replacing the data stored in the frame buffer blendFB, and replacing the first dynamic translucent object with the second dynamic translucent object.

[0192] Optionally, as another possible implementation method, the frame prediction module can determine the third dynamic semi-transparent object texture corresponding to the predicted frame based on the first dynamic semi-transparent object texture and the second dynamic semi-transparent object texture; then, the mixing module mixes the third main scene pipeline texture and the third dynamic semi-transparent object texture to obtain the predicted frame.

[0193] In the method provided in the embodiment of the present application, the main scene pipeline texture corresponding to the N+1 frame can be determined based on the main scene pipeline texture drawn in the Nth frame and the main scene pipeline texture drawn in the N+2 frame; the dynamic semi-transparent object texture drawn in the Nth frame is reused as the dynamic semi-transparent object texture drawn in the N+1 frame, and then the reused dynamic semi-transparent object texture and the main scene pipeline texture corresponding to the N+1 frame are mixed, so that the N+1 frame can be obtained. In this process, since the dynamic semi-transparent object texture is drawn separately, the dynamic semi-transparent object texture will not be affected by the superimposed objects, and the drawing quality is relatively high, thereby improving the picture quality of the predicted frame, avoiding jitter, and thus improving the user experience.

[0194] The above is implemented based on the frame prediction process of predicting the N+1 frame according to the N frame and the N+2 frame. Similarly, the process of predicting the N+2 frame according to the N frame and the N+1 frame is similar to the above. Fig.12 To give a hint.

[0195] like Fig.12 As shown, the image rendering method provided in the embodiment of the present application may include the following S210 to S260, which are introduced below.

[0196] S210 . Determine a corresponding first dynamic semi-transparent object texture for the Nth image frame.

[0197] S220 . Determine a first main scene pipeline texture for the Nth image frame.

[0198] S230 . Determine a corresponding second dynamic semi-transparent object texture for the (N+1)th image frame.

[0199] S240 . Determine a second main scene pipeline texture for the (N+1)th image frame.

[0200] S250. For the N+2th image frame, the frame prediction module determines a predicted frame main scene pipeline texture (or called a third main scene pipeline texture) according to the first main scene pipeline texture and the second main scene pipeline texture.

[0201] S260: For the N+2th image frame, the mixing module mixes the main scene pipeline texture of the prediction frame and the second dynamic semi-transparent object texture to obtain a prediction frame, which is the N+2th image frame.

[0202] Optionally, as another possible implementation manner, the mixing module may mix the main scene pipeline texture of the prediction frame and the first dynamic semi-transparent object texture to obtain the prediction frame.

[0203] Optionally, as another possible implementation method, the frame prediction module can determine the third dynamic semi-transparent object texture corresponding to the predicted frame based on the first dynamic semi-transparent object texture and the second dynamic semi-transparent object texture; then, the mixing module mixes the third main scene pipeline texture and the third dynamic semi-transparent object texture to obtain the predicted frame.

[0204] In the method provided in the embodiment of the present application, the main scene pipeline texture corresponding to the N+2 frame can be determined based on the main scene pipeline texture drawn in the Nth frame and the main scene pipeline texture drawn in the N+1th frame; the dynamic semi-transparent object texture drawn in the N+1th frame is reused as the dynamic semi-transparent object texture drawn in the N+2th frame, and then the reused dynamic semi-transparent object texture and the main scene pipeline texture corresponding to the N+2th frame are mixed, so that the N+2th frame can be obtained. In this process, since the dynamic semi-transparent object texture is drawn separately, the dynamic semi-transparent object texture will not be affected by the superimposed objects, and the drawing quality is relatively high, which can improve the picture quality of the predicted frame, avoid jitter, and thus improve the user experience.

[0205] It should be understood that in the above-mentioned mixing methods for determining the prediction frame, the UI texture obtained based on the UI pipeline drawing can also be mixed to obtain the prediction frame.

[0206] Fig.13 It is shown that an embodiment of the present application provides another image rendering method.

[0207] like Fig.13 As shown, the image rendering method provided in the embodiment of the present application may include the following S310 to S340, which is applied to an electronic device with an application installed, and is introduced below.

[0208] S310: The application sends a first rendering instruction stream, where the first rendering instruction stream is used to instruct the electronic device to perform a rendering operation on a first frame image, where the first frame image includes a first main scene and a first dynamic semi-transparent object.

[0209] The first rendering instruction stream may indicate Fig.11 The rendering instruction stream corresponding to the Nth frame image described in the first frame image indicates the Nth frame image; the first rendering instruction stream may also indicate Fig.12 In the rendering instruction stream corresponding to the N-th frame image, the first frame image indicates the N-th frame image.

[0210] S320: The application sends a second rendering instruction stream, where the second rendering instruction stream is used to instruct the electronic device to perform a rendering operation on a second frame of image, where the second frame of image includes a second main scene and a second dynamic semi-transparent object.

[0211] The second rendering instruction stream may indicate Fig.11 The rendering instruction stream corresponding to the N+2 frame image described in the figure, the second frame image indicates the N+2 frame image; the second rendering instruction stream may also indicate Fig.12 The rendering instruction stream corresponding to the N+1th frame image described in the figure, the second frame image indicates the N+1th frame image.

[0212] S330. The electronic device determines a third rendering result according to the first rendering result and the second rendering result; the first rendering result is a rendering result corresponding to the first main scene, and the second rendering result is a rendering result corresponding to the second main scene.

[0213] The third rendering result may indicate Fig.11 or Fig.12 The rendering result corresponding to the third main scene.

[0214] S340. The electronic device mixes the third rendering result and the fourth rendering result to determine a predicted frame image; the fourth rendering result is a rendering result corresponding to the first dynamic semi-transparent object.

[0215] This application can obtain a separately drawn dynamic semi-transparent object texture and a main scene pipeline texture for the rendering instruction stream corresponding to each frame of the image; then, the main scene rendering result of the predicted frame is determined based on the rendering results corresponding to the two frames of the main scene pipeline, and the dynamic semi-transparent object rendering result of the predicted frame is determined based on the rendering result of one frame of the dynamic semi-transparent object, and the predicted frame is obtained by combining the two. Here, since the dynamic semi-transparent object is drawn separately, it will not be affected by other superimposed objects. Therefore, the drawing quality of the dynamic semi-transparent object can be improved, and the display effect of the dynamic semi-transparent object in the predicted frame can be improved.

[0216] Combination of the above Figures 1 to 13 The image rendering method provided by the embodiment of the present application is described in detail; Fig.14 It should be understood that the electronic device in the embodiments of the present application can execute the various methods in the embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the aforementioned method embodiments.

[0217] Fig.14 A structural schematic diagram of an electronic device provided by the present application is shown. Fig.14 The dotted line in the figure indicates that the unit or the module is optional. The electronic device 200 can be used to implement the method described in the above method embodiment.

[0218] The electronic device 200 includes one or more processors 201, which can support the electronic device 200 to implement the image rendering method in the method embodiment. The processor 201 can be a general-purpose processor or a special-purpose processor. For example, the processor 201 can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, such as discrete gates, transistor logic devices or discrete hardware components.

[0219] The processor 201 can be used to control the electronic device 200, execute software programs, and process data of the software programs. The electronic device 200 can also include a communication unit 205 to implement input (reception) and output (transmission) of signals.

[0220] For example, the electronic device 200 may be a chip, the communication unit 205 may be an input and / or output circuit of the chip, or the communication unit 205 may be a communication interface of the chip, and the chip may be a component of a terminal device or other electronic devices.

[0221] For another example, the electronic device 200 may be a terminal device, and the communication unit 205 may be a transceiver of the terminal device, or the communication unit 205 may be a transceiver circuit of the terminal device.

[0222] The electronic device 200 may include one or more memories 202 on which a program 204 is stored. The program 204 can be executed by the processor 201 to generate instructions 203, so that the processor 201 executes the method described in the above method embodiment according to the instructions 203.

[0223] Optionally, data may be stored in the memory 202. Optionally, the processor 201 may read data stored in the memory 202. The data may be stored at the same storage address as the program 204, or may be stored at a different storage address than the program 204.

[0224] The processor 201 and the memory 202 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of a terminal device.

[0225] Exemplarily, the memory 202 can be used to store the relevant program 204 of the image rendering method provided in the embodiment of the present application, and the processor 201 can be used to call the relevant program 204 of the image rendering method stored in the memory 202 during video processing to execute the image rendering method of the embodiment of the present application; for example, the application sends a first rendering instruction stream, and the first rendering instruction stream is used to instruct the processor to perform a rendering operation of a first frame image, and the first frame image includes a first main scene and a first dynamic semi-transparent object; the application sends a second rendering instruction stream, and the second rendering instruction stream is used to instruct the processor to perform a rendering operation of a second frame image, and the second frame image includes a second main scene and a second dynamic semi-transparent object; the processor determines a third rendering result based on the first rendering result and the second rendering result; the first rendering result is the rendering result corresponding to the first main scene, and the second rendering result is the rendering result corresponding to the second main scene; the processor mixes the third rendering result and the fourth rendering result to determine a predicted frame image; the fourth rendering result is the rendering result corresponding to the first dynamic semi-transparent object.

[0226] The present application also provides a computer program product, which, when executed by the processor 201, implements the method described in any method embodiment of the present application.

[0227] The computer program product may be stored in the memory 202 , for example, a program 204 , which is finally converted into an executable target file that can be executed by the processor 201 after preprocessing, compiling, assembling, and linking.

[0228] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the method described in any method embodiment of the present application is implemented. The computer program can be a high-level language program or an executable target program.

[0229] The computer-readable storage medium is, for example, memory 202. Memory 202 may be a volatile memory or a nonvolatile memory, or memory 202 may include both a volatile memory and a nonvolatile memory. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0230] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described devices and equipment and the technical effects produced can refer to the corresponding processes and technical effects in the aforementioned method embodiments, and will not be repeated here.

[0231] In several embodiments provided in the present application, the disclosed systems, devices and methods can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not performed. The device embodiments described above are merely schematic, and the division of units is only a logical function division. There may be other division methods in actual implementation, and multiple units or components may be combined or integrated into another system. In addition, the coupling between the units or the coupling between the components may be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical or other forms of connection.

[0232] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0233] In addition, the terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0234] In short, the above is only a preferred embodiment of the technical solution of this application, and is not intended to limit the protection scope of this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application should be included in the protection scope of this application.

Claims

1. An image rendering method, characterized in that: Applied to an electronic device having an application installed thereon, the method comprises: The application sends a first rendering instruction stream, where the first rendering instruction stream is used to instruct the electronic device to perform a rendering operation on a first frame of image, where the first frame of image includes a first main scene and a first dynamic semi-transparent object; The application sends a second rendering instruction stream, where the second rendering instruction stream is used to instruct the electronic device to perform a rendering operation on a second frame of image, where the second frame of image includes a second main scene and a second dynamic semi-transparent object; The electronic device determines a third rendering result according to the first rendering result and the second rendering result; the first rendering result is a rendering result corresponding to the first main scene, and the second rendering result is a rendering result corresponding to the second main scene; The electronic device mixes the third rendering result and the fourth rendering result to determine a predicted frame image, or; The electronic device mixes the third rendering result and the fifth rendering result to determine the predicted frame image, or; The electronic device determines a sixth rendering result according to the fourth rendering result and the fifth rendering result; the electronic device mixes the third rendering result and the sixth rendering result to determine the predicted frame image; The fourth rendering result is a rendering result corresponding to the first dynamic semi-transparent object, and the fifth rendering result is a rendering result corresponding to the second dynamic semi-transparent object.

2. The method according to claim 1, characterized in that The first rendering instruction stream includes a first instruction stream and a second instruction stream, the first instruction stream is used to instruct the electronic device to render the first main scene to obtain the first rendering result, and the second instruction stream is used to instruct the electronic device to render the first dynamic semi-transparent object to obtain the fourth rendering result; Before the electronic device mixes the third rendering result and the fourth rendering result, the method further includes: The electronic device performs rendering according to the first instruction stream to obtain the first rendering result, and stores the first rendering result in a first frame buffer, wherein the first instruction stream includes instructions pointing to the first frame buffer; The electronic device performs rendering according to the second instruction stream to obtain the fourth rendering result, and stores the fourth rendering result in a second frame buffer, wherein the second instruction stream includes instructions pointing to the second frame buffer, and the second frame buffer is different from the first frame buffer.

3. The method according to claim 2, characterized in that Before the electronic device performs rendering according to the second instruction stream to obtain the fourth rendering result, the method further includes: the electronic device creates the second frame buffer.

4. The method according to claim 2 or 3, characterized in that: When the electronic device performs rendering, it determines that what is drawn is a semi-transparent object based on the preset first instruction and second instruction in the second instruction stream.

5. The method according to any one of claims 2 to 4, characterized in that When the electronic device performs rendering, it determines that what is being drawn is a dynamic object based on the third instruction, the fourth instruction and the fifth instruction included in the second instruction stream.

6. The method according to any one of claims 1 to 5, characterized in that The first frame image is the Nth frame image, the second frame image is the N+2th frame image, and the predicted frame image is the N+1th frame image, or; The first frame image is the Nth frame image, the second frame image is the N+1th frame image, and the predicted frame image is the N+2th frame image.

7. An electronic device, characterized in that: include: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method as claimed in any one of claims 1 to 6.

8. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by an electronic device, the electronic device executes the method according to any one of claims 1 to 6.

Images