Image rendering method and apparatus, and electronic device and computer-readable medium

Abstract

The present application relates to the technical field of mobile terminals. Disclosed are an image rendering method and apparatus, and an electronic device and a computer-readable medium. The method comprises: a first thread executing, by means of an image processing unit, a rendering operation on a plurality of pieces of image data to be rendered; a second thread executing a frame interpolation operation and a rendering operation on the basis of the plurality of pieces of image data by means of the image processing unit; and displaying a rendered image. Thus, a rendering operation on a plurality of pieces of image data to be rendered, and a frame interpolation operation and a rendering operation which are performed on the basis of the image data are executed by means of two different threads, such that the frame interpolation operation does not interfere with a native image rendering operation, the frame interpolation efficiency is very high, and a frame rate is very stable.

Description

Image rendering method, device, electronic device and computer-readable medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 202410373289.5, filed with the Chinese Patent Office on March 28, 2024, entitled “Image rendering method, device, electronic device and computer-readable medium,” the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of mobile terminal technology, and more specifically, to an image rendering method, device, electronic device, and computer-readable medium. Background Art

[0004] Currently, game interpolation technology is gaining popularity, mostly implemented using plug-in chips. These chips connect the phone's SoC and display via the MIPI interface. The interpolation chip receives one or more image frames from the SoC, runs an interpolation algorithm, and generates new frames. Depending on the interpolation type, these new frames are then sent to the display in a specific order, along with the original frames, to achieve the interpolation effect. Summary of the Invention

[0005] This application proposes an image rendering method, device, electronic device and computer-readable medium to improve the above-mentioned defects.

[0006] In a first aspect, the present application provides an image rendering method, comprising: a first thread performs a rendering operation on a plurality of image data to be rendered through an image processing unit; a second thread performs an interpolation rendering operation based on the plurality of image data through the image processing unit; and the rendered image is displayed.

[0007] In a second aspect, the present application further provides an image rendering device, comprising: a first rendering unit, a second rendering unit, and a display unit. The first rendering unit is configured to cause a first thread to perform a rendering operation on a plurality of image data to be rendered via an image processing unit. The second rendering unit is configured to cause a second thread to perform an interpolation rendering operation based on the plurality of image data via the image processing unit. The display unit is configured to display the rendered image.

[0008] In a third aspect, the present application also provides an electronic device comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the above method.

[0009] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] FIG1 shows a flowchart of an image rendering method provided by an embodiment of the present application;

[0012] FIG2 is a schematic diagram showing an extrapolation frame operation provided by an embodiment of the present application;

[0013] FIG3 is a schematic diagram showing an interpolation frame operation provided by an embodiment of the present application;

[0014] FIG4 shows a flowchart of an image rendering method provided by another embodiment of the present application;

[0015] FIG5 shows an interaction diagram corresponding to the image rendering method provided by an embodiment of the present application;

[0016] FIG6 shows a flowchart of an image rendering method provided by another embodiment of the present application;

[0017] FIG7 shows an interaction diagram corresponding to an image rendering method provided by another embodiment of the present application;

[0018] FIG8 shows a module block diagram of an image rendering device provided by an embodiment of the present application;

[0019] FIG9 shows a structural block diagram of an electronic device provided in an embodiment of the present application;

[0020] FIG10 shows a storage unit for storing or carrying program codes for implementing the method according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work fall within the scope of protection of the present application.

[0022] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0023] In recent years, significant progress has been made in the development of image interpolation technology for gaming. Currently, the recently emerging game interpolation technology on Android mobile phones is mostly implemented using plug-in chips. These chips are connected between the phone's SoC and the display via the MIPI interface. The interpolation chip receives one or more image frames from the SoC, runs an interpolation algorithm, and generates new frames. Depending on the interpolation type, these frames are then sent to the display in a specific order, along with the original frames, to achieve the interpolation effect. Commonly used interpolation solutions include the Iris interpolation solution and the AFME interpolation solution.

[0024] The Iris interpolation solution treats the composited image of the game and system applications as a single entity, without distinguishing which areas of the composite image originate from the game application and which from the system application. This is a purely display backend technology. The supporting software solution focuses on modifying modules related to display services and does not require internal game information, making it relatively easy to implement. The algorithm is embedded in a dedicated chip, resulting in acceptable power consumption. The AFME interpolation solution is a GPU-based interpolation algorithm that does not require an external chip. It is packaged as a GLES extension and can be used wherever GLES and its supporting drivers are available.

[0025] Then, the inventors discovered during their research that the above-mentioned interpolation schemes have some shortcomings. Specifically, the Iris interpolation scheme has been deployed on mobile platforms due to its simple deployment and controllable power consumption. But its shortcomings are also very obvious: the Iris scheme relies on the pw chip and has poor versatility; when interpolating at low frame rates (such as 30fps into 60fps), the power consumption is high due to the chip's low current; the algorithm cannot utilize the 3D information inside the game and can only perform 2D image processing, resulting in mediocre interpolation effects. The AFME interpolation scheme is a pure algorithm implementation, not an end-to-end game interpolation scheme. In order to run in mobile game scenarios, a supporting software framework needs to be developed. In addition, it is based on the characteristics of GPU hardware. Although it has higher programming flexibility, its power consumption performance is inferior to that of dedicated chips.

[0026] Therefore, in order to overcome the above-mentioned defects, an embodiment of the present application provides an image rendering method, as shown in FIG1 , the method includes: S101 to S103 .

[0027] S101: A first thread performs a rendering operation on a plurality of image data to be rendered through an image processing unit.

[0028] In an embodiment of the present application, the multiple image data to be rendered can be the initial image data corresponding to the multiple images to be rendered by the specified application. For example, the specified application is a game application, and the display of the game scene is usually achieved by rendering the image frame by frame. In the game, the smoothness and continuity of the picture are crucial to the player's experience, so the game engine will render the scene at a speed of multiple frames per second (usually in FPS (Frames Per Second)) and output each frame of the image to the screen. The game engine will perform operations such as scene updates, object position calculations, lighting effect calculations, collision detection, etc. in each frame, and finally generate a complete frame of image. These images are continuously displayed on the screen in a certain order and rate, forming a dynamic game screen. By rendering the image frame by frame, the game can present a vivid and smooth picture effect.

[0029] Then the multiple image data to be rendered refer to the multiple image data corresponding to the video of the game scene. The first thread is a thread for rendering the screen of the game scene of the game application, and can also be named the game native rendering thread. The game native rendering thread (Game Native Rendering Thread) refers to the thread in the game engine that is responsible for performing rendering operations. It is usually an independent thread that is specifically used to handle tasks related to graphics rendering, including geometry processing, lighting calculations, texture mapping, shader calculations, etc. The game native rendering thread plays a key role in the rendering pipeline of the game engine. It is responsible for converting the models, textures, effects, etc. in the scene into the final image output. In other words, if the interpolation operation is not performed on the game application, the picture rendered by the first thread can be directly rendered, thereby completing the rendering of the game scene picture. After the rendered image is displayed, the user can smoothly see the picture of the game scene.

[0030] It can be understood that the first thread performs frame-by-frame rendering operations on multiple image data to be rendered, and obtains a rendered image corresponding to each image data, which is named the first image. It should be noted that the first image does not refer to the number or order of images, but refers to the rendered image obtained based on the first thread.

[0031] As an embodiment, the image processing unit is used to perform a rendering operation on the image data currently to be rendered. The image processing unit can be a central processing unit (CPU) or a graphics processing unit (GPU), that is, the rendering operation in the embodiment of the present application can be performed by the CPU or the GPU. In CPU rendering, the game engine executes rendering commands and processes graphics data by using the computing power of the CPU. This method is suitable for simple 2D games or scenes that do not require high graphics performance. CPU rendering can use a software renderer (such as OpenGL software implementation) for drawing, but its efficiency is low and usually cannot meet the needs of complex 3D games. In GPU rendering, the game engine executes rendering commands and processes graphics data by utilizing the parallel computing power of the GPU. The GPU has a large number of processing cores and a dedicated graphics processing unit, which can efficiently perform graphics computing tasks. Modern games generally use GPU rendering because it can handle more complex 3D scenes and provide higher rendering performance and visual effects. In the embodiment of the present application, the image processing unit can be a GPU.

[0032] S102: The second thread performs an interpolation rendering operation based on the plurality of image data through an image processing unit.

[0033] It should be noted that the interpolation rendering operation can be understood as determining at least two image data sets from multiple image data sets to be rendered as a reference image data set, performing an interpolation operation based on this reference image data set to obtain interpolated image data, and then performing a rendering operation on this interpolated image data to obtain a rendered second image. It should be noted that the second image does not refer to the number or order of images, but rather refers to the interpolated and rendered image obtained by the second thread.

[0034] It is understood that in the implementation of S102, the second thread determines at least two image data from the multiple image data to be rendered as a reference image data set, performs a frame interpolation operation based on the reference image data set to obtain interpolated image data, and then performs a rendering operation on the interpolated image data through the image processing unit to obtain a rendered second image. The second thread is an independent thread compared to the first thread, and therefore the frame interpolation operation does not disrupt the operation of the first thread, so the frame interpolation operation does not interfere with the first thread's rendering operation of the multiple image data to be rendered.

[0035] As an implementation, assume that the first thread is the game's native rendering thread and the second thread is the interpolation thread. The interpolation thread can reuse the game's native rendering thread's ANativeWindow. ANativeWindow is a native layer window abstraction interface provided by the Android platform, used for graphics rendering and display in the native layer.

[0036] For example, when interpolation begins, the interpolation thread explicitly dequeues a buffer from ANativeWindow. It then performs the interpolation operation and renders the resulting image data, saving the interpolated result (i.e., the second image) to the buffer. After interpolation is complete, the buffer is explicitly queued back. Specifically, a buffer is explicitly dequeued from ANativeWindow. In Android, ANativeWindow implementations typically maintain a list of buffers available to applications, storing image data to be displayed. Applications can use the dequeueBuffer() function of the ANativeWindow interface to retrieve one of these buffers. Interpolation then proceeds. Interpolation typically involves processing original image frames to generate new ones. For example, in a video player, interpolation can interpolate or repeat original video frames based on parameters such as the current playback rate and display frame rate to generate new ones. The interpolation result is then saved to the buffer. The new image frame generated after interpolation needs to be saved to the previously dequeued buffer. This can be achieved by copying the new image frame data into a buffer and explicitly queueing the buffer. After the interpolated frame result is saved to the buffer, the buffer needs to be explicitly queued so that the ANativeWindow can display it. This can be achieved using the queueBuffer() function of the ANativeWindow interface.

[0037] S103: Displaying the rendered image.

[0038] The first image rendered by the first thread and the second image rendered by the second thread through interpolation are displayed in a specific order. Assume that the multiple images to be rendered correspond to an image queue, and each image to be rendered has a display sequence number within the image queue. If the first images rendered by the first thread are displayed one by one, the first images corresponding to the image data to be rendered should be displayed one by one based on the display sequence number within the image queue. However, if interpolation is performed, the resulting images will disrupt the display order of the images in the image queue.

[0039] Exemplarily, based on a queue of images to be rendered, the first thread sequentially performs rendering operations on each image data to be rendered in the queue, obtaining multiple rendered first images. A second thread retrieves two adjacent images to be rendered from the queue of images to be rendered, using them as a reference image group. The second thread, through an image processing unit, performs an interpolation rendering operation on the reference image group to obtain second images. Based on the type of interpolation operation, the second thread sets the display order of the second images relative to the two image data within the reference image group corresponding to the second images. Each rendered image is displayed one by one based on the display order of each second image and the first image corresponding to the two image data within the reference image group corresponding to the second image. The display order of the display data of each reference image group is set based on the order of the image data within each reference image group within the image queue. If the interpolation operation type is interpolation, the display order of the second image is between the display order of the first images corresponding to the two image data within the reference image group corresponding to the second image. If the interpolation operation type is extrapolation, the display order of the second image is between the reference image group corresponding to the second image and the next reference image group.

[0040] It should be noted that interpolation operations generally include two methods: interpolation and extrapolation. Among them, interpolation refers to generating a new intermediate frame between two known frames, that is, calculating the intermediate frame based on the two existing frames. This method is usually implemented by interpolating the pixel information between the existing frames to fill the gap between the two frames, thereby improving the smoothness and continuity of the picture. Extrapolation refers to generating new frames outside the existing frame sequence, that is, inferring future frames based on the frames that have already been generated. This method usually requires analysis and prediction of the existing frame sequence to infer the content of the future frames. Extrapolation frame technology is often used in applications such as video prediction, motion capture, and motion estimation. It can expand the existing frame sequence and generate longer video sequences.

[0041] As an implementation method, as shown in Figure 2, the game native thread is the aforementioned first thread, which is responsible for the screen rendering of the game scene of the game application and the native rendering logic of the game application. The interpolation thread is the aforementioned second thread, which is responsible for performing interpolation operations based on the game screen data and rendering the image data obtained after interpolation. Among them, Vsync is a synchronization signal used to control the synchronization between the display and the GPU. Vsync can prevent problems such as image tearing and stuttering caused by the lack of synchronization between the display screen frequency and the GPU rendering speed. SurfaceFlinger is an important component in the operating system, responsible for managing and synthesizing the graphics surface (Surface) of the application interface (UI). It is mainly responsible for processing the drawing requests of the application, and synthesizing the various graphics surfaces into the final display image, and then sending it to the display device for display. It can be seen that the first thread and the second thread will hand over the rendered image to SurfaceFlinger for display.

[0042] It should be noted that as shown in Figure 2, the game native thread corresponds to multiple image data to be rendered, such as N-1, N, N+1, N+2, etc., and the interpolation thread corresponds to N+0.5, N+1.5, and N+2.5, which are the images generated by the interpolation operation. It can be seen that the interpolation operation in Figure 2 is an external interpolation operation. For example, the interpolated image N+0.5 is obtained by interpolation based on the N-1 image data to be rendered and the N image data to be rendered. Therefore, based on the logic of content change, the N+0.5 image should be located after the N image data to be rendered. In addition, N-1, N+1, N+0.5, etc. corresponding to SurfaceFlinger represent the images to be displayed for each frame. The order of the images to be displayed by SurfaceFlinger is N-1, N, N+0.5, N+1, N+1.5, N+2, N+2.5, etc. It can be understood that the reference image data group corresponding to N+0.5 is the N-1th image data to be rendered and the Nth image data to be rendered, and the reference image data group corresponding to N+1.5 is the Nth image data to be rendered and the N+1th image data to be rendered. It can be seen that the display order of N+0.5 is between the reference image data group to which it corresponds and the next reference image data group. It can be seen from the serial number that N+0.5 is between N and N+1.

[0043] Similarly, referring to Figure 3, the game native thread corresponds to multiple image data to be rendered, such as N-1, N, N+1, N+2, etc., and the interpolation thread corresponds to N-0.5, N+0.5, and N+1.5, which are multiple images after the interpolation operation that need to be generated. It can be seen that the interpolation operation in Figure 3 is an interpolation operation. For example, the N-0.5th image obtained by interpolation is obtained by interpolation based on the N-1th image data to be rendered and the Nth image data to be rendered. It can also be seen from the sequence number of the interpolated image that N-0.5 is between N-1 and N. In addition, N-1, N-0.5, N, etc. corresponding to SurfaceFlinger represent the images to be displayed for each frame. The order of the images to be displayed corresponding to SurfaceFlinger is N-1, N-0.5, N, N+0.5, N+1, N+1.5, etc. It can be understood that the reference image data group corresponding to N-0.5 is the N-1th image data to be rendered and the Nth image data to be rendered, and the reference image data group corresponding to N+0.5 is the Nth image data to be rendered and the N+1th image data to be rendered. It can be seen that the display order of N+0.5 is between the display orders of the two image data in the reference image data group to which it corresponds. It can be seen from the serial number that N-0.5 is between N-1 and N.

[0044] Therefore, in this embodiment of the present application, a first thread uses the image processing unit to perform a rendering operation on multiple image data to be rendered; a second thread uses the image processing unit to perform a frame insertion rendering operation based on the multiple image data; and the rendered image is displayed. Therefore, the rendering operation on the multiple image data to be rendered and the operation of performing a frame insertion operation and rendering based on the image data are performed by two different threads, so that the frame insertion operation does not interfere with the native image rendering operation, the frame insertion efficiency is very high, and the frame rate is very stable.

[0045] Please refer to FIG. 4 , which shows an image rendering method provided by an embodiment of the present application. The method includes: S401 to S409 .

[0046] S401: A first thread obtains access rights to a first storage space corresponding to target image data to be processed.

[0047] It's important to note that operating systems (such as Android) use a BufferQueue mechanism, which allows for two buffer roles: Producer and Consumer. Producers dequeue a buffer before using it and enqueue it after it's used. Specifically, during graphics rendering, a game process or other graphics rendering program typically acts as a producer. Before using a buffer, a producer must dequeue an available buffer from the BufferQueue and then render image data into that buffer. Once rendering is complete or interpolation is complete, the producer enqueues the used buffer back into the BufferQueue for consumption by the consumer. During graphics rendering, SurfaceFlinger (the window compositor in Android) typically acts as a consumer. SurfaceFlinger retrieves the buffers queued by the producer from the BufferQueue and synthesizes and displays the image data. The consumer is responsible for processing the enqueued buffers to ensure the image is displayed correctly on the screen. In the embodiment of the present application, the aforementioned first thread and second thread both act as Producers, and execute a queue action once rendering is completed or frame insertion is completed.

[0048] Android separates buffer access and usage rights, using fences to ensure data synchronization. Before the application completes drawing in the buffer, it will release the buffer in a certain order according to certain rules. Similarly, the Android display system will receive these buffers in an orderly manner, obtain ownership of the buffer during the next display cycle, perform some preparatory work before sending it to the display, and finally wait for the right to use the buffer until the drawing is complete. When the application completes rendering by calling the GPU through rendering instructions, the GPU triggers the signal fence. At this point, the Android display system receives the signal and finally sends the buffer to the screen for display.

[0049] As an implementation method, the interaction of buffers between the application and the Android display system can be sequential, that is, the buffer sent first is displayed first, and the buffer sent later is displayed later. For example, the display order required by SurfaceFlinger is the image after rendering the N-1th frame, the image after rendering the N-0.5th frame, and the image after rendering the Nth frame. Then the order of enqueuing should also be N-1, N-0.5, and N, so that after enqueuing, SurfaceFlinger can be notified that a certain frame has been enqueued, that is, the order of notifying SurfaceFlinger should also be N-1, N-0.5, and N. Therefore, when the application prepares resources to submit the rendering task to the GPU, a fence is created. When the rendering task is completed, the GPU automatically triggers the signal fence.

[0050] As an implementation method, as mentioned above, a plurality of image data to be rendered corresponds to an image queue, and each image data to be rendered in the image queue corresponds to an order. The first thread usually performs rendering operations on the image data one by one according to the order of each image data to be rendered in the image queue. Usually, after obtaining the access permission to the storage space corresponding to a certain image data, after the rendering operation, the access permission is returned, and then the access permission to the storage space of the next image data is obtained, and so on.

[0051] It can be understood that the target image data refers to the image data on which the first thread is currently performing a rendering operation, for example, the image data currently to be processed determined based on the aforementioned image queue, and the first storage space corresponding to the target image data refers to the storage space where the target image data is stored. After the first thread obtains access rights to the first storage space, it can read the image data in the first storage space, and at the same time, it can also prevent other threads from accessing the target image data.

[0052] S402: Generate a first rendering command and a first barrier identifier corresponding to the first storage space, and send the first rendering command and the first barrier identifier to the image processing unit.

[0053] As an implementation manner, the first rendering command refers to a rendering command submitted by the first thread to the graphics processing unit. Similarly, the first fence identifier is a fence identifier generated by the first thread, wherein the fence identifier is fence.

[0054] In addition, the image processing unit is configured to perform a rendering operation based on a received rendering command, generate a signal fence identifier corresponding to the received fence identifier, and write the signal fence identifier and the rendered image into the storage space corresponding to the received fence identifier. Specifically, after receiving the rendering command and fence identifier, the image processing unit determines the storage space to be used based on the fence identifier and then performs a rendering operation on the image data in the storage space based on the rendering command. Typically, a fence carries information about the rendering command and buffer, enabling the GPU to correctly identify and match the data to be rendered. After the graphics processor completes the rendering operation on the image data corresponding to the storage space, it generates a signal fence identifier corresponding to the received fence identifier and writes the signal fence identifier and the rendered image into the storage space corresponding to the received fence identifier. A signal fence identifier, also known as a signal fence, is a mechanism for synchronizing operations and is typically used to manage dependencies between asynchronous operations in graphics rendering and computation. Signal fences allow programs to wait for the GPU or other device to complete specific operations before continuing with subsequent operations, thereby achieving precise synchronization control. In graphics processing, signal fences are often used to resolve dependencies between multiple graphics rendering tasks. When one operation depends on the result of another operation, you can generate a signal fence in the previous operation and wait for it in the next operation to ensure that the next operation will not be executed until the previous operation is completed. The basic principle of signal fence is that when a signal fence is generated, a signal is sent to the system to indicate that the current operation has been completed; while waiting for a signal fence, the program will suspend execution until it receives a signal from the signal fence, and then continue to execute subsequent operations.

[0055] In addition, the operation of submitting a rendering command can also be called preparing a rendering command. Usually, the operation of preparing a rendering command includes the following steps: setting the rendering target, that is, determining the target to be rendered, such as the screen, texture, etc.; configuring the state of the rendering pipeline, including the vertex shader, fragment shader, depth test, blending mode, etc.; setting the parameters required for rendering, such as the view matrix, projection matrix, lighting parameters, etc.; setting the vertex data, that is, passing the vertex data (vertex coordinates, normals, texture coordinates, etc.) of the object to be rendered to the GPU. In addition, if you need to use a texture for rendering, pass the texture data to the GPU. Set the uniform variables that need to be used in the shader, such as lighting direction, material properties, etc. Finally, submit the drawing command, that is, package all the rendering parameters and data set above into a drawing command and submit it to the GPU for processing.

[0056] It can be seen that the operation of preparing rendering commands takes up more time. Therefore, it is usually necessary to submit the next rendering command after submitting one rendering command.

[0057] S403: Determine whether the frame insertion condition is met.

[0058] If the judgment result of S403 is yes, then execute S406; if the judgment result of S403 is no, then execute S404.

[0059] It should be noted that both external and internal interpolation frame operations require the use of at least two image data. Therefore, after some image data begins rendering, the interpolation operation may not be triggered, while after some image data begins rendering, the interpolation operation will be triggered. For example, as described above, each interpolated image corresponds to a reference image data group, and the second thread will also generate interpolated images one by one based on a certain order. Then, each interpolated image to be generated corresponds to a reference image data group, and the two image data in the reference image data group can be divided into the previous image data and the next image data according to the order of the image queue.

[0060] The method for determining whether the interpolation condition is met is to determine the reference image data group corresponding to the interpolation image currently to be generated, and then determine whether the target image data belongs to the previous image data or the next image data in the reference image data group. If it is the previous image data, the interpolation condition is determined not to be met; if it is the next image data, the interpolation condition is determined to be met. For example, if the interpolation image currently to be generated is N-0.5, and its corresponding reference image data groups are N-1 and N frames of image data, if the first thread is currently requesting to render the N-1 frame of image data, it can be determined that the interpolation condition is not met because the rendering operation for the Nth frame of image data has not yet begun.

[0061] S404: Determine whether the preset conditions are met.

[0062] As an implementation, the preset condition is a condition for the first thread to determine whether it can immediately return access to the first storage space and render subsequent image data. Specifically, two aspects can be considered. On the one hand, if no interpolation operation is currently in progress, then after the rendering command and fence identifier are submitted for the current target image data, the next frame of image data can be rendered directly. Therefore, with respect to this aspect, the preset condition is that the interpolation condition is not met, i.e., there is no need to wait for the second thread to submit a rendering command, and the rendering operation can be performed directly on the next image data, i.e., executing S405 and S406.

[0063] On the other hand, if the interpolation conditions are currently met, it is usually necessary to return the access rights to the first storage space after the fence identifier corresponding to the interpolation operation is submitted, and prepare for the rendering of the next image data, so that SurfaceFlinger can be correctly notified to display the rendered image data in the correct order according to the correct display order. Therefore, if the interpolation conditions are met, whether the preset conditions are met needs to be determined based on the interpolation operation. Please refer to the subsequent steps.

[0064] S405: The first thread gives the access permission of the first storage space to the display unit, and determines the next image data of the target image data as new target image data based on the image queue to be rendered.

[0065] When the preset conditions are met, the first thread gives the access rights to the first storage space to the display unit, determines the next image data of the target image data as the new target image data based on the image queue to be rendered, and returns to execute the operation of the first thread obtaining the access rights to the first storage space corresponding to the current target image data to be processed and subsequent operations.

[0066] It can be understood that when the first thread performs a rendering operation on the target image data, it will first obtain the access permission to the first storage space corresponding to the target image data, and then prepare the rendering command, obtain the first rendering command and the first fence identifier corresponding to the target image data, send the first fence identifier and the first rendering command to the GPU, and then return the access permission to the first storage space, so that the display unit (for example, SurfaceFlinger) can obtain the access permission to the first storage space, and then after the GPU completes rendering the target image data and puts the rendered image into the storage space, obtain the image based on the access permission to the storage space and display it.

[0067] Therefore, after determining that the preset conditions are met, the first thread transfers the access rights to the first storage space to the display unit, and the GPU writes the signal fence identifier and the rendered image to the storage space. After detecting the signal fence identifier in the storage space of the current access rights, the display unit displays the image in the storage space. It should be noted that the display unit obtains the access rights to the storage space, which is related to the order in which the first thread or the second thread returns the access rights to the storage space. Usually, after the first thread or the second thread returns the access rights to the enqueued storage space, it notifies the display unit that the content of the storage space is waiting to be displayed. The display unit then obtains the access rights to the storage space and waits for the signal fence identifier in the storage space.

[0068] S406: The second thread obtains access rights to the second storage space corresponding to the current reference image data group.

[0069] If the interpolation condition is met, S406 is executed, that is, the second thread obtains the access permission of the second storage space corresponding to the current reference image data group, wherein the second storage space stores two image data in the reference image data group, or the second storage space corresponds to the storage path of the two image data in the reference image data group, and the second thread or other accessors can obtain the two image data in the reference image data group through the storage path.

[0070] Specifically, an interpolation queue can be determined based on the image queue, and two image data within the reference image data corresponding to each interpolated image can be determined, thereby determining the execution timing of the interpolation operation corresponding to each interpolated image, i.e., the interpolation condition. The interpolation condition is satisfied when a first rendering command and a first barrier identifier for the latter of two image data within a reference image data set are detected to be sent to the image processing unit, wherein the reference image data set includes two adjacent image data within the image queue to be rendered.

[0071] S407: After detecting the signal fence identifier of two image data in the reference image data group, the second thread generates a second rendering command and a second fence identifier corresponding to the second storage space based on the reference image data group and the interpolation strategy, and sends the second rendering command and the second fence identifier to the image processing unit.

[0072] In one embodiment, after the second thread sends the second rendering command and the second fence identifier to the image processing unit, since in the fence mechanism, if any party receives the fence identifier, all participating parties can also receive the fence identifier. That is, after the second fence identifier is generated, regardless of whether it is sent to the image processing unit, the first thread, the second thread, the graphics processing unit, and the display unit can all detect the second fence identifier. Therefore, the first thread can detect the second fence identifier and determine that the rendering command corresponding to the second fence identifier has been completed. The first thread can then return access rights to the first storage space, i.e., the preset condition is currently met. Therefore, if the interpolation condition is met, the preset condition can be that the second fence identifier corresponding to the currently pending interpolation rendering operation has been generated, i.e., the second fence identifier corresponding to the current interpolation operation has been generated. All that remains is for the GPU to generate a signal fence identifier corresponding to the second fence identifier after rendering is completed, and for the graphics processing unit to be able to obtain the signal fence identifier when it is waiting for the signal fence identifier. Therefore, the first thread can be triggered to continue rendering the next image data.

[0073] S408: The second thread gives the access permission of the second storage space to the display unit.

[0074] S409: Determine the next image data group of the reference image data group as a new reference image data group based on the image queue to be rendered.

[0075] Exemplarily, the interpolated frame is taken as an interpolated frame, and the above embodiment is described using the interaction scheme shown in FIG5 .

[0076] As shown in Figure 5, the game application includes a first thread and a second thread. BufferQueue is a mechanism used in the Android system to manage image data buffers and can be referred to as a buffer management unit. It is typically used to transfer image data between producers and consumers to enable image display or processing. Specifically, BufferQueue provides a buffer queue containing multiple image buffers and a set of interfaces and methods to manage the allocation, filling, display, and release of these buffers. In the Android system, operations involving images (such as camera preview, video playback, and graphics rendering) may all utilize BufferQueue. Producers (such as cameras and video decoders) fill the generated image data into buffers, which are then passed to consumers (such as SurfaceFlinger and image renderers) via BufferQueue, enabling the transfer and display of image data.

[0077] The scheme shown in Figure 5 includes the rendering of two image data to be rendered and the generation of an interpolated image, namely, the rendering of the 1st frame image and the 2nd frame image, and the 1.5th frame image, wherein the 1.5th frame image is an image obtained based on the interpolation operation of the 1st frame image and the 2nd frame image, the storage space corresponding to the 1st frame image is framebuffer1 and the corresponding fence identifier is fence1, the storage space corresponding to the 2nd frame image is framebuffer2 and the corresponding fence identifier is fence2, and the storage space corresponding to the 1.5th frame image is framebuffer1.5 and the corresponding fence identifier is fence1.5.

[0078] For example, the first thread dequeues framebuffer1 from bufferqueue and obtains access rights to framebuffer1. It then prepares rendering commands corresponding to framebuffer1 and submits the rendering commands and fence1 to the GPU, triggering the GPU to render the image data in framebuffer1. The first thread dequeues framebuffer1 and returns access rights to framebuffer1 to bufferqueue. Bufferqueue then notifies surfaceflinger of the new framebuffer1, allowing surfaceflinger to know that an image is currently being displayed and that it can retrieve the image from framebuffer1.

[0079] Surfaceflinger acquires framebuffer1 from bufferqueue, thereby obtaining access rights to framebuffer1. At this point, the first thread transfers access rights to framebuffer1 to surfaceflinger and sends fence1 to surfaceflinger through framebuffer1, so that surfaceflinger knows that framebuffer1 corresponds to fence1. Next, surfaceflinger waits for signal fence1. The GPU executes the rendering command, and after rendering is completed, it stores the rendered image in framebuffer1 and writes the signal fence identifier corresponding to fence1, signal fence1, to framebuffer1. Surfaceflinger reads signal fence1 based on framebuffer1, knows that the image corresponding to fence1 has been rendered, and then displays the image in framebuffer1, that is, sends framebuffer1 to display.

[0080] Next, since no images currently require interpolation, it can be determined that the preset conditions are met. The first thread then dequeues framebuffer2 from the bufferqueue and obtains access rights to framebuffer2. The first thread then prepares rendering commands corresponding to framebuffer2 and submits the rendering commands and fence2 to the GPU, triggering the GPU to perform rendering operations on the image data in framebuffer2. At this point, it can be determined that the interpolation conditions are met, i.e., the two images in the current corresponding reference image data group are the rendering of frame 1 and frame 2, and the rendering commands and fence2 for the latter image data (i.e., frame 2) are sent to the GPU. The second thread then dequeues framebuffer1.5 from the bufferqueue and obtains access rights to framebuffer1.5. The second thread then waits for signal fence1 and signal fence2. After signal fence1 and signal fence2 are received, the second thread prepares rendering commands corresponding to framebuffer1.5 and submits the rendering commands and fence1.5 to the GPU, triggering the GPU to perform interpolation and rendering operations on the image data in framebuffer1.5.

[0081] It should be noted that no matter which thread or unit triggers the fence or signal fence, other threads or units can also, for example, the second thread obtains the signal fence1 and signal fence2 output by the GPU, and surfaceflinger can also know the signal fence1 and signal fence2.

[0082] After the second thread submits the rendering command and fence1.5 to the GPU, the first thread queues framebuffer2 and the second thread queues framebuffer1.5. It should be noted that the timing of the second thread queuing framebuffer1.5 can be after the second thread submits the rendering command and fence1.5 to the GPU and before the first thread queues framebuffer2, or after the first thread queues framebuffer2. In this embodiment of the present application, if the type of the interpolation operation is an interpolation type, then before the access permission of the first storage space corresponding to the latter of the two image data in the current reference image data group is handed over to the display unit, the second thread hands over the access permission of the second storage space to the display unit.

[0083] After that, bufferqueue notifies surfaceflinger of the new framebuffer1.5 and the new framebuffer2 in turn, and surfaceflinger displays the images obtained by the interpolation and rendering operations in framebuffer1.5 and the rendered images in framebuffer2 in turn.

[0084] Therefore, the fence synchronization mechanism provided in the embodiment of the present application can ensure that the interpolation operation is performed after the rendering operation of the two image data in its corresponding reference image data group is completed, and it can also ensure that surfaceflinger performs the display operation after the image currently requested to be displayed is completed, thereby avoiding display freezes.

[0085] Please refer to FIG. 6 , which shows an image rendering method provided by an embodiment of the present application. The method includes: S601 to S609 .

[0086] S601: A first thread obtains access rights to a first storage space corresponding to target image data to be processed.

[0087] S602: Generate a first rendering command and a first barrier identifier corresponding to the first storage space, and send the first rendering command and the first barrier identifier to the image processing unit.

[0088] S603: Determine whether the frame insertion condition is met.

[0089] S604: Determine whether the preset conditions are met.

[0090] S605: The first thread gives the access permission of the first storage space to the display unit, and determines the next image data of the target image data as new target image data based on the image queue to be rendered.

[0091] S606: The second thread obtains access rights to the second storage space corresponding to the current reference image data group.

[0092] S607: Before a signal barrier identifier corresponding to the latter image data among the two image data in the reference image data group is generated, generate a second barrier identifier corresponding to the second storage space.

[0093] As shown in the example shown in Figure 4 , buffer interactions between the application and the Android display system occur sequentially. Buffers sent first are displayed first, followed by buffers sent later. When the application prepares resources and submits a rendering task to the GPU, a fence is created. When the rendering task is complete, the GPU automatically triggers a signal fence. Fence creation and signaling are impossible to manually intervene in.

[0094] Typically, to ensure buffer synchronization between the game process and the SurfaceFlinger synthesis process, the game process needs to create a fence before the rendering resources are ready, and then relinquish access to the buffer. To this end, the present embodiment provides a preset software fence module. The implementation of S607 is to apply to the preset software fence module for obtaining the second fence identifier corresponding to the second storage space before the signal fence identifier corresponding to the latter of the two image data in the reference image data group is generated.

[0095] Exemplarily, the software fence (sw_fence) module is a file descriptor containing a set of sync_pts (synchronization time points). Each sync_pt represents a point on the sync_timeline, indicating when an operation has completed. Specifically, in the Linux kernel, sw_fence (software barrier) is essentially a file descriptor. This mechanism allows different components or threads to synchronize when performing asynchronous operations, thereby coordinating their work. In graphics rendering or other asynchronous operations, sw_fence can be used to indicate the completion status of an operation, allowing other components to perform corresponding processing based on this status. Because sw_fence is represented by a file descriptor, it can be operated on using file descriptor-related system calls, such as wait and notify. This design allows sw_fence to be easily combined with other synchronization mechanisms to implement complex synchronization operations. sw_fence can be passed between kernel space and user space, as well as between different drivers and processes.

[0096] That is, for the second thread, each frame insertion operation is a synchronization time point, and multiple frame insertion operations are distributed on the synchronization timeline in sequence. After one frame insertion operation is completed, the next frame insertion operation starts to be executed.

[0097] It can be seen from the above-mentioned embodiments that the second thread needs to generate the second rendering command and the second fence identifier corresponding to the second storage space after the signal fence identifier corresponding to the latter image data in the two image data in the reference image data group is generated, that is, after the signal fence identifiers corresponding to the two image data in the reference image data group are generated, and the first thread needs to wait until the signal fence identifier corresponding to the second fence identifier is generated before it can execute subsequent rendering operations. It takes a certain amount of time for the second thread to prepare the second rendering command corresponding to the second storage space. During this period, the first thread can only be in a waiting state.

[0098] In combination with the above content, it can be seen that in the above embodiment, the first thread needs to wait until the second fence identifier corresponding to the interpolation operation is generated before returning the first storage space of the target image data currently being operated on, so as to perform the rendering operation on the next image data. In the embodiment of the present application, the second fence identifier corresponding to the current interpolation operation is generated in advance before the signal fence identifier corresponding to the latter image data of the two image data in the reference image data group is generated. This allows the first thread to return the first storage space of the target image data currently being operated on in advance before the second thread has issued the rendering command corresponding to the current interpolation operation, thereby performing the rendering operation on the next image data in advance. As a result, the first thread can perform the rendering operation on the next image data while the second thread is preparing the current second rendering command.

[0099] It should be noted that the second barrier identifier corresponding to the second storage space generated in S607 differs from the second barrier identifier in the aforementioned embodiment in that the second barrier identifier currently does not correspond to a second rendering command, i.e., the second rendering command has not yet been generated. However, as mentioned above, even if the second thread has not yet generated the second rendering command and therefore it is not currently appropriate to send the second barrier identifier to the GPU, the first thread can still detect that the second barrier identifier has been generated, and thus can still trigger the first thread to execute the operation in S605, i.e., can determine that the preset condition is currently met.

[0100] S608: The second thread gives the access permission of the second storage space to the display unit.

[0101] S609: Generate a second rendering command corresponding to the second storage space based on the reference image data group and the interpolation strategy, and send the second rendering command and the second barrier identifier to the image processing unit.

[0102] It should be noted that, unlike the aforementioned embodiment, the second thread gives the access rights to the second storage space to the display unit before the second rendering command is submitted to the GPU. This is because the second fence identifier has been generated at this time, that is, the display unit already knows that there is an image in the second storage space corresponding to the second fence identifier that needs to be displayed. For the display unit, the correct sorting of the content to be displayed can be completed, and it only needs to wait patiently for the signal fence identifier corresponding to the second fence identifier.

[0103] After the second rendering command is prepared, the second thread sends the second rendering command and the second barrier identifier to the image processing unit, so that the GPU can perform an interpolation rendering operation based on the second rendering command and the second barrier identifier.

[0104] S610: Determine the next image data group of the reference image data group as a new reference image data group based on the image queue to be rendered.

[0105] Exemplarily, the interpolated frames are used as interpolated frames, and the above embodiment is described with reference to the interaction scheme shown in Figure 7. As shown in Figure 7, the game application includes a first thread and a second thread.

[0106] The scheme shown in Figure 7 includes the rendering of two image data to be rendered and the generation of an interpolated image, namely, the rendering of the 1st frame image and the 2nd frame image, and the 1.5th frame image, wherein the 1.5th frame image is an image obtained based on the interpolation operation of the 1st frame image and the 2nd frame image, the storage space corresponding to the 1st frame image is framebuffer1 and the corresponding fence identifier is fence1, the storage space corresponding to the 2nd frame image is framebuffer2 and the corresponding fence identifier is fence2, and the storage space corresponding to the 1.5th frame image is framebuffer1.5 and the corresponding fence identifier is fence1.5.

[0107] The first thread dequeues framebuffer1 from bufferqueue, obtaining access to framebuffer1. It then prepares rendering commands for framebuffer1 and submits the rendering commands and fence1 to the GPU, triggering the GPU to render the image data in framebuffer1. The first thread dequeues framebuffer1 and returns access to framebuffer1 to bufferqueue. Bufferqueue notifies surfaceflinger of the new framebuffer1, allowing surfaceflinger to know that an image is available for display and can be retrieved from framebuffer1.

[0108] Surfaceflinger acquires framebuffer1 from bufferqueue, thereby obtaining access rights to framebuffer1. At this point, the first thread transfers access rights to framebuffer1 to surfaceflinger and sends fence1 to surfaceflinger through framebuffer1, so that surfaceflinger knows that framebuffer1 corresponds to fence1. Next, surfaceflinger waits for signal fence1. The GPU executes the rendering command, and after rendering is completed, it stores the rendered image in framebuffer1 and writes the signal fence identifier corresponding to fence1, signal fence1, to framebuffer1. Surfaceflinger reads signal fence1 based on framebuffer1, knows that the image corresponding to fence1 has been rendered, and then displays the image in framebuffer1, that is, sends framebuffer1 to display.

[0109] Next, since there is no image that currently requires interpolation, it can be determined that the preset conditions are met, and the first thread then dequeues framebuffer2 from the bufferqueue to obtain access rights to framebuffer2. Then, the first thread prepares the rendering command corresponding to framebuffer2, and then submits the rendering command and fence2 to the GPU, triggering the GPU to perform the rendering operation of the image data in framebuffer2. At this point, it can be determined that the interpolation conditions are met, that is, the two images in the current corresponding reference image data group are the rendering of the first frame image and the second frame image, and the rendering command and fence2 of the latter (i.e., the second frame image) image data are sent to the GPU, then the second thread dequeues framebuffer1.5 from the bufferqueue to obtain access rights to framebuffer1.5.

[0110] The second thread applies for the second fence identifier from sw_fence, that is, sw-fence1.5, and obtains sw-fence1.5 returned by sw_fence. This allows the first thread to know that sw-fence1.5 has been generated. Then the second thread queues framebuffer1.5, and the first thread queues framebuffer2. The second thread prepares the rendering commands corresponding to framebuffer1.5, and submits the rendering commands and fence1.5 to the GPU. The GPU executes the rendering commands corresponding to fence1.5, and performs interpolation and rendering operations. In addition, after the second thread queues framebuffer1.5 and the first thread queues framebuffer2, bufferqueue notifies surfaceflinger that there is a new framebuffer1.5. Surfaceflinger knows that the image to be displayed is in framebuffer1.5, and thus waits for signal fence1.5.

[0111] After the GPU completes the rendering command for fence1.5, it obtains the interpolated image corresponding to the current interpolation operation. The second thread then receives signal fence1.5 from the GPU. Surfaceflinger also detects signal fence1.5 and displays the interpolated image in framebuffer1.5. Furthermore, the second thread requests signal sw_fence1.5 from sw_fence, notifying sw_fence of the completion of fence1.5. Sw_fence then sends signal fence1.5 to surfaceflinger, allowing it to subsequently execute the next synchronization point in fence1.5, such as fence2.5.

[0112] Finally, after signal fence 1.5 is generated, bufferqueue notifies surfaceflinger that there is a new framebuffer2, and surfaceflinger performs subsequent display operations on the image in framebuffer2.

[0113] Therefore, in an embodiment of the present application, before the rendering command corresponding to the current interpolation operation is generated, the second thread can generate the corresponding second fence identifier in advance, so that the first thread can determine that the preset condition is currently met, and then the first thread can return the access rights to the first storage space corresponding to the target image data of the current operation in advance, so that the rendering operation of the next image data can be executed in advance.

[0114] Therefore, in the rendering thread, the CPU no longer waits for the GPU to complete; the rendering thread immediately returns to the game application, leaving the waiting process for the current frame buffer to the interpolation thread. SurfaceFlinger uses Android's fence mechanism to ensure compositing synchronization. The key to reducing interpolated frame latency lies in ensuring that the eglSwapBuffers function in the game rendering thread returns as soon as possible, allowing the game to regain CPU control and begin computing the next frame. Buffer synchronization is now performed during layer compositing, leveraging the SurfaceFlinger process's sync_wait buffer wait mechanism.

[0115] It's important to note that interpolation is achieved through a new thread, with intermediate frame generation occurring within the game process. This differs from solutions based on SurfaceFlinger and the PixelWorks approach, which relies on external chips. By leveraging Android's fence mechanism and wait-free buffer multi-threaded scheduling, low-latency frame delivery during in-game interpolation is achieved without modifying the Android SurfaceFlinger code.

[0116] Please refer to FIG8 , which shows a structural block diagram of an image rendering device provided by an embodiment of the present application. The device may include: a first rendering unit 801 , a second rendering unit 802 , and a display unit 803 .

[0117] The first rendering unit 801 is configured to execute a rendering operation on a plurality of image data to be rendered by a first thread through an image processing unit.

[0118] Furthermore, the first rendering unit 801 is also used to obtain access rights to a first storage space corresponding to the target image data currently to be processed; generate a first rendering command and a first fence identifier corresponding to the first storage space, and send the first rendering command and the first fence identifier to the image processing unit, wherein the image processing unit is used to perform a rendering operation based on the received rendering command and generate a signal fence identifier corresponding to the received fence identifier, and write the signal fence identifier and the rendered image into the storage space corresponding to the received fence identifier; when preset conditions are met, the first thread hands over the access rights to the first storage space to the display unit, determines the next image data of the target image data as the new target image data based on the image queue to be rendered, and returns to execute the operation of the first thread obtaining access rights to the first storage space corresponding to the target image data currently to be processed and subsequent operations, wherein the display unit is used to display the image of the storage space after detecting the signal fence identifier in the storage space of the current access rights.

[0119] The second rendering unit 802 is configured to execute, by a second thread, an interpolation rendering operation based on the plurality of image data through an image processing unit.

[0120] Furthermore, the second rendering unit 802 is also used to, when the interpolation condition is met, enable the second thread to obtain access rights to the second storage space corresponding to the current reference image data group, wherein meeting the interpolation condition includes detecting that the first rendering command and the first fence identifier of the latter image data in the two image data in the reference image data group are sent to the image processing unit, and the reference image data group includes two adjacent image data in the image queue to be rendered; after detecting the signal fence identifier of the two image data in the reference image data group, the second thread generates a second rendering command and a second fence identifier corresponding to the second storage space based on the reference image data group and the interpolation strategy, and sends the second rendering command and the second fence identifier to the image processing unit; the second thread gives the access rights to the second storage space to the display unit; determines the next image data group of the reference image data group as the new reference image data group based on the image queue to be rendered, and returns to execute the operation of obtaining access rights to the second storage space corresponding to the current reference image data group and subsequent operations when the interpolation condition is met.

[0121] Furthermore, the second rendering unit 802 is also used to, when an interpolation condition is met, enable the second thread to obtain access rights to a second storage space corresponding to the current reference image data group, wherein meeting the interpolation condition includes detecting that a first rendering command and a first fence identifier of the latter image data among the two image data in the reference image data group are sent to the image processing unit, and the reference image data group includes two adjacent image data in the image queue to be rendered; before a signal fence identifier corresponding to the latter image data among the two image data in the reference image data group is generated, a second fence identifier corresponding to the second storage space is generated; the second thread transfers access rights to the second storage space to the display unit; generates a second rendering command corresponding to the second storage space based on the reference image data group and the interpolation strategy, and sends the second rendering command and the second fence identifier to the image processing unit; determines the next image data group of the reference image data group as a new reference image data group based on the image queue to be rendered, and returns to execute the operation of obtaining access rights to the second storage space corresponding to the current reference image data group and subsequent operations when the interpolation condition is met.

[0122] Furthermore, the preset condition includes that a second barrier identifier corresponding to the currently to-be-executed interpolation rendering operation is generated.

[0123] Furthermore, the second rendering unit 802 is further configured to apply to a preset software barrier module for obtaining a second barrier identifier corresponding to the second storage space before a signal barrier identifier corresponding to the latter image data in the reference image data group is generated.

[0124] Furthermore, the second rendering unit 802 is also used to, if the type of the interpolation operation is an interpolation type, before the access permission of the first storage space corresponding to the latter image data of the two image data in the current reference image data group is handed over to the display unit, the second thread hands over the access permission of the second storage space to the display unit.

[0125] The display unit 803 is used to display the rendered image.

[0126] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0127] In several embodiments provided in this application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0128] In addition, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or software functional modules.

[0129] Please refer to Figure 9, which shows a structural block diagram of an electronic device provided in an embodiment of the present application. The electronic device 100 can be an electronic device capable of running applications, such as a smartphone, a tablet computer, or an e-book. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0130] The processor 110 may include one or more processing cores. The processor 110 utilizes various interfaces and circuits to connect various components within the electronic device 100. It executes instructions, programs, code sets, or instruction sets stored in the memory 120, and accesses data stored in the memory 120 to perform various functions and process data within the electronic device 100. Optionally, the processor 110 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 110 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 110 and may be implemented separately via a communication chip.

[0131] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the electronic device 100 during use (such as a phone book, audio and video data, chat history data), etc.

[0132] Please refer to Figure 10, which shows a block diagram of a computer-readable medium provided in an embodiment of the present application. The computer-readable medium 1000 stores program code, which can be called by a processor to execute the method described in the above method embodiment.

[0133] The computer-readable medium 1000 can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable medium 1000 includes a non-transitory computer-readable storage medium. The computer-readable medium 1000 has storage space for program code 1010 for executing any of the method steps described above. These program codes can be read from or written to one or more computer program products. The program code 1010 can be compressed, for example, in a suitable form.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image rendering method, characterized in that: include: The first thread performs a rendering operation on a plurality of image data to be rendered through an image processing unit; The second thread performs an interpolation rendering operation based on the plurality of image data through the image processing unit; Display the rendered image.

2. The method according to claim 1, characterized in that The first thread performs a rendering operation on a plurality of image data to be rendered, including: The first thread obtains access rights to the first storage space corresponding to the target image data to be processed currently; generating a first rendering command and a first barrier identifier corresponding to the first storage space, and sending the first rendering command and the first barrier identifier to the image processing unit, wherein the image processing unit is configured to perform a rendering operation based on the received rendering command and generate a signal barrier identifier corresponding to the received barrier identifier, and write the signal barrier identifier and the rendered image into the storage space corresponding to the received barrier identifier; When the preset conditions are met, the first thread gives the access permission of the first storage space to the display unit, determines the next image data of the target image data as the new target image data based on the image queue to be rendered, and returns to execute the operation of the first thread obtaining the access permission of the first storage space corresponding to the target image data to be processed and subsequent operations, wherein the display unit is used to display the image of the storage space after detecting the signal fence identifier in the storage space of the current access permission.

3. The method according to claim 2, characterized in that The second thread performs an interpolation rendering operation based on the plurality of image data to obtain a rendered image to be interpolated, including: When an interpolation condition is satisfied, the second thread obtains access rights to a second storage space corresponding to the current reference image data group, wherein satisfying the interpolation condition includes detecting that a first rendering command and a first barrier identifier for a later image data of two image data in the reference image data group are sent to the image processing unit, and the reference image data group includes two adjacent image data in the image queue to be rendered; After detecting the signal barrier identifiers of two image data in the reference image data group, the second thread generates a second rendering command and a second barrier identifier corresponding to the second storage space based on the reference image data group and the interpolation strategy, and sends the second rendering command and the second barrier identifier to the image processing unit; The second thread gives the access permission of the second storage space to the display unit; Based on the image queue to be rendered, the next image data group of the reference image data group is determined as a new reference image data group, and the operation of obtaining access rights to the second storage space corresponding to the current reference image data group and subsequent operations are returned to execute when the interpolation condition is met.

4. The method according to claim 2, characterized in that The second thread performs an interpolation rendering operation based on the plurality of image data to obtain a rendered image to be interpolated, including: When an interpolation condition is satisfied, the second thread obtains access rights to a second storage space corresponding to the current reference image data group, wherein satisfying the interpolation condition includes detecting that a first rendering command and a first barrier identifier for a later image data of two image data in the reference image data group are sent to the image processing unit, and the reference image data group includes two adjacent image data in the image queue to be rendered; before a signal barrier identifier corresponding to a latter image data among two image data in the reference image data group is generated, generating a second barrier identifier corresponding to the second storage space; The second thread gives the access permission of the second storage space to the display unit; generating a second rendering command corresponding to the second storage space based on the reference image data group and the interpolation strategy, and sending the second rendering command and the second barrier identifier to the image processing unit; Based on the image queue to be rendered, the next image data group of the reference image data group is determined as a new reference image data group, and the operation of obtaining access rights to the second storage space corresponding to the current reference image data group and subsequent operations are returned to execute when the interpolation condition is met.

5. The method according to claim 2, characterized in that The preset condition includes that a second barrier identifier corresponding to the currently to-be-executed interpolation rendering operation is generated.

6. The method according to claim 3, characterized in that When the interpolation condition is met, the preset condition is that a second barrier identifier corresponding to the interpolation rendering operation to be executed is generated.

7. The method according to claim 3, characterized in that When the interpolation condition is not met, the preset condition is that the target image data belongs to the previous image data in the reference image data group.

8. The method according to claim 4, characterized in that Before a signal barrier identifier corresponding to a later image data in the two image data in the reference image data group is generated, generating a second barrier identifier corresponding to the second storage space includes: Before the signal barrier identifier corresponding to the latter image data in the two image data in the reference image data group is generated, an application is made to a preset software barrier module to obtain a second barrier identifier corresponding to the second storage space.

9. The method according to claim 3 or 4, characterized in that The second thread granting access rights to the second storage space to the display unit includes: If the type of the interpolation operation is an interpolation type, the second thread gives the access right to the second storage space to the display unit before the access right to the first storage space corresponding to the latter image data in the current reference image data group is given to the display unit.

10. The method according to claim 2, characterized in that The fence is identified as fence, and the signal fence is identified as signal fence.

11. The method according to claim 1, wherein The plurality of image data to be rendered correspond to an image queue, each image data to be rendered corresponds to a display sequence number in the image queue, and the first thread performs a rendering operation on the plurality of image data to be rendered through an image processing unit; The second thread performs an interpolation rendering operation based on the plurality of image data through the image processing unit, including: The first thread sequentially performs rendering operations on each of the image data to be rendered in the image queue to obtain a plurality of rendered first images; The second thread obtains two adjacent image data to be rendered that are currently to be processed from the image queue to be rendered as a reference image group; The second thread performs an interpolation rendering operation based on the reference image group through the image processing unit to obtain a second image.

12. The method according to claim 11, characterized in that Displaying the rendered image includes: setting, based on the type of the interpolation operation, a display order of the second image relative to two image data in a reference image group corresponding to the second image; Each rendered image is displayed one by one based on the display order of each second image and the first images corresponding to the two image data in the reference image group corresponding to the second image.

13. The method according to claim 12, characterized in that The setting, based on the type of the interpolation operation, of a display order of the second image relative to two image data in a reference image group corresponding to the second image, includes: If the type of the interpolation operation is interpolation, the display order of the second image is set to be in the middle of the display order of the first images corresponding to the two image data in the reference image group corresponding to the second image.

14. The method according to claim 12, characterized in that The setting, based on the type of the interpolation operation, of a display order of the second image relative to two image data in a reference image group corresponding to the second image, includes: If the type of the interpolation operation is extrapolation, the display order of the second image is set to be between the reference image group corresponding to the second image and the next reference image group.

15. The method according to any one of claims 1 to 14, characterized in that: The first thread and the second thread are independent of each other.

16. The method according to claim 15, characterized in that The first thread is a native game thread, which is used for rendering the game scene of the game application, and the second thread is an interpolation thread.

17. The method according to any one of claims 1 to 14, characterized in that: The plurality of image data to be rendered are image data corresponding to a plurality of images to be rendered by a game application.

18. An image rendering device, characterized in that: include: A first rendering unit, configured for a first thread to perform a rendering operation on a plurality of image data to be rendered through an image processing unit; a second rendering unit, configured for a second thread to perform an interpolation rendering operation based on the plurality of image data through the image processing unit; The display unit is used to display the rendered image.

19. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1 to 17.

20. A computer-readable medium, characterized in that The computer-readable medium stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the method according to any one of claims 1 to 17.

Images