Image rendering method, and related apparatus

Abstract

Disclosed in the present application are an image rendering method, and a related apparatus. During the execution of a first logic frame, first image rendering data of N graphics frames is determined by means of the parallel computation of N threads, such that one logic frame can generate image rendering data for a plurality of graphics frames, such that the rendering of the next graphics frame can be quickly performed after one graphics frame has been rendered, without the need to wait for the completion of the execution of the next logic frame, thereby greatly improving the efficiency of image rendering and increasing the frame rate of graphics frames. In addition, a plurality of pieces of image rendering data can represent rendering data within a time period from a historical execution time to a first execution time, and by means of displaying a plurality of images within the time period, changes in state variables such as physical motions or lighting in the images are more detailed and smoother, such that the images are displayed more naturally. In summary, the solution can increase the frame rate of graphics frames and improve the smoothness of frame display; moreover, there is no need to increase the operation speed of logic frames, and the power consumption and heat generation can also be reduced.

Description

An image rendering method and related apparatus

[0001] This application claims priority to Chinese Patent Application No. 2024111871480, filed on August 27, 2024, entitled "An Image Rendering Method and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing, and in particular to an image rendering method and related apparatus. Background Technology

[0003] During image rendering, image rendering data of the image to be displayed can be calculated by executing a certain program. Each loop of the program can be regarded as a logical frame. The graphics frame is mainly used to render based on the image rendering data and send the rendered image to be displayed to the display device for display.

[0004] In the current rendering framework, after a logical frame is executed, image rendering data for a graphics frame can be generated. The graphics frame is then rendered based on this image rendering data to generate an image to be displayed.

[0005] However, this method results in slow image rendering data generation, low image rendering efficiency, low frame rate of graphics frames, and poor display effect. Summary of the Invention

[0006] To address the aforementioned technical problems, this application provides an image rendering method and related apparatus, which can improve the generation speed of image rendering data, improve image rendering efficiency, increase the frame rate of graphics frames, and thus improve the smoothness of screen display.

[0007] The embodiments of this application disclose the following technical solutions:

[0008] On one hand, this application provides an image rendering method, executed by a computer device, the method comprising:

[0009] When the first execution time of the first logical frame is reached, the historical state variable after the execution of the historical logical frame is obtained; the historical execution time of the historical logical frame is before the first execution time, and the historical state variable is used to identify the historical image rendering data of the historical graphic frame obtained after the execution of the historical logical frame.

[0010] During the execution of the first logical frame, based on the historical state variables, the first image rendering data of each of the N first graphics frames is determined by N threads; the N first graphics frames correspond one-to-one with the N first moments, and different threads are used to determine the first image rendering data of the first graphics frames corresponding to different first moments. The N first moments are later than the historical execution moment and earlier than or equal to the first execution moment, and N is an integer greater than 1.

[0011] Rendering is performed based on the first image rendering data of the N first graphic frames to obtain N images.

[0012] In another aspect, this application provides an image rendering apparatus, the apparatus comprising:

[0013] The acquisition unit is used to acquire historical state variables after the execution of the historical logical frame is completed when the first execution time of the first logical frame is reached; the historical execution time of the historical logical frame is before the first execution time, and the historical state variables are used to identify the historical image rendering data of the historical graphic frame obtained after the execution of the historical logical frame;

[0014] The determining unit is used to determine the first image rendering data of each of the N first graphics frames based on the historical state variables during the execution of the first logical frame through N threads; the N first graphics frames correspond one-to-one with N first moments, and different threads are used to determine the first image rendering data of the first graphics frames corresponding to different first moments, the N first moments are later than the historical execution moments and earlier than or equal to the first execution moments, and N is an integer greater than 1;

[0015] The rendering unit is used to render N images based on the first image rendering data of the N first graphics frames.

[0016] In another aspect, this application provides a computer device, which includes a processor and a memory:

[0017] The memory is used to store computer programs and to transfer the computer programs to the processor;

[0018] The processor is configured to execute the method according to the computer program.

[0019] In another aspect, this application provides a computer-readable storage medium for storing a computer program that, when executed by a computer device, implements the method described thereon.

[0020] In another aspect, this application provides a computer program product including a computer program that, when run on a computer device, causes the computer device to perform the method.

[0021] As can be seen from the above technical solution, the first logical frame can be executed according to a certain execution cycle. Upon reaching the first execution moment of the first logical frame, execution can begin to calculate the first image rendering data of the first graphics frame. To execute the first logical frame, it is necessary to obtain the historical state variables after the execution of previous logical frames. The previous logical frame is the last logical frame executed before the first logical frame; that is, the historical execution moment of the previous logical frame is before and adjacent to the first execution moment. After the previous logical frame is executed, the historical image rendering data of the previous graphics frame can be obtained. This allows the display of the historical image at the previous execution moment, i.e., the historical image rendering data can be used to identify the historical image corresponding to the previous execution moment, and the historical state variables can be used to identify this historical image rendering data. By obtaining these historical state variables, the changes in the image rendering data after the previous execution moment can be determined.

[0022] Furthermore, when a computer device has multiple threads, to improve image rendering efficiency, multiple threads can be invoked simultaneously during the execution of the first logical frame. That is, based on historical state variables, N threads can determine the first image rendering data for N first graphics frames. These N first graphics frames correspond to N first time points, with different threads determining the first image rendering data for different first time points. This parallel computation by N threads generates N first image rendering data. Rendering based on this data yields N images. Compared to related technologies where only one graphics frame's image rendering data is generated after one logical frame is executed, this solution can obtain image rendering data for multiple graphics frames at once. After rendering one graphics frame, the next frame can be rendered quickly without waiting for the next logical frame to complete, significantly improving image rendering efficiency and increasing the frame rate. Furthermore, since each of the N first image rendering data points corresponds to N first time points that are later than the historical execution time point but earlier than or equal to the first execution time point, multiple image rendering data points can represent the rendering data within the time period from the historical execution time point to the first execution time point. By displaying multiple images within this time period, the changes in physical actions or lighting and other state variables in the images will be more detailed and smoother, resulting in a more natural image display. In summary, this solution can improve the frame rate of graphics frames and enhance the smoothness of screen display. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 is a schematic diagram of the processing of logic frames and graphics frames in related technologies;

[0025] Figure 2 is a scene diagram of an image rendering method provided in an embodiment of this application;

[0026] Figure 3 is a flowchart illustrating an image rendering method provided in an embodiment of this application;

[0027] Figure 4 is a schematic diagram of an image rendering process provided in an embodiment of this application;

[0028] Figure 5 is a schematic diagram of the calculation process of image rendering data provided in an embodiment of this application;

[0029] Figure 6 is a schematic diagram of the execution process of interpolating to generate image rendering data according to an embodiment of this application;

[0030] Figure 7 is a schematic diagram of the execution process of another interpolation-generated image rendering data provided in an embodiment of this application;

[0031] Figure 8 is a schematic diagram of an image rendering apparatus provided in an embodiment of this application;

[0032] Figure 9 is a structural diagram of a terminal device provided in an embodiment of this application;

[0033] Figure 10 is a structural diagram of a server provided in an embodiment of this application. Detailed Implementation

[0034] The embodiments of this application will now be described with reference to the accompanying drawings.

[0035] As described in the background section, the image rendering data generation method in the related technology can only generate one graphic frame after the execution of a logical frame, that is, the image rendering method based on single data. The ratio of logical frames to graphic frames is 1:1, which makes the image rendering data generation speed slow, the image rendering efficiency low, the frame rate of graphic frames low, and the display effect poor.

[0036] Referring to Figure 1, which is a schematic diagram of the processing of logical frames and graphics frames in the related technology, logical frames can be executed in the central processing unit (CPU) thread 1, and the graphics frame rendering process can be performed in the graphics processing unit (GPU). After logical frame L1 is executed in CPU thread 1, image rendering data of graphics frame A1 can be generated and submitted to the GPU for rendering. Then, after logical frame L2 is executed, image rendering data of graphics frame A2 can be generated for rendering in the GPU. Then, logical frame L3 is executed to generate image rendering data of graphics frame A3 for rendering.

[0037] Furthermore, in related technologies, in order to increase the frame rate of graphics frames, the frequency of a single CPU core can only be increased to improve the processing speed of logical frames. Since the frequency of a CPU core is not linearly related to its power consumption and heat generation, when the processing speed of a single CPU core doubles, its power consumption and heat generation may increase several times over. Such a large amount of power consumption and heat generation poses a great challenge to high frame rate rendering.

[0038] This application provides an image rendering method and related apparatus. This application enables image rendering based on multiple data sets. Specifically, during the execution of any logical frame (e.g., the first logical frame), N threads (N being an integer greater than 1) are used to parallelly calculate and determine the first image rendering data for N graphics frames. This achieves the generation of image rendering data for multiple graphics frames based on one logical frame, with a logical frame to graphics frame ratio of 1:N. This allows for rapid rendering of the next graphics frame after rendering one, without waiting for the next logical frame to complete, significantly improving image rendering efficiency and increasing the frame rate. Furthermore, multiple image rendering data sets can represent rendering data from the historical execution time to the first execution time. By displaying multiple images within this time period, changes in physical actions or lighting state variables in the images become more detailed and smoother, resulting in a more natural image display. In summary, this solution can improve the frame rate of graphics frames, enhance the smoothness of screen display, and reduce power consumption and heat generation without increasing the computation speed of logical frames.

[0039] The image rendering method provided in this application can be implemented using a computer device, which can be a terminal device or a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Terminal devices include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, and extended reality (XR) devices. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this connection.

[0040] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0041] Referring to Figure 2, which is a scene diagram of an image rendering method provided in an embodiment of this application, the image rendering method can be applied to a server or terminal device. In Figure 2, a server is used as an example of the aforementioned computer device for illustration.

[0042] The first logical frame can be executed according to a certain execution cycle. Execution of the first logical frame can begin when the first execution time t is reached. 执行1 For example, at 100ms, the first image rendering data of the first graphics frame is calculated to display the corresponding image. Since the first graphics frame needs to be determined based on the image rendering data of its previous graphics frame, the historical state variables of the historical logical frames after execution need to be obtained when the first execution time is reached. The historical logical frame at historical execution time t... 历史 Execution, historical execution time t 历史 At the first execution time t 执行1 Previously, and with the first execution time t 执行1 Adjacent, meaning the historical logical frame is the logical frame preceding the first logical frame, with the historical execution time t. 历史 For example, at 50ms. After the historical logical frame is executed, image rendering data for at least one graphic frame can be obtained, and each graphic frame corresponds to a time point to display the image at that time. For example, some graphic frames correspond to 30ms, some to 40ms, and some to 50ms. The graphic frame corresponding to the historical execution time of 50ms can be recorded as a historical graphic frame. The historical state variable can be used to identify the historical image rendering data of the historical graphic frame obtained after the historical logical frame is executed. The historical image rendering data is used to identify the historical execution time t. 历史Corresponding historical images.

[0043] To improve image rendering efficiency when the server has multiple threads, multiple threads can be invoked simultaneously during the execution of the first logical frame. This means that based on historical state variables, N threads can determine the first image rendering data for N first graphics frames. These N first graphics frames correspond to N first time points, with different threads determining the first image rendering data for each frame at a different time point. For example, thread 1 can determine the first image rendering data for frame A1 at time point 60ms, thread 2 can determine the first image rendering data for frame A2 at time point 70ms, and thread N can determine the first image rendering data for frame An at time point 100ms. In this way, during the execution of the first logical frame, N threads can perform parallel calculations to generate the first image rendering data for N first graphics frames.

[0044] The server renders N images based on the first image rendering data of N first graphics frames. Compared to related technologies where only one graphics frame's image rendering data can be generated after one logical frame is executed, this solution can obtain image rendering data for multiple graphics frames at once through the execution of one logical frame. After rendering one graphics frame, the rendering of the next graphics frame can proceed quickly without waiting for the next logical frame to complete. This allows for the rendering of more graphics frames per unit time, greatly improving image rendering efficiency and increasing the frame rate of graphics frames.

[0045] Furthermore, since the N first image rendering data each correspond to N first time points that are later than the historical execution time point, and earlier than or equal to the first execution time point (for example, when the first time point is 60ms, 70ms, or 100ms), they are all at the historical execution time point t. 历 史 = 50ms later, and no later than the first execution time t 执行1 =100ms. This means that multiple image rendering data can represent the rendering data from the historical execution time to the first execution time. By displaying multiple images within this time period, such as adding two images corresponding to 60ms and 70ms, compared to directly switching from the image corresponding to 50ms to the image corresponding to 100ms, displaying multiple images during this period makes the changes in physical actions or lighting state variables in the images more detailed and smooth, resulting in a more natural image display. In summary, this solution can improve the frame rate of graphics frames, improve the smoothness of screen display, and does not require increasing the processing speed of logical frames.

[0046] Referring to Figure 3, which is a flowchart of an image rendering method provided in an embodiment of this application, the method can be executed by a computer device. In this embodiment, the computer device is a server as an example for explanation.

[0047] Step 301: When the first execution time of the first logical frame is reached, obtain the historical state variables after the historical logical frame has been executed.

[0048] A logic frame is a frame used for logical calculations, such as changes in object position, character movement, and lighting. Each execution of the program is recorded as a logic frame, and logic frames can be executed at a certain cycle, for example, every 50ms. After the logic frame is completed, image rendering data for a graphics frame is generated. A graphics frame is a frame that renders an image based on the image rendering data, transforming the image rendering data into an image that the user can see and display on the screen. Image rendering data consists of the data required for image rendering.

[0049] A logical frame can include historical logical frames and a first logical frame. The first logical frame can be any logical frame that needs to be executed, and the historical logical frame can be the previous logical frame of the first logical frame, meaning that the historical logical frame and the first logical frame are adjacent. Each logical frame has a corresponding execution time. When the execution time of a logical frame is reached at the current time, that logical frame begins execution. The execution time of a historical logical frame can be recorded as the historical execution time, and the execution time of the first logical frame can be recorded as the first execution time. The historical execution time can be before the first execution time and adjacent to the first execution time; that is, the historical execution time is the execution time preceding the first execution time. The historical logical frame is the logical frame that was executed before the first logical frame, meaning that the historical logical frame is the last logical frame executed before the first logical frame. For example, taking the execution of a logical frame every 50ms as an example, the first execution time t1 corresponding to the first logical frame is 100ms, and the historical execution time t of the historical logical frame is... 历史 It is the 50ms mark.

[0050] After the historical logical frame is executed, image rendering data for at least one graphics frame can be obtained. For example, image rendering data at the historical execution time can be obtained. Furthermore, image rendering data for graphics frames at times prior to the historical execution time can also be obtained. In short, after the historical logical frame is executed, image rendering data for at least one graphics frame within the time period between the historical logical frame and its previous logical frame can be generated. The number of graphics frames can be preset. Before the historical logical frame is executed, the number of graphics frames to be generated is pre-set so that the server can call the corresponding number of threads to obtain the corresponding number of graphics frames.

[0051] Furthermore, at least one graphic frame can be arbitrarily distributed within this time period; that is, the time corresponding to each graphic frame only needs to fall within this time period. The time intervals between the times of adjacent graphic frames can be equal or unequal. For example, within the time period between a historical logical frame and the logical frame preceding it, the times corresponding to multiple graphic frames can be evenly distributed or unevenly distributed within this time period. Of course, the times corresponding to the graphic frames also need to be preset, that is, the number of graphic frames and the time corresponding to each graphic frame need to be preset to generate multiple graphic frames that meet the quantity and time requirements.

[0052] For example, each logical frame generates 5 graphic frames, and the 5 graphic frames are evenly distributed. The historical execution time corresponding to the historical logical frame is 50ms, and the execution time of its previous logical frame is 0ms. After the historical logical frame is executed, the image rendering data of five graphic frames can be obtained. The five graphic frames correspond to 10ms, 20ms, 30ms, 40ms and the historical execution time of 50ms, respectively. After rendering, the images at these five times can be displayed. The time interval between adjacent images in these five images is equal and they are evenly distributed.

[0053] For the image rendering data of multiple graphics frames generated from historical logical frames, the graphics frame corresponding to the historical execution time can be recorded as the historical graphics frame, and the image rendering data of the graphics frame can be recorded as the historical image rendering data. The image obtained by rendering the historical image rendering data can be recorded as the historical image. That is, the historical image rendering data can be used to identify the historical image corresponding to the historical execution time. For example, when the historical execution time is 50ms, the graphics frame corresponding to 50ms is the historical graphics frame, and the image rendering data corresponding to the graphics frame of 50ms is the historical image rendering data. The historical image rendering data can identify the historical image corresponding to 50ms.

[0054] Historical state variables can be used to identify the historical image rendering data of historical graphics frames obtained after the execution of historical logical frames. Specifically, state variables can identify the image rendering data of graphics frames, which can be the state of the content contained in the image (such as objects or people), including physical location, lighting conditions, etc. The state of the same content may change in graphics frames at different times; that is, state variables are variable. Historical state variables can be understood as the state of content in historical graphics frames, corresponding to the historical execution time, and can identify the historical image rendering data of historical graphics frames.

[0055] Since the first logical frame is executed after the historical logical frames, and the first graphic frame is a graphic frame that follows the historical graphic frames, the calculation of the first image rendering data for the first graphic frame depends on the historical image rendering data of the historical graphic frames. That is, the historical state variables of the historical graphic frames are used as initial values ​​to calculate the first image rendering data of the first graphic frame after a certain period. Therefore, when the first execution moment of the first logical frame is reached, the server can obtain the historical state variables after the historical logical frames have been executed, so that when executing the first logical frame, new state variables can be generated based on these historical state variables to determine the new image for display.

[0056] Step 302: During the execution of the first logical frame, based on the historical state variables, the first image rendering data of each of the N first graphics frames is determined by N threads.

[0057] When the server has multiple threads, in order to improve image rendering efficiency, multiple threads can be called simultaneously for calculation during the execution of the first logical frame. That is, based on historical state variables, the first image rendering data of N first graphics frames can be determined by N threads, where N is an integer greater than 1. The N first graphics frames can correspond to N first moments, and different threads are used to determine the first image rendering data of the first graphics frames at different first moments.

[0058] In this context, the first graphics frame can be understood as a new graphics frame generated during the execution of the first logical frame, and the first image rendering data can be the image rendering data of the first graphics frame. Furthermore, each first graphics frame corresponds to a specific moment, which can be designated as the first moment. That is, there is a one-to-one correspondence between the first graphics frame and the first moment. At a given first moment, the corresponding image can be rendered and displayed based on the first image rendering data of the first graphics frame corresponding to that first moment. Thus, during the execution of the first logical frame, each thread can calculate the first image rendering data of one first graphics frame, resulting in a total of N first image rendering data sets. For example, if the number of graphics frames (e.g., the first graphics frame) generated in a logical frame is 3, then during the execution of the first logical frame, the calculation can be performed simultaneously in three threads, thereby generating the first image rendering data for three first graphics frames.

[0059] Furthermore, there are N first moments that are later than the historical execution moments, but earlier than or equal to the first execution moment. Specifically, since the first logical frame is executed after the historical logical frame, the image rendering data of the graphics frame after the historical execution moment can be determined. Therefore, the first moment corresponding to the generated first graphics frame is later than the historical execution moment. In addition, after the first logical frame is executed, the latest first moment corresponding to the generated first graphics frame is the first execution moment, that is, the first graphics frame corresponding to the first execution moment is generated. Therefore, the first moment is not later than the first execution moment.

[0060] For example, historical execution time t 历史 =50ms, the first execution time t1 = 100ms, then the first image rendering data of multiple first graphics frames within the time period (50ms, 100ms) can be generated, that is, the historical execution time t 历史 <First moment ≤ First execution moment t1, for example, the first graphic frame corresponding to 60ms, the first graphic frame corresponding to 70ms, and the first graphic frame corresponding to 100ms can be determined.

[0061] In practical applications, the first logical frame can be executed in the CPU thread of the server to determine the first image rendering data of multiple first graphics frames, and then submitted to the GPU of the server for rendering. When submitting, the rendering application programming interface (API) provided by the GPU can be used. The rendering API provides the function of submitting image rendering data in multiple threads. Make full use of this function to complete the submission of image rendering data.

[0062] Referring to Figure 4, which is a schematic diagram of an image rendering process provided in an embodiment of this application, the server's CPU has two threads, namely CPU thread 1 and CPU thread 2. When the first execution time of the first logical frame L1 is reached, the first logical frame L1 is executed. During the execution of the first logical frame L1, the first image rendering data corresponding to the first graphics frame is calculated in each of the two threads. For example, in CPU thread 1, the first image rendering data of the first graphics frame A1 can be calculated, and in CPU thread 2, the first image rendering data of the first graphics frame A2 can be calculated. When the first execution time of the first logical frame L2 is reached, the first logical frame L2 is executed. During its execution, in CPU thread 1, the first image rendering data of the first graphics frame A3 can be calculated, and in CPU thread 2, the first image rendering data of the first graphics frame A4 can be calculated.

[0063] When generating first image rendering data for N first graphics frames using N threads, the selection of N threads is not specifically limited. The number of threads, N, is the number of graphics frames (e.g., the first graphics frame) generated in a logical frame. This number can be preset; that is, before the execution of the first logical frame, the value of N can be pre-set so that the server can call N threads from multiple threads to obtain the first image rendering data for N first graphics frames. Furthermore, the selection of threads is also not specifically limited; that is, there is no fixed rule for selecting N threads from multiple threads to calculate the first image rendering data. For example, the N threads can be randomly selected or selected based on the thread load, thereby ensuring load balancing among the threads, avoiding excessive load on any single thread, and thus improving the computational efficiency of the first image rendering data.

[0064] Furthermore, multiple first graphics frames can be arbitrarily distributed within the time period between the historical logical frame and the first logical frame; that is, the first moment only needs to be within this time period. The time interval between adjacent first graphics frames can be equal or unequal. In other words, within this time period, the first moments corresponding to the generated first graphics frames can be evenly distributed. For example, one first graphics frame can be generated every 10ms, ensuring a uniform distribution of the first graphics frames and allowing the first image rendering data to change evenly, resulting in better display effects. Of course, the first moments can also be unevenly distributed; for example, the first moment can be closer to the first execution moment, allowing for smoother and more detailed display of the image near the first execution moment. Understandably, the first moments also need to be pre-set, i.e., the number of first graphics frames and the first moment corresponding to each first graphics frame are pre-defined to generate multiple first graphics frames that meet the quantity and timing requirements.

[0065] Furthermore, before calculating the first image rendering data for the first graphics frame via a thread, the common context of the first logical frame can be calculated. The common context can be understood as data unrelated to the graphics. Calculating the common context can include processing user input, processing network synchronization data, and updating user interface (UI) data. By calculating the common context, the impact on state variables can be determined so that the first image rendering data can be calculated. For example, calculating the common context could involve calculating the impact of a user's click action on the state variable, determining that the click action affects the state variable to "turn on the flashlight," so that the corresponding first image rendering data can be subsequently calculated to display the image of the flashlight being turned on.

[0066] Furthermore, the calculation of the common context can be performed in a few threads, for example, only in one thread, and the calculation result can be synchronized in N threads, allowing N threads to share the common context, thereby reducing the amount of computation per thread and avoiding redundant calculations. Referring to Figure 4, before calculating the first image rendering data of the first graphics frames A1 and A2, the common context corresponding to the first logical frame L1 can be calculated in CPU thread 1; before calculating the first image rendering data of the first graphics frames A3 and A4, the common context corresponding to the first logical frame L2 can be calculated in CPU thread 1.

[0067] Step 303: Render based on the first image rendering data of N first graphics frames to obtain N images.

[0068] After the server obtains the first image rendering data of N first graphics frames, the server can render based on these N first image rendering data to obtain N images. The N images can be understood as new images that can be displayed after the first logical frame is executed.

[0069] Referring to Figure 4, after the two threads obtain the first image rendering data of the first graphics frame A1 and the first image rendering data of the first graphics frame A2, respectively, they can submit them to the GPU. The GPU will then render the first graphics frame A1 and the first graphics frame A2, respectively, to obtain the corresponding two images. Similarly, after the two threads obtain the first image rendering data of the first graphics frames A3 and A4, they can also submit them to the GPU for rendering, resulting in the corresponding two images.

[0070] In summary, compared to related technologies where only one graphics frame's image rendering data can be generated after one logical frame is executed, this solution can obtain image rendering data for multiple graphics frames at once through the execution of one logical frame. After rendering one graphics frame, the next graphics frame can be rendered quickly without waiting for the next logical frame to complete. This allows for the rendering of more graphics frames per unit time, significantly improving image rendering efficiency and increasing the frame rate. This solution fully utilizes the capabilities of multiple CPU cores and effectively leverages the parallel capabilities of multi-threading on a multi-core CPU to achieve parallel computing.

[0071] Furthermore, in this application, a first logical frame can generate the first image rendering data required for multiple first graphics frames, thereby rendering multiple images. Each image corresponds to a first moment, which is located within the time interval between the execution times of two logical frames, thus enabling the generation of multiple images within this time interval. For example, by adding two images corresponding to 60ms and 70ms, compared to directly switching from the image corresponding to 50ms to the image corresponding to 100ms, the display of multiple images during this period allows for more detailed and smoother changes in physical actions or lighting state variables in the images. For instance, physical actions are more coherent, and lighting changes are smoother, resulting in a more natural image display. In summary, this solution can increase the frame rate of graphics frames, achieving a logical frame to graphics frame ratio of 1:N, thereby improving the smoothness of screen display. Moreover, while increasing the frame rate of rendering frames, it does not require increasing the computation speed of logical frames, thus reducing power consumption and heat generation.

[0072] During the execution of a logical frame, after the first logical frame is completed, the next logical frame, i.e., the second logical frame, can be executed. During the execution of the second logical frame, multiple threads can also be called to perform calculations. At this time, the threads required for the execution of the second logical frame can be arbitrarily selected.

[0073] In one possible implementation, when the second execution time of the second logical frame is reached, the first state variable after the execution of the first logical frame is obtained. During the execution of the second logical frame, based on the first state variable, the second image rendering data of each of the M second graphics frames is determined by M threads. The M second graphics frames correspond one-to-one with the M second times. Different threads are used to determine the second image rendering data of the second graphics frames corresponding to different second times. The M second times are later than the first execution time and earlier than or equal to the second execution time. The M threads and N threads are at least partially the same or completely different.

[0074] The execution time of the second logical frame can be recorded as the second execution time, for example, the second execution time can be the 150ms. When the current time is the second execution time, the server can obtain the first state variable obtained after the first logical frame is completed. The first state variable can identify the first image rendering data of the first graphics frame. The first state variable can reflect the state of the content in the image corresponding to the first image rendering data. For example, the target state variable can be the current physical position of the object in the image, the current lighting conditions, etc.

[0075] During the execution of the second logical frame, M threads can be called to calculate simultaneously, and the second image rendering data of the M second graphics frames can be determined based on the first state variable. Here, M is an integer greater than 1, that is, by utilizing the ability of multi-threaded parallel computing, the second image rendering data of multiple second graphics frames can be obtained at the same time.

[0076] The second graphics frame can be understood as a new graphics frame generated after the execution of the second logical frame. The second image rendering data is the image rendering data required by the second graphics frame during rendering. Since one second logical frame can generate the image rendering data required by multiple second graphics frames, and each second graphics frame corresponds to a time point, this time point can be recorded as the second time point. That is, there is a one-to-one correspondence between the second graphics frame and the second time point. One thread can determine the second graphics frame corresponding to a second time point. In this way, M threads can determine M second graphics frames, each corresponding to one of the M second time points.

[0077] In addition, the first execution time < the second execution time ≤ the second execution time, that is, within the time period between the first execution time and the second execution time, M threads can determine the M second graphics frames within that time period, thereby displaying multiple images within that time period and improving the smoothness of the screen display.

[0078] More importantly, the M threads used in the execution of the second logical frame do not correspond to the N threads used in the execution of the first logical frame. The values ​​of M and N can be the same or different; for example, N can be 3 and M can be 4. However, for ease of calculation, M and N are generally set to the same value. Furthermore, the M and N threads can have the same number of threads, such as both including thread 1, or they can have completely different threads. That is, after N threads are selected from multiple threads to complete the execution of the first logical frame, M threads are then selected from the remaining threads to complete the execution of the second logical frame. For example, when N is 3, the N threads can include thread 1, thread 2, and thread 3; when M is 4, the M threads can include thread 2, thread 4, thread 6, and thread 7.

[0079] In other words, by setting M threads and N threads to be at least partially the same or completely different, the selection of multiple threads required when executing different logic frames can be arbitrary, without having to keep M threads and N threads completely identical. This increases the freedom of thread selection, provides more ways to select multiple threads, and makes thread selection more flexible and reasonable, thus achieving load balancing of multiple threads.

[0080] During the execution of the first logical frame, when the first image rendering data of the N first graphics frames is determined by N threads, in order to ensure the accuracy of each first image rendering data, the first image rendering data can be calculated independently by each thread, and the N threads will not affect each other.

[0081] In one possible implementation, step 302, during the execution of the first logical frame, determines the first image rendering data of the N first graphics frames based on historical state variables through N threads. Specifically, it determines the execution logic of the first logical frame. During the execution of the first logical frame, for each of the N threads, based on historical state variables and the first moment corresponding to the thread, the first image rendering data of the corresponding first graphics frame at the first moment is determined according to the execution logic.

[0082] During the execution of a logic frame, in order to determine the image rendering data of a graphics frame, the logic frame can be executed according to certain logic, which can be referred to as execution logic. That is, the execution logic can represent how the logic frame calculates the image rendering data of the graphics frame, which is the calculation process of the program.

[0083] The server can determine the execution logic of the first logical frame. This execution logic identifies the logic used to calculate the first image rendering data during the execution of the first logical frame. In other words, this execution logic enables the server to know how to calculate the first image rendering data; that is, the execution logic of the first logical frame can represent the change process of state variables within the time period from the historical execution time to the first execution time.

[0084] For example, if the change in the state variable is a change in the physical position of an object triggered by the user, the execution logic of the first logic frame can be the object position calculation logic, which can calculate the physical position of the object after it moves, as well as the movement path of the object. When the change in the state variable is a change in lighting, the execution logic of the first logic frame can be the lighting calculation logic, which can calculate the lighting change process and the lighting conditions after the change.

[0085] In practical applications, the execution logic of the first logical frame can be selected from a large number of pre-set calculation logics, such as lighting calculation logic and physical position calculation logic. Thus, when the first logical frame is executed, a suitable calculation logic is selected as the execution logic of the first logical frame.

[0086] After the server determines the execution logic of the first logical frame, during the execution of the first logical frame, each of the N threads can determine the first image rendering data required for the first graphics frame at that first moment based on the historical state variables and the first moment corresponding to that thread, according to the calculation logic of the execution logic. Here, the first moment is a pre-set moment so that the server knows which moment's image rendering data to calculate during the calculation process.

[0087] In practical applications, execution logic can be represented by functions. For example, the Tick() function can be used to represent the execution logic for calculating the first image rendering data. All N threads use this function for calculation, meaning that the execution process of these threads is the same and they all need to execute the first logical frame according to this execution logic. However, the first moment that these threads use for calculation is different. Each thread determines the first image rendering data at that first moment through the execution logic based on its corresponding first moment.

[0088] Since each thread independently calculates the first image rendering data based on historical state variables and according to the execution logic of the first logical frame, in order to ensure the accuracy of the calculation of each thread, after completing the calculation of the common context, the impact on the state variables can be calculated and synchronized in N threads. Furthermore, the historical state variables at the historical execution time can be synchronized in N threads to ensure that the initial values ​​of each thread are the same during calculation.

[0089] For example, suppose the first image rendering data is the position of an object after 1 second, 2 seconds, and 3 seconds. The execution logic of the first logical frame is "result position = initial position + time × speed". Then the initial position in the three threads needs to be the same, that is, the same initial value, to ensure the consistency of the calculation results.

[0090] Referring to Figure 5, which is a schematic diagram of the calculation process of image rendering data provided in an embodiment of this application, when there are three CPU threads, data synchronization can be performed in CPU thread 1, that is, synchronizing historical state variables and their impact. Then, multi-threaded tasks are dispatched, sending the execution logic of the first logical frame, historical state variables, and the first time point corresponding to each thread to each thread. The time interval between the historical execution time and the first execution time can be denoted as Delta Time. Then, CPU threads 1, 2, and 3 can calculate the first image rendering data of the first graphics frame at 0.333*Delta Time, 0.667*Delta Time, and Delta Time, respectively. The three threads need to calculate Tick(0.333*Delta Time), Tick(0.667*Delta Time), and Tick(Delta Time), respectively, to obtain the first image rendering data at the corresponding time point, that is, to obtain the first image rendering data of the first graphics frames A1, A2, and A3. The times corresponding to these three first image rendering data are evenly distributed, and the time difference between adjacent first graphics frames is equal, all being 1 / 3 Delta. Time makes the changes in the image more even and consistent.

[0091] In summary, for N threads, each thread independently calculates the first image rendering data of the corresponding first graphics frame based on the historical state variables and the execution logic of the first logical frame. The N threads do not affect each other. Each first image rendering data is calculated based on the historical state variables through the correct physical position calculation logic or lighting calculation logic, etc. The accuracy of each first image rendering data is very high. It can ensure that the rendering data of each graphics frame is accurate enough when using threads to calculate the first image rendering data of the first graphics frame in parallel.

[0092] The following section will explain in detail the specific calculation process of the first image rendering data of the first graphics frame for each thread.

[0093] In one possible implementation, based on historical state variables and the first moment corresponding to the thread, the first image rendering data of the first graphics frame corresponding to the first moment is determined according to the execution logic. Specifically, this can be done by calculating the time difference between the first moment corresponding to the thread and the historical execution moment; determining the target state variable after the time difference according to the execution logic; and performing data transformation on the target state variable to obtain the first image rendering data of the first graphics frame corresponding to the first moment.

[0094] For each of the N threads, the historical execution time corresponding to the historical logical frame and the first time corresponding to the first graphic frame to be generated can be obtained, and the time difference between these two times can be calculated. Since the state variables change over time, the server can calculate the target state variable after the time difference, based on the historical state variables corresponding to the historical execution time and the execution logic of the first logical frame. This target state variable is the state variable at the first time, representing the updated value of the state variable. For example, when using the Tick() function to represent the execution logic of the first logical frame, the target state variable can be calculated using Tick(time difference, historical state variable). The server can perform data transformation on the target state variable to obtain the first image rendering data for the first graphic frame. Data transformation can be understood as performing data format conversion, data merging, data normalization, etc., and can be performed according to the actual situation.

[0095] In summary, by calculating the time difference between the first moment and the historical execution moment, the updated target state variable is determined based on this time difference, and data transformation is performed. This allows for the calculation of more accurate first image rendering data, which more accurately corresponds to the first moment. The calculation process is more concise and precise, thus improving the accuracy of the image rendering data.

[0096] In the process of generating image rendering data for multiple graphics frames from a single logical frame, in order to further reduce the amount of computation, only some threads can calculate the corresponding image rendering data, while other threads perform certain processing on the image rendering data to obtain their own corresponding image rendering data.

[0097] In one possible implementation, during the execution of the first logical frame, step 302 determines the first image rendering data of each of the N first graphics frames based on historical state variables through N threads, which may specifically include steps 3021-3023 (not shown in the figure).

[0098] Step 3021: Obtain the first sub-image rendering data at the corresponding first sub-time point through N1 first threads.

[0099] The N threads can include N1 first threads and N2 second threads, where N1 + N2 = N, and N1 is a positive integer greater than 0 and less than N. That is, a portion of the N threads can be designated as first threads, and the remaining portion as second threads. The number of first threads is denoted as N1, and there can be at least one first thread. The number of second threads is denoted as N2, and there can also be at least one second thread.

[0100] The first thread can be used to determine the first sub-image rendering data corresponding to the first sub-time point. The first sub-time point can be any one of N first times. The first image rendering data corresponding to the thread that is the first thread is recorded as the first sub-image rendering data. Therefore, the first sub-image rendering data determined by the first thread corresponds to the first sub-time point.

[0101] Furthermore, the first sub-image rendering data is calculated based on historical state variables. This rendering data can be obtained by executing the first logical frame in a certain thread, so that the first thread can obtain the fully calculated first sub-image rendering data. That is, the first sub-image rendering data at the first sub-time point corresponding to each first thread is obtained based on historical state variables.

[0102] Therefore, for each of the N1 first threads, the first sub-image rendering data obtained not only corresponds to the first sub-time point, but is also calculated according to the execution logic of the first logical frame. In other words, there are N1 first threads among the N threads, and the first sub-image rendering data they obtain is actually calculated and has high accuracy.

[0103] It is understandable that when the number of threads N1 is 1, that is, only in one thread, the first image rendering data is calculated based on the historical state variables and according to the execution logic of the first logical frame. In this case, the calculation of the state variables is only performed in one thread, so there is no need to synchronize the historical state variables and other data between multiple threads. Other threads can obtain the corresponding first image rendering data based on interpolation.

[0104] Step 3022: For each of the N2 second threads, based on the first sub-image rendering data, perform interpolation processing through the second thread to determine the corresponding second sub-time point of the second sub-image rendering data.

[0105] If each of the N threads independently calculates the rendering data according to the execution logic of the first logical frame, the amount of computation will increase significantly. In order to reduce the amount of computation, the image rendering data corresponding to a portion of the threads can be obtained by interpolation from the already calculated image rendering data.

[0106] Specifically, there are N² second threads. For each second thread, interpolation can be performed based on the previously calculated first sub-image rendering data to obtain the corresponding image rendering data for the second thread, which is then recorded as the second sub-image rendering data. The time corresponding to the second sub-image rendering data is recorded as the second sub-time. During interpolation, the unknown second sub-image rendering data can be estimated based on the known first sub-image rendering data. For example, a correlation function can be constructed based on the first sub-image rendering data, and then the second sub-image rendering data at the second sub-time can be determined based on this function.

[0107] For the first sub-time and the second sub-time, each first sub-time and each second sub-time are different first times among N first times. For example, any first time can be taken as the first sub-time, and another first time can be selected as the second sub-time.

[0108] Furthermore, during interpolation, interpolation can be performed between adjacent first sub-image rendering data to obtain second sub-image rendering data; that is, the second sub-time point lies between adjacent first sub-time points. For example, if adjacent first sub-time points are 60ms and 100ms, then the second sub-time point can be 70ms. Of course, interpolation can also be performed outside of the first sub-image rendering data to obtain the second sub-image rendering data; for example, if two first sub-time points are 60ms and 70ms, then the second sub-time point can be 80ms.

[0109] To ensure the accuracy of the interpolated second sub-image rendering data, the first execution time can be used as one of the first sub-times. This allows for intermediate interpolation based on the first sub-image rendering data corresponding to the first execution time and the first sub-image rendering data corresponding to historical execution times. This improves the accuracy of the second sub-image rendering data and ensures that the image corresponding to the first execution time is more accurate, thereby enhancing the accuracy of the screen display.

[0110] Step 3023: Take N1 first sub-image rendering data and N2 second sub-image rendering data as N first image rendering data.

[0111] Each first thread can correspond to one first sub-image rendering data, and each second thread can correspond to one second sub-image rendering data. The N1 first sub-image rendering data and the N2 second sub-image rendering data can be used as N first image rendering data to complete the process of generating N first image rendering data.

[0112] In summary, during the generation of N first image rendering data, a portion of the first image rendering data can be calculated first, and the remaining number of first image rendering data can be obtained by interpolation based on the calculated first image rendering data. Therefore, it is not necessary for each thread to calculate it once. Some threads can obtain the first image rendering data through interpolation. The computational amount of interpolation is relatively small. Under the premise of ensuring the accuracy of the first image rendering data, the computational amount can be reduced, thereby speeding up the generation of N first image rendering data and further improving rendering efficiency.

[0113] The first sub-image rendering data obtained by the first thread can be calculated by the first thread itself, or it can be calculated by other threads and then sent to the first thread, thereby increasing the flexibility of processing between threads.

[0114] In one possible implementation, the N threads include a third thread, which is either a first thread or a second thread. Step 3021 involves obtaining the first sub-image rendering data at the corresponding first sub-time point through the N1 first threads. Specifically, based on historical state variables and the N1 first sub-time points, the third thread determines the first sub-image rendering data corresponding to each of the N1 first sub-time points; and the N1 first threads obtain the first sub-image rendering data at the corresponding first sub-time point from the third thread.

[0115] Specifically, any one of the N threads can be designated as the third thread. The third thread can be the same as the first thread or the second thread. The third thread can be used to calculate the rendering data of the first sub-image. The number of third threads is not limited here; there can be one or more.

[0116] For each third thread, calculations can be performed based on historical state variables and the first sub-time point corresponding to that thread, according to the execution logic of the first logical frame, to determine the first sub-image rendering data corresponding to that first sub-time point. In this way, N1 third threads can calculate N1 sets of first sub-image rendering data, and these rendering data correspond one-to-one with the N1 first sub-time points. Furthermore, N1 first threads can each obtain N1 sets of first sub-image rendering data from the N1 third threads.

[0117] As an example, the third thread can be the same as the first thread, that is, it can perform calculations based on historical state variables and directly use the calculated first image rendering data in the same thread, that is, obtain the data. Referring to Figure 6, which is a schematic diagram of the execution process of interpolation to generate image rendering data according to an embodiment of this application, CPU thread 1 is both the first thread and the third thread, and the two second threads are CPU thread 2 and CPU thread 3. In CPU thread 1, based on the historical state variables, it can calculate according to the execution logic of the first logical frame, that is, calculate Tick(DeltaTime), where DeltaTime is the time interval between the historical execution time and the first execution time. The calculation result represents the first image rendering data of the first graphics frame A3. Then, CPU thread 1 dispatches multi-threaded tasks, so CPU thread 3 performs interpolation calculation based on the calculation result to obtain the first image rendering data of the first graphics frame A1 and submit it. That is, the first graphics frame A1 at time 0.333*DeltaTime is obtained through interpolation. CPU thread 2 performs interpolation calculation based on the calculation result to obtain the first image rendering data of the first graphics frame A2 at time 0.667*DeltaTime and submits it. CPU thread 1 directly uses the calculation result and directly submits the first image rendering data of the first graphics frame A3.

[0118] As another example, the third thread can be the same as the second thread. That is, after calculating the first image rendering data based on historical state variables in one thread, it is sent to another thread for use so that the thread can obtain the first image rendering data. Referring to Figure 7, which is a schematic diagram of the execution process of interpolation to generate image rendering data according to another embodiment of this application, one first thread is CPU thread 3, two second threads are CPU thread 2 and CPU thread 1, and the third thread is CPU thread 1. That is, CPU thread 1 calculates Tick (DeltaTime) according to the execution logic of the first logical frame. The calculation result represents the first image rendering data of the first graphics frame A3. Then, it dispatches a multi-threaded task so that CPU thread 3 can directly use the calculation result and submit it. CPU thread 1 and CPU thread 2 need to perform interpolation calculations to obtain the first image rendering data of the first graphics frames A1 and A2 respectively and submit them. In short, the calculation result calculated by CPU thread 1 is sent to CPU thread 3 for use.

[0119] In summary, the third thread can be used to calculate image rendering data according to the execution logic of the first logical frame. The first thread is the thread that directly uses the image rendering data. That is, thread 1 can calculate the image rendering data and send it to another thread 2 for use. The first image rendering data corresponding to thread 1 can be obtained by interpolation calculation. That is, a thread can complete both the calculation process and the interpolation process, thereby making full use of the thread's powerful computing power. In short, in the process of determining N first image rendering data, the processing of each thread can be more flexible and free.

[0120] During image changes, there may be changes in static variables such as illumination, as well as changes in dynamic variables such as physical motion. Since static variables are mostly changes in parameter values ​​and are relatively simple, while dynamic variables involve coordinate transformations of physical positions and are more complex to process, static and dynamic variables can be processed in different ways.

[0121] The first image rendering data is rendering data used to reflect target state variables, which include static variables used to identify changes in attribute states. Specifically, state variables can identify the image rendering data of a graphics frame, and can be the state of content contained in the image (such as objects or people), including physical position, lighting conditions, etc. Target state variables can be state variables in the first image rendering data, reflecting the state of content in the image corresponding to the first image rendering data. For example, target state variables could be the current physical position of an object in the target image, the current lighting conditions, etc.

[0122] When the attribute state of a target state variable changes, this type of target state variable can be denoted as a static variable. The attribute state of a target state variable refers to a relatively static attribute, such as color or intensity. Furthermore, a static variable is a target state variable whose physical location remains unchanged. Static variables are mainly used to represent changes in the value of certain content in an image, while the physical location of that content does not change. In other words, a static variable is a variable whose physical location remains unchanged; for example, static variables can be light intensity or light direction.

[0123] Based on the first sub-image rendering data, interpolation processing is performed by the second thread to determine the corresponding second sub-time point rendering data. Specifically, based on the first sub-image rendering data and historical image rendering data, linear interpolation processing is performed by the second thread to determine the corresponding second sub-time point rendering data.

[0124] In other words, for this type of static attribute, since it does not involve changes in physical location, the interpolation is less difficult. Therefore, linear interpolation can be performed in the second thread based on the first sub-image rendering data at the first sub-time point and the historical image rendering data at the historical execution time points. Linear interpolation refers to approximating the original function with a straight line determined based on these two types of data, and then determining the image rendering data at other times from this straight line.

[0125] In summary, using linear interpolation to interpolate static variables is a relatively simple method that better reflects the continuous and uniform changes of static variables. The interpolation results are more accurate, and the interpolation process is less difficult and computationally convenient.

[0126] When image changes involve dynamic variables such as physical motion, the description of physical motion differs across reference frames, and physical motion can be non-linear. Directly interpolating the positions of objects in the scene can lead to errors in the player's camera coordinate system (view space), resulting in rendering errors such as jitter and teleportation that severely impact visual performance. Screen space, on the other hand, refers to the two-dimensional space on the display screen, the area where the image is displayed. Players or users can directly view the content displayed in screen space. Therefore, to make the display of physical motion smoother and more accurate, the interpolation of dynamic variables can be performed in screen space, i.e., using screen space interpolation.

[0127] In one possible implementation, the first image rendering data is rendering data used to reflect target state variables, which include dynamic variables used to identify physical motion. That is, if the target state variables involve physical motion, such target state variables can be denoted as dynamic variables, which can be used to represent variables whose physical position changes. For example, dynamic variables can include the physical position of an object.

[0128] Based on the first sub-image rendering data, the second thread performs interpolation processing to determine the corresponding second sub-time point's second sub-image rendering data. Specifically, based on the first sub-image rendering data and historical image rendering data, the second thread performs screen space interpolation processing to obtain the second sub-image rendering data corresponding to the second sub-time point.

[0129] In this system, the first sub-image rendering data consists of the screen space coordinates of the dynamic variables at the first sub-time point, while the historical image rendering data consists of the screen space coordinates of the dynamic variables at the historical execution time points. In other words, both the first and historical image rendering data are represented by the screen space coordinates of the dynamic variables. Therefore, when interpolating between these two data points, which is essentially interpolation within screen space, the resulting second sub-image rendering data is also the screen space coordinates of the dynamic variables. In short, interpolation can be performed based on the screen space coordinates of the dynamic variables at different times (i.e., the first sub-time point and the historical execution time points) to obtain the coordinates at another time point (i.e., the second sub-time point), which can then be used as the second sub-image rendering data.

[0130] In summary, since the image viewed by the user is located within screen space, which is the closest space to the user compared to other spaces such as view space, interpolating the coordinates within screen space yields coordinates that are also within screen space. This allows the interpolation to obtain new physical positions of dynamic variables, making the changes in physical positions more consistent with known changes at other times (i.e., the first sub-time and historical execution times). Objects will not exhibit jitter or teleportation, and physical motion will be more natural and smooth, thereby improving the accuracy of the physical positions obtained through interpolation.

[0131] When interpolating dynamic variables in screen space, if there are too many dynamic variables, such as a large number of objects whose physical positions change, interpolating for each coordinate would significantly increase the computational cost. For example, a mesh used for rendering might have tens of thousands of vertices, and interpolating for each vertex would be prohibitively expensive. To reduce computational cost, interpolation can be performed on the transformation matrix between different spaces to obtain the second sub-image rendering data corresponding to the second sub-time point.

[0132] In one possible implementation, the first image rendering data is rendering data used to reflect target state variables, which include dynamic variables used to identify physical motion. Based on the first sub-image rendering data, interpolation processing is performed by a second thread to determine the second sub-image rendering data at the corresponding second sub-time point. This may specifically include steps 401-403 (not shown in the figure).

[0133] Step 401: Based on the first model matrix M1 and the historical model matrix M 历史 The second thread performs screen space interpolation to determine the second model matrix M2 corresponding to the second sub-time point.

[0134] In image rendering, the spatial transformation of an object's 3D model often involves the transformation between local space, world space, view space (also known as camera space), and screen space. During rendering, the object is first defined in local space, then transformed to world space, then to view space, and finally to screen space for display on the screen.

[0135] Local space is the coordinate space in which an object resides, that is, the initial location of the object; it is the object's own coordinate system. World space is the common coordinate system of all objects in the scene. An object's transformation from local space to world space is typically achieved by applying translation, rotation, and scaling transformations. In world space, an object's position and orientation are defined relative to other objects and the environment. View space is the coordinate system relative to the observer. When an object transforms from world space to view space, a viewpoint transformation occurs, including the camera's position and orientation. Screen space is the coordinate system ultimately rendered to the display or screen; coordinates in screen space are two-dimensional.

[0136] In spatial transformation, model matrix (M matrix), view matrix (V matrix), and projection matrix (P matrix) are often used to achieve coordinate transformation. The model matrix M is used to transform the coordinates of an object from local space to world space. In other words, a coordinate value in local space multiplied by the M matrix yields the corresponding coordinates in world space.

[0137] The view matrix V is used to transform the coordinates of an object from world space to view space. Multiplying a world space coordinate value by the V matrix yields its corresponding coordinates in view space. The projection matrix P is responsible for mapping the 3D world onto the 2D screen, transforming coordinates from view space to screen space. Multiplying a view space coordinate value by the P matrix yields its corresponding coordinates in screen space.

[0138] In computer graphics, three transformation matrices are used to transform the coordinates of an object in local space to its coordinates in screen space. This process involves multiplying the homogeneous coordinates of the local space position sequentially by the model matrix M, the view matrix V, and the projection matrix P, as shown in the formula below. clip =PVMs local

[0139] Among them, s clip This represents the coordinates of point s in screen space. local This represents the coordinates of point s in local space.

[0140] When the server determines the second sub-image rendering data at the second sub-time point, assuming the object itself is not deformed, the local spatial coordinates s of the vertices of the mesh volume are... local It will not change and can be considered to remain constant. If the P matrix, V matrix, and M matrix are known at the second sub-time, it can be seen from the above formula that by combining these three matrices with the known coordinates s in the local space... local By multiplying, we can obtain the coordinates in screen space, which is the second sub-image rendering data. Therefore, the key to interpolation is to determine the corresponding P matrix, V matrix, and M matrix at the second sub-time point. So, the following will introduce how to determine the P matrix, V matrix, and M matrix at the second sub-time point.

[0141] Specifically, the first sub-image rendering data includes the first model matrix M1 at the first sub-time. The first model matrix M1 is used to identify the transformation matrix of the dynamic variables from local space to world space at the first sub-time. That is, the first model matrix M1 can represent how the dynamic variables at the first sub-time are transformed from local space to world space. In other words, by multiplying the coordinates of a point in local space by M1, the coordinates of that point in world space at the first sub-time can be obtained.

[0142] Historical image rendering data includes the historical model matrix M at historical execution times. 历史 Historical model matrix M 历史 The history model matrix M is used to identify the transformation matrix of dynamic variables from local space to world space at historical execution moments. 历史 It can represent how dynamic variables at historical execution moments are transformed from local space to world space.

[0143] The model matrix corresponding to the second sub-time point can be denoted as the second model matrix M2. That is, the second model matrix M2 is used to identify the transformation matrix of the dynamic variables from local space to world space at the second sub-time point. In other words, by multiplying the coordinates of a point in local space with the second model matrix M2, we can obtain the coordinates of that point in world space at the second sub-time point.

[0144] The second thread can be based on the first model matrix M1 and the historical model matrix M. 历史 Interpolation is performed to obtain the second model matrix M2. Specifically, this is achieved by interpolating the first model matrix M1 and the historical model matrix M... 历史 The basic trend of change of the model matrix M can be determined, and the second model matrix M2 at the second sub-time can be obtained by interpolation.

[0145] For example, the first model matrix M1 is the model matrix at the 100ms level of the first sub-time point, and the historical model matrix M... 历史 Given the model matrix at the 50ms historical execution time, the second model matrix M2 at the 75ms second sub-time can be expressed as M2 = (M1 + M... 历史 ) / 2.

[0146] Step 402: Obtain the second view matrix V2, which is the dynamic variable transformed from world space to view space at the second sub-time, and obtain the second projection matrix P2, which is the dynamic variable transformed from view space to screen space at the second sub-time.

[0147] The view matrix corresponding to the second sub-time point can be denoted as the second view matrix V2, and the projection matrix corresponding to that time point can be denoted as the second projection matrix P2. That is, multiplying the coordinates of a point in world space by the second view matrix V2 yields the coordinates of that point in view space at the second sub-time point. Multiplying the coordinates of a point in view space by the second projection matrix P2 yields the coordinates of that point in screen space at the second sub-time point. In practical applications, the second view matrix V2 and the second projection matrix P2 can be generated from interpolated camera parameters, which will not be elaborated further. The screen space interpolation method mainly involves solving for the second model matrix M2.

[0148] Step 403: Use the second view matrix V2, the second projection matrix P2, and the second model matrix M2 as the second sub-image rendering data corresponding to the second sub-time point.

[0149] The second view matrix V2, the second projection matrix P2, and the second model matrix M2 can form the spatial transformation matrix corresponding to the second sub-time point, which can be denoted as the second spatial transformation matrix. The server can use the second spatial transformation matrix as the second sub-image rendering data corresponding to the second sub-time point so as to send it to the GPU for rendering processing.

[0150] Therefore, when performing screen space interpolation, the model matrix M can be interpolated to obtain the second model matrix M2 at the second sub-time point. This allows for spatial transformation calculations based on the second view matrix V2, the second projection matrix P2, the second model matrix M2, and the coordinates of each vertex of the object in the local space, thereby obtaining the coordinates of each point in the screen space, which is the second sub-image rendering data. In this way, there is no need to perform interpolation calculations for each point, which can greatly reduce the amount of computation, improve the generation speed of rendering data, and increase the frame rate of graphics.

[0151] When interpolating to obtain the second model matrix M2 at the second sub-time, in order to further reduce the amount of computation, the coordinates of only a few points can be calculated during the calculation process to determine the second model matrix M2. The specific calculation process is as follows.

[0152] In one possible implementation, based on the first model matrix and the historical model matrix, screen space interpolation is performed by a second thread to determine the second model matrix corresponding to the second sub-time point. This may specifically include steps 501-504 (not shown in the figure).

[0153] Step 501: Using the second thread, obtain the first spatial transformation matrix corresponding to the first sub-time point and the historical spatial transformation matrix corresponding to the historical execution time point.

[0154] The spatial transformation matrix for transforming a point on an object from local space to screen space at the first sub-time point is denoted as the first spatial transformation matrix. That is, multiplying the coordinates of a point in local space by the first spatial transformation matrix yields the coordinates of that point in screen space at the first sub-time point. The first spatial transformation matrix includes a first view matrix V1, a first projection matrix P1, and a first model matrix M1, used to identify the transformation of dynamic variables from world space to view space at the first sub-time point. In short, the first view matrix V1 represents how coordinates in world space are transformed to coordinates in view space at the first sub-time point, and the first projection matrix P1 represents how coordinates in view space are transformed to coordinates in screen space at the first sub-time point.

[0155] The spatial transformation relationship between points on an object and their local space at a given execution time is denoted as the history space transformation matrix. That is, multiplying the coordinates of a point in local space by the history space transformation matrix yields the coordinates of that point in screen space at the given execution time. The history space transformation matrix includes the history view matrix V, which identifies the transformation of dynamic variables from world space to view space at the given execution time. 历史 The historical projection matrix P from view space to screen space 历史 and the historical model matrix M 历史In short, the historical view matrix V 历史 The history projection matrix P represents how coordinates in world space are transformed into coordinates in view space at a given historical execution time. 历史 It can represent how to transform the coordinates in view space to screen space at a historical execution time.

[0156] In other words, the second thread can obtain the first view matrix V1, the first projection matrix P1, and the first model matrix M1 at the first sub-time point, as well as the historical view matrix V at the historical execution time point. 历史 Historical projection matrix P 历史 and historical model matrix M 历史 .

[0157] Step 502: Transform the sample local space matrix S using the first spatial transformation matrix. local Perform spatial transformation processing to determine the first sample screen space matrix S of the sample points in the screen space at the first sub-time point. clip-1 And through the historical space transformation matrix, the sample local space matrix S local Perform spatial transformation processing to determine the historical sample screen space matrix S of the sample points in the screen space at the historical execution time. clip-历史 .

[0158] Sample local space matrix S local This includes the coordinates of multiple linearly independent sample points in the local space; that is, the coordinates of several linearly independent local space points can be prepared in advance to form S. local The number of sample points doesn't need to be too large; for example, it can be 4, 5, or 8. The local spatial matrix S of the samples can then be used. local Calculations involving spatial transformation.

[0159] The first sample screen space matrix S clip-1 The coordinates of multiple linearly independent sample points in screen space at the first sub-time point can be represented by the following formula: S clip-1 =P1 V1 M1S local

[0160] Historical sample screen space matrix S clip-历史 The coordinates of multiple linearly independent sample points in screen space at historical execution times can be represented by the following formula: S clip-历史 =P 历史 V 历史 M 历史 S local

[0161] In short, for the first sub-time point and the historical execution time points, the sample local space matrix S can be used.local Multiply by the PVM matrix corresponding to that time step to obtain the screen space coordinates of the sample point at that time step.

[0162] Step 503: Based on the first sample screen space matrix S clip-1 , and historical sample screen space matrix S clip-历史 The second sample screen space matrix S corresponding to the second sub-time point is determined by interpolation using the target weights corresponding to the second sub-time point. clip-2 .

[0163] Among them, the historical execution time t 历史 The second sub-time point is less than or equal to the first execution time point t1. Different second sub-time points correspond to different target weights Q. The target weight Q is related to the second sub-time point to the historical execution time t1. 历史 The time intervals between them are positively correlated, that is, the second sub-time point and the historical execution time t are positively correlated. 历史 The longer the time interval, the greater the target weight.

[0164] When performing intermediate interpolation, the target weight can take values ​​within the range (0,1). Taking the first sub-time as the first execution time t1 as an example, when the second sub-time is also the first execution time t1, the corresponding target weight Q is 1. If the second sub-time is located at a historical execution time t... 历史 If the time interval between the first execution time t1 and the first execution time is exactly midway, then the target weight Q is 0.5. For example, if the time interval between the historical execution time and the first execution time is denoted as Delta Time, then when the second sub-time is 0.333 * Delta Time, the target weight Q is 0.333; when the second sub-time is 0.667 * Delta Time, the target weight Q is 0.667.

[0165] Specifically, the second sample screen space matrix S can be determined by the following formula. clip-2 S clip-2 =Q·(S) clip-1 +S clip-历史 )

[0166] Thus, by determining the value of the target weight Q, the screen space matrix S of the first sample can be clearly defined. clip -1 and historical sample screen space matrix S clip -Which historical moment's second sample screen space matrix S is interpolated between? clip-2 .

[0167] Step 504: Based on the second sample screen space matrix S clip-2 Sample local space matrix S localThe second view matrix V2 and the second projection matrix P2 are used to determine the second model matrix M2 corresponding to the second sub-time point.

[0168] Specifically, due to the second sample screen space matrix S clip-2 It is also possible to use the second view matrix V2, the second projection matrix P2, and the second model matrix M2 at the second sub-time point to analyze the sample local space matrix S. local We obtain it by spatial transformation, i.e., S clip-2 =P2 V2 M2S local Then, by performing related operations on the inverse matrix, the second model matrix M2 can be obtained:

[0169] in, This represents the inverse matrix of the second view matrix V2. This represents the inverse of the second projection matrix P2. S represents the sample local space matrix local The inverse matrix.

[0170] In this way, after obtaining the second model matrix M2, the second view matrix V2, and the second projection matrix P2 at the second sub-time point, these can be sent to the GPU as the second sub-image rendering data for rendering. The GPU can then perform spatial transformation based on these three matrices and the coordinates of all points in the local space, thereby obtaining the new physical position coordinates of the object in the screen space at the second sub-target time point.

[0171] In summary, by introducing the sample local space matrix S local The sample local space matrix S participates in the interpolation calculation of the second model matrix M2. local The number of sample points is relatively small, which can greatly reduce the amount of computation, and the second sample screen space matrix S is calculated. clip-2 This simple matrix can be used to accurately represent the position changes of points in the screen space at the second sub-time point. Then, the second model matrix M2 can be obtained through the operation of the inverse matrix, which can improve the interpolation accuracy of the second model matrix M2.

[0172] For a single object, such as a solid box, its physical position is simply the physical position of the box itself; there are no other physical positions. However, for a skeletal model, its physical position consists of two parts: one is the change in the physical position of the entire skeletal model, such as moving from position one to position two; the other is the change in the physical position of the individual bones that make up the skeletal model. For example, there will be relative movement between different bones in an arm, and the arm can be raised or lowered.

[0173] In one possible implementation, the dynamic variables may include a first sub-dynamic variable and a second sub-dynamic variable. The first sub-dynamic variable is used to identify changes in the physical position of the skeleton in the skeleton model, and the second sub-dynamic variable is used to identify changes in the physical position of each bone in the skeleton model.

[0174] The skeletal model can be a model composed of multiple bones, which can move relative to each other. For the skeletal model, its dynamic variables can be divided into first sub-dynamic variables and second sub-dynamic variables. The physical position of the entire skeletal model can be represented using the first sub-dynamic variable, while the physical position of each individual bone can be represented using the second sub-dynamic variable.

[0175] Similarly, the first model matrix can also include at least two parts: a first position matrix representing the physical motion of the entire skeleton, and a first skeleton matrix representing the physical motion of a single bone. That is, the first model matrix includes a first skeleton matrix and a first position matrix. The first position matrix is ​​used to identify the transformation matrix of the first sub-dynamic variable from local space to world space at the first sub-time point, and the first skeleton matrix is ​​used to identify the transformation matrix of the second sub-dynamic variable from local space to world space at the first sub-time point.

[0176] In short, the first position matrix reveals how to transform the entire skeletal model from local space to world space at the first sub-time point, and the first bone matrix reveals how to transform a single bone from local space to world space at the first sub-time point.

[0177] The historical model matrix can also include at least two parts: the historical skeleton matrix and the historical position matrix. The historical position matrix is ​​used to identify the transformation matrix of the first sub-dynamic variable from local space to world space at the historical execution time, and the historical skeleton matrix is ​​used to identify the transformation matrix of the second sub-dynamic variable from local space to world space at the historical execution time.

[0178] In short, the historical position matrix reveals how the entire skeletal model is transformed from local space to world space at a given historical execution moment, and the historical skeleton matrix reveals how a single bone is transformed from local space to world space at a given historical execution moment.

[0179] Based on the first model matrix and the historical model matrix, screen space interpolation is performed by the second thread to determine the second model matrix corresponding to the second sub-time point. Specifically, based on the first skeleton matrix and the historical skeleton matrix, screen space interpolation is performed by the second thread to determine the second skeleton matrix corresponding to the second sub-time point, and based on the first position matrix and the historical position matrix, screen space interpolation is performed by the second thread to determine the second position matrix corresponding to the second sub-time point; the second skeleton matrix and the second position matrix are used as the second model matrix.

[0180] Specifically, when determining the second model matrix through interpolation, the skeletal model corresponding to the entire skeleton and the positional models corresponding to each bone can be interpolated separately. The second model matrix includes a second position matrix reflecting the positional changes of the entire skeleton and a second skeleton matrix reflecting the positional changes of individual bones. That is, the second position matrix is ​​used to identify the transformation matrix of the first sub-dynamic variable from local space to world space at the second sub-target time, and the second skeleton matrix is ​​used to identify the transformation matrix of the second sub-dynamic variable from local space to world space at the second sub-target time.

[0181] In other words, when calculating the second model matrix, the second position matrix and the second skeleton matrix can be obtained through interpolation. When determining the second position matrix, interpolation is performed based on the first position matrix and the historical position matrix; similarly, when determining the second skeleton matrix, interpolation is performed based on the first skeleton matrix and the historical skeleton matrix. In other words, when determining the new physical position of the skeleton model at the second sub-time point, the physical position changes of the entire skeleton and individual bones need to be determined separately.

[0182] In summary, for the dynamic variables of the skeletal model, by separately interpolating the position matrix of the entire skeletal body and the skeletal matrix of individual bones to form a second model matrix, the movement of the skeletal body can be made more natural and accurate, more consistent with the movement of real people, and the realism of the image display can be improved.

[0183] Next, we can introduce the process of the image rendering method from the overall framework. During the execution of the first logical frame, based on the historical state variables of the previous logical frame, i.e., the historical logical frame, N threads can be called to determine the first image rendering data of N first graphics frames. The N first graphics frames correspond to N first moments. By rendering the N first graphics frames, N images can be obtained, thereby improving the rendering speed and frame rate, making the image display more detailed and smooth.

[0184] In one approach, when determining the first image rendering data for N first graphics frames using N threads, each thread can determine the first image rendering data for a first graphics frame based on historical state variables and the thread's corresponding first moment, according to the target state variables, thereby improving the accuracy of each first image rendering data. In another approach, some threads (i.e., N1 first threads) can obtain the first sub-image rendering data calculated based on historical state variables. For another group of threads (i.e., N2 second threads), interpolation processing can be performed based on the first sub-image rendering data to obtain the second sub-image rendering data corresponding to each thread. This approach ensures the accuracy of the rendering data while reducing computational load and increasing computational speed.

[0185] During interpolation, static variables such as lighting can be interpolated using a thread. For dynamic variables related to physical motion, interpolation can be performed directly on the screen space coordinates or on the model matrix M to obtain the second model matrix at the second sub-time point, which is then used as the rendering data for the second sub-image. For the skeletal model, interpolation of the model matrix needs to be performed separately for the entire skeletal model and for individual bones to improve the smoothness and realism of the skeletal model's motion.

[0186] It should be noted that, based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods.

[0187] This application also provides an image rendering apparatus. Referring to FIG8, which is a schematic diagram of an image rendering apparatus 800 provided in this application, the image rendering apparatus 800 includes:

[0188] The acquisition unit 801 is used to acquire the historical state variable after the execution of the historical logical frame is completed when the first execution time of the first logical frame is reached; the historical execution time of the historical logical frame is before the first execution time, and the historical state variable is used to identify the historical image rendering data of the historical graphic frame obtained after the execution of the historical logical frame is completed.

[0189] The determining unit 802 is used to determine the first image rendering data of each of the N first graphics frames based on the historical state variables during the execution of the first logical frame through N threads; the N first graphics frames correspond one-to-one with N first moments, and different threads are used to determine the first image rendering data of the first graphics frames corresponding to different first moments, the N first moments are later than the historical execution moments and earlier than or equal to the first execution moments, and N is an integer greater than 1;

[0190] The rendering unit 803 is used to render based on the first image rendering data of the N first graphics frames to obtain N images.

[0191] Optionally, the determining unit is configured to:

[0192] The execution logic of the first logical frame is determined; the execution logic is used to identify the logic by which the first logical frame calculates the first image rendering data during execution.

[0193] During the execution of the first logical frame, for each of the N threads, based on the historical state variables and the first moment corresponding to the thread, the first image rendering data of the first graphics frame corresponding to the first moment is determined according to the execution logic.

[0194] Optionally, the determining unit is configured to:

[0195] Calculate the time difference between the first moment corresponding to the thread and the historical execution moment;

[0196] According to the execution logic, determine the target state variable after the time difference of the historical state variable;

[0197] The target state variable is transformed to obtain the first image rendering data of the first graphic frame corresponding to the first time moment.

[0198] Optionally, the N threads include N1 first threads and N2 second threads, where N1 + N2 = N, and N1 is a positive integer greater than 0 and less than N;

[0199] The determining unit is used for:

[0200] The first sub-image rendering data at the corresponding first sub-time point is obtained by the N1 first threads respectively, and the first sub-image rendering data of each first thread at the corresponding first sub-time point is obtained based on the historical state variables;

[0201] For each of the N2 second threads, based on the first sub-image rendering data, the second thread performs interpolation processing to determine the second sub-image rendering data at the corresponding second sub-time point. Each first sub-time point and each second sub-time point are different first times among the N first times.

[0202] N1 first sub-image rendering data and N2 second sub-image rendering data are used as N first image rendering data.

[0203] Optionally, the N threads include a third thread, which is either the first thread or the second thread, and the determining unit is used to:

[0204] Based on the historical state variables and N1 first sub-time points, the third thread determines the first sub-image rendering data corresponding to each of the N1 first sub-time points.

[0205] The first sub-image rendering data at the first sub-time point is obtained from the third thread through N1 first threads.

[0206] Optionally, the first image rendering data is rendering data used to reflect target state variables, and the target state variables include static variables used to identify changes in attribute state;

[0207] The determining unit is used for:

[0208] Based on the first sub-image rendering data and the historical image rendering data, linear interpolation is performed by the second thread to determine the corresponding second sub-time point's second sub-image rendering data.

[0209] Optionally, the first image rendering data is rendering data used to reflect target state variables, and the target state variables include dynamic variables used to identify physical motion;

[0210] The determining unit is used for:

[0211] Based on the first sub-image rendering data and the historical image rendering data, screen space interpolation processing is performed by the second thread to obtain the second sub-image rendering data corresponding to the second sub-time point; the first sub-image rendering data is the coordinates of the dynamic variable in screen space at the first sub-time point, and the historical image rendering data is the coordinates of the dynamic variable in screen space at the historical execution time point.

[0212] Optionally, the first image rendering data is rendering data used to reflect target state variables, and the target state variables include dynamic variables used to identify physical motion;

[0213] The first sub-image rendering data includes a first model matrix at the first sub-time point, which is used to identify the transformation matrix of the dynamic variable from local space to world space at the first sub-time point; the historical image rendering data includes a historical model matrix at the historical execution time point, which is used to identify the transformation matrix of the dynamic variable from local space to world space at the historical execution time point.

[0214] The determining unit is used for:

[0215] Based on the first model matrix and the historical model matrix, screen space interpolation is performed by the second thread to determine the second model matrix corresponding to the second sub-time point; the second model matrix is ​​used to identify the transformation matrix of the dynamic variable from the local space to the world space at the second sub-time point;

[0216] Obtain the second view matrix at the second sub-time point where the dynamic variable is transformed from the world space to the view space, and obtain the second projection matrix at the second sub-time point where the dynamic variable is transformed from the view space to the screen space;

[0217] The second view matrix, the second projection matrix, and the second model matrix are used as the second sub-image rendering data corresponding to the second sub-time point.

[0218] Optionally, the determining unit is configured to:

[0219] The second thread obtains the first spatial transformation matrix corresponding to the first sub-time point and the historical spatial transformation matrix corresponding to the historical execution time point. The first spatial transformation matrix includes a first view matrix, a first projection matrix, and a first model matrix, used to identify the transformation of the dynamic variable from the world space to the view space at the first sub-time point; the historical spatial transformation matrix includes a historical view matrix, a historical projection matrix, and a historical model matrix, used to identify the transformation of the dynamic variable from the world space to the view space at the historical execution time point;

[0220] The first spatial transformation matrix is ​​used to perform spatial transformation processing on the sample local space matrix to determine the first sample screen space matrix of the sample point in the screen space at the first sub-time point. The historical spatial transformation matrix is ​​used to perform spatial transformation processing on the sample local space matrix to determine the historical sample screen space matrix of the sample point in the screen space at the historical execution time point. The sample local space matrix includes the coordinates of multiple linearly independent sample points in the local space.

[0221] Based on the first sample screen space matrix and the historical sample screen space matrix, interpolation is performed using the target weight corresponding to the second sub-time point to determine the second sample screen space matrix corresponding to the second sub-time point; the target weight is positively correlated with the time interval between the second sub-time point and the historical execution time.

[0222] Based on the second sample screen space matrix, the sample local space matrix, the second view matrix, and the second projection matrix, the second model matrix corresponding to the second sub-time point is determined.

[0223] Optionally, the dynamic variable includes a first sub-dynamic variable and a second sub-dynamic variable. The first sub-dynamic variable is a dynamic variable used to identify changes in the physical position of the skeleton in the skeleton model, and the second sub-dynamic variable is a dynamic variable used to identify changes in the physical position of each bone in the skeleton model.

[0224] The first model matrix includes a first skeleton matrix and a first position matrix. The first position matrix is ​​used to identify the transformation matrix of the first sub-dynamic variable from the local space to the world space at the first sub-time point. The first skeleton matrix is ​​used to identify the transformation matrix of the second sub-dynamic variable from the local space to the world space at the first sub-time point.

[0225] The historical model matrix includes a historical skeleton matrix and a historical position matrix. The historical position matrix is ​​used to identify the transformation matrix of the first sub-dynamic variable from the local space to the world space at the historical execution time. The historical skeleton matrix is ​​used to identify the transformation matrix of the second sub-dynamic variable from the local space to the world space at the historical execution time.

[0226] The determining unit is used for:

[0227] Based on the first skeleton matrix and the historical skeleton matrix, screen space interpolation is performed by the second thread to determine the second skeleton matrix corresponding to the second sub-time point. Similarly, based on the first position matrix and the historical position matrix, screen space interpolation is performed by the second thread to determine the second position matrix corresponding to the second sub-time point. The second position matrix is ​​used to identify the transformation matrix of the first sub-dynamic variable from the local space to the world space at the second sub-time point, and the second skeleton matrix is ​​used to identify the transformation matrix of the second sub-dynamic variable from the local space to the world space at the second sub-time point.

[0228] The second skeleton matrix and the second position matrix are used as the second model matrix.

[0229] Optionally, the device further includes:

[0230] The acquisition unit is used to acquire the first state variable after the execution of the first logical frame is completed when the second execution time of the second logical frame is reached;

[0231] The determining unit is used, during the execution of the second logical frame, to determine the second image rendering data of each of the M second graphics frames based on the first state variable through M threads; the M second graphics frames correspond one-to-one with the M second moments, and different threads are used to determine the second image rendering data of the second graphics frames corresponding to different second moments, the M second moments being later than the first execution moment and earlier than or equal to the second execution moment; the M threads and the N threads are at least partially the same or completely different, and M is an integer greater than 1.

[0232] This application also provides a computer device, which is the computer device described above, and may include a terminal device or a server. The aforementioned image rendering apparatus may be configured in this computer device. The computer device will now be described in conjunction with the accompanying drawings.

[0233] If the computer device is a terminal device, please refer to Figure 9. This application embodiment provides a terminal device, taking a mobile phone as an example:

[0234] Figure 9 shows a block diagram of a portion of the structure of a mobile phone related to the terminal device provided in the embodiments of this application. Referring to Figure 9, the mobile phone includes components such as a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a Wi-Fi module 1470, a processor 1480, and a power supply 1490. Those skilled in the art will understand that the mobile phone structure shown in Figure 9 does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0235] The following section, with reference to Figure 9, provides a detailed introduction to each component of the mobile phone:

[0236] The RF circuit 1410 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 1480; in addition, it transmits uplink data to the base station.

[0237] The memory 1420 can be used to store software programs and modules. The processor 1480 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1420. The memory 1420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0238] The input unit 1430 can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1430 may include a touch panel 1431 and other input devices 1432.

[0239] The display unit 1440 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1440 may include a display panel 1441.

[0240] The mobile phone may also include at least one sensor 1450, such as a light sensor, a motion sensor, and other sensors.

[0241] Audio circuitry 1460, speaker 1461, and microphone 1462 provide an audio interface between the user and the mobile phone.

[0242] WiFi is a short-range wireless transmission technology. Through the WiFi module 1470, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access.

[0243] The processor 1480 is the control center of the mobile phone. It connects various parts of the mobile phone through various interfaces and lines. It performs various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 1420 and calling data stored in the memory 1420.

[0244] The mobile phone also includes a power supply 1490 (such as a battery) that powers the various components.

[0245] In this embodiment, the processor 1480 included in the terminal device is also used to execute an image rendering method.

[0246] If the computer device is a server, this application embodiment also provides a server, as shown in Figure 10. Figure 10 is a structural diagram of the server 1500 provided in this application embodiment. The server 1500 can vary significantly due to different configurations or performance, and may include one or more processors, such as a central processing unit (CPU), and a memory 1532, and one or more storage media 1530 (e.g., one or more mass storage devices) for storing application programs 1542 or data 1544. The memory 1532 and storage media 1530 can be temporary or persistent storage. The program stored in the storage media 1530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1522 may be configured to communicate with the storage media 1530 and execute the series of instruction operations in the storage media 1530 on the server 1500.

[0247] Server 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input / output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0248] The steps performed by the server in the above embodiments can be based on the server structure shown in Figure 10.

[0249] In addition, this application embodiment also provides a storage medium for storing a computer program for executing the method provided in the above embodiment.

[0250] This application also provides a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the method provided in the above embodiments.

[0251] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk or optical disk, and other media that can store computer programs.

[0252] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0253] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0254] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Moreover, based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image rendering method, executed by a computer device, the method comprising: When the first execution moment of the first logical frame is reached, the historical state variables after the execution of the historical logical frame are obtained; The historical execution time of the historical logical frame is before the first execution time, and the historical state variable is used to identify the historical image rendering data of the historical graphic frame obtained after the historical logical frame is executed. During the execution of the first logical frame, based on the historical state variables, the first image rendering data of each of the N first graphics frames is determined by N threads; the N first graphics frames correspond one-to-one with the N first moments, and different threads are used to determine the first image rendering data of the first graphics frames corresponding to different first moments. The N first moments are later than the historical execution moment and earlier than or equal to the first execution moment, and N is an integer greater than 1. Rendering is performed based on the first image rendering data of the N first graphic frames to obtain N images.

2. The method according to claim 1, wherein, During the execution of the first logical frame, based on the historical state variables, the first image rendering data for each of the N first graphics frames is determined through N threads, including: The execution logic of the first logical frame is determined; the execution logic is used to identify the logic by which the first logical frame calculates the first image rendering data during execution. During the execution of the first logical frame, for each of the N threads, based on the historical state variables and the first moment corresponding to the thread, the first image rendering data of the first graphics frame corresponding to the first moment is determined according to the execution logic.

3. The method according to claim 2, wherein, The step of determining the first image rendering data of the first graphics frame corresponding to the first moment according to the execution logic based on the historical state variables and the first moment corresponding to the thread includes: Calculate the time difference between the first moment corresponding to the thread and the historical execution moment; According to the execution logic, determine the target state variable after the time difference of the historical state variable; The target state variable is transformed to obtain the first image rendering data of the first graphic frame corresponding to the first time moment.

4. The method according to any one of claims 1 to 3, wherein, The N threads include N1 first threads and N2 second threads, where N1 + N2 = N, and N1 is a positive integer greater than 0 and less than N; During the execution of the first logical frame, based on the historical state variables, the first image rendering data for each of the N first graphics frames is determined through N threads, including: The first sub-image rendering data at the corresponding first sub-time point is obtained by the N1 first threads respectively, and the first sub-image rendering data of each first thread at the corresponding first sub-time point is obtained based on the historical state variables; For each of the N2 second threads, based on the first sub-image rendering data, the second thread performs interpolation processing to determine the second sub-image rendering data at the corresponding second sub-time point. Each first sub-time point and each second sub-time point are different first times among the N first times. N1 first sub-image rendering data and N2 second sub-image rendering data are used as N first image rendering data.

5. The method according to claim 4, wherein, The N threads include a third thread, which is either the first thread or the second thread. The step of obtaining the first sub-image rendering data at the corresponding first sub-time point through the N1 first threads includes: Based on the historical state variables and N1 first sub-time points, the third thread determines the first sub-image rendering data corresponding to each of the N1 first sub-time points. The first sub-image rendering data at the first sub-time point is obtained from the third thread through N1 first threads.

6. The method according to claim 4 or 5, wherein, The first image rendering data is rendering data used to reflect target state variables, and the target state variables include static variables used to identify changes in attribute state; The step of determining the second sub-image rendering data at the corresponding second sub-time point by performing interpolation processing through the second thread based on the first sub-image rendering data includes: Based on the first sub-image rendering data and the historical image rendering data, linear interpolation is performed by the second thread to determine the corresponding second sub-time point's second sub-image rendering data.

7. The method according to any one of claims 4 to 6, wherein, The first image rendering data is rendering data used to reflect target state variables, and the target state variables include dynamic variables used to identify physical motion; The step of determining the second sub-image rendering data at the corresponding second sub-time point by performing interpolation processing through the second thread based on the first sub-image rendering data includes: Based on the first sub-image rendering data and the historical image rendering data, screen space interpolation processing is performed by the second thread to obtain the second sub-image rendering data corresponding to the second sub-time point; the first sub-image rendering data is the coordinates of the dynamic variable in screen space at the first sub-time point, and the historical image rendering data is the coordinates of the dynamic variable in screen space at the historical execution time point.

8. The method according to any one of claims 4 to 7, wherein, The first image rendering data is rendering data used to reflect target state variables, and the target state variables include dynamic variables used to identify physical motion; The first sub-image rendering data includes a first model matrix at the first sub-time point, which is used to identify the transformation matrix of the dynamic variable from local space to world space at the first sub-time point; the historical image rendering data includes a historical model matrix at the historical execution time point, which is used to identify the transformation matrix of the dynamic variable from local space to world space at the historical execution time point. The step of determining the second sub-image rendering data at the corresponding second sub-time point by performing interpolation processing through the second thread based on the first sub-image rendering data includes: Based on the first model matrix and the historical model matrix, screen space interpolation is performed by the second thread to determine the second model matrix corresponding to the second sub-time point; the second model matrix is ​​used to identify the transformation matrix of the dynamic variable from the local space to the world space at the second sub-time point; Obtain the second view matrix at the second sub-time point where the dynamic variable is transformed from the world space to the view space, and obtain the second projection matrix at the second sub-time point where the dynamic variable is transformed from the view space to the screen space; The second view matrix, the second projection matrix, and the second model matrix are used as the second sub-image rendering data corresponding to the second sub-time point.

9. The method according to claim 8, wherein, The step of determining the second model matrix corresponding to the second sub-time point by performing screen space interpolation processing through the second thread based on the first model matrix and the historical model matrix includes: The second thread obtains the first spatial transformation matrix corresponding to the first sub-time point and the historical spatial transformation matrix corresponding to the historical execution time point. The first spatial transformation matrix includes a first view matrix, a first projection matrix, and a first model matrix, used to identify the transformation of the dynamic variable from the world space to the view space at the first sub-time point; the historical spatial transformation matrix includes a historical view matrix, a historical projection matrix, and a historical model matrix, used to identify the transformation of the dynamic variable from the world space to the view space at the historical execution time point; The first spatial transformation matrix is ​​used to perform spatial transformation processing on the sample local space matrix to determine the first sample screen space matrix of the sample point in the screen space at the first sub-time point. The historical spatial transformation matrix is ​​used to perform spatial transformation processing on the sample local space matrix to determine the historical sample screen space matrix of the sample point in the screen space at the historical execution time point. The sample local space matrix includes the coordinates of multiple linearly independent sample points in the local space. Based on the first sample screen space matrix and the historical sample screen space matrix, interpolation is performed using the target weight corresponding to the second sub-time point to determine the second sample screen space matrix corresponding to the second sub-time point; the target weight is positively correlated with the time interval between the second sub-time point and the historical execution time. Based on the second sample screen space matrix, the sample local space matrix, the second view matrix, and the second projection matrix, the second model matrix corresponding to the second sub-time point is determined.

10. The method according to claim 8 or 9, wherein, The dynamic variables include a first sub-dynamic variable and a second sub-dynamic variable. The first sub-dynamic variable is a dynamic variable used to identify changes in the physical position of the skeleton in the skeleton model, and the second sub-dynamic variable is a dynamic variable used to identify changes in the physical position of each bone in the skeleton model. The first model matrix includes a first skeleton matrix and a first position matrix. The first position matrix is ​​used to identify the transformation matrix of the first sub-dynamic variable from the local space to the world space at the first sub-time point. The first skeleton matrix is ​​used to identify the transformation matrix of the second sub-dynamic variable from the local space to the world space at the first sub-time point. The historical model matrix includes a historical skeleton matrix and a historical position matrix. The historical position matrix is ​​used to identify the transformation matrix of the first sub-dynamic variable from the local space to the world space at the historical execution time. The historical skeleton matrix is ​​used to identify the transformation matrix of the second sub-dynamic variable from the local space to the world space at the historical execution time. The step of determining the second model matrix corresponding to the second sub-time point by performing screen space interpolation processing through the second thread based on the first model matrix and the historical model matrix includes: Based on the first skeleton matrix and the historical skeleton matrix, screen space interpolation is performed by the second thread to determine the second skeleton matrix corresponding to the second sub-time point. Similarly, based on the first position matrix and the historical position matrix, screen space interpolation is performed by the second thread to determine the second position matrix corresponding to the second sub-time point. The second position matrix is ​​used to identify the transformation matrix of the first sub-dynamic variable from the local space to the world space at the second sub-time point, and the second skeleton matrix is ​​used to identify the transformation matrix of the second sub-dynamic variable from the local space to the world space at the second sub-time point. The second bone matrix and the second position matrix are used as the second model matrix.

11. The method according to any one of claims 1-10, wherein, The method further includes: When the second execution time of the second logical frame is reached, the first state variable after the execution of the first logical frame is completed is obtained; During the execution of the second logical frame, based on the first state variable, the second image rendering data of each of the M second graphics frames is determined by M threads; the M second graphics frames correspond one-to-one with the M second moments, and different threads are used to determine the second image rendering data of the second graphics frames corresponding to different second moments. The M second moments are later than the first execution moment and earlier than or equal to the second execution moment; the M threads and the N threads are at least partially the same or completely different, and M is an integer greater than 1.

12. An image rendering apparatus, wherein, The device includes: The acquisition unit is used to acquire historical state variables after the execution of the historical logical frame is completed when the first execution time of the first logical frame is reached; the historical execution time of the historical logical frame is before the first execution time, and the historical state variables are used to identify the historical image rendering data of the historical graphic frame obtained after the execution of the historical logical frame; The determining unit is used to determine the first image rendering data of each of the N first graphics frames based on the historical state variables during the execution of the first logical frame through N threads; the N first graphics frames correspond one-to-one with N first moments, and different threads are used to determine the first image rendering data of the first graphics frames corresponding to different first moments, the N first target moments are later than the historical execution moments and earlier than or equal to the first execution moments, and N is an integer greater than 1; The rendering unit is used to render N images based on the first image rendering data of the N first graphics frames.

13. A computer device, wherein, The computer device includes a processor and memory: The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is configured to perform the method according to any one of claims 1-11 according to the computer program.

14. A computer-readable storage medium, wherein, The computer-readable storage medium is used to store a computer program that, when executed by a computer device, performs the method described in any one of claims 1-11.

15. A computer program product comprising a computer program, which, when run on a computer device, causes the computer device to perform the method of any one of claims 1-11.

Images