Image rendering method and computing system

Abstract

The present application relates to the technical field of computers, and provides an image rendering method and a computing system, which are used for improving image rendering efficiency. A first computing node in the computing system of the present application receives a control instruction sent by a user terminal, and determines an image rendering task corresponding to the control instruction, the image rendering task comprising a plurality of subtasks. The first computing node determines, on the basis of affinity between the plurality of subtasks and computing nodes, a corresponding computing node for each subtask, wherein the plurality of subtasks correspond to at least two types of computing nodes, and different types of computing nodes comprise different types of processors. The first computing node schedules the corresponding computing node of each subtask to execute the corresponding subtask. The first computing node allocates at least two types of computing nodes for the image rendering task, and the at least two types of computing nodes execute the respective corresponding subtasks. On the basis of such a heterogeneous architecture involving multiple types of computing nodes, image rendering efficiency can be improved.

Description

Image rendering method and computing system

[0001] Cross-reference to related applications

[0002] The present application claims priority to the Chinese patent application No. 202411125457.5, filed on August 15, 2024, and entitled "Image rendering method and computing system", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the field of computer technology, and in particular to an image rendering method and a computing system. BACKGROUND

[0004] Rendering refers to a process of generating an image from a model using software, where the model is a description of a three-dimensional object in a strictly defined language or data structure, which includes geometry, viewpoint, texture, and lighting information. With the development of deep neural networks, it is increasingly common to optimize the image synthesis and rendering process through neural network models.

[0005] Currently, neural rendering technology usually uses a graphic processing unit (GPU) for direct rendering. However, as real-time rendering applications require higher and higher quality and real-time performance, the amount of computation and the speed of calculation in the rendering process are very high. Taking a 3D game as an example, for a refresh rate of 30 frames per second, the rendering time of each frame of image cannot exceed 33 milliseconds. Therefore, based on a single GPU, the rendering efficiency is low and cannot meet the real-time rendering requirements. SUMMARY

[0006] The present application provides an image rendering method and a computing system to improve the efficiency of image rendering.

[0007] In a first aspect, the present application provides an image rendering method, which can be executed by a first computing node in a computing system, or a chip inside the first computing node, or a processor (such as a central processing unit (CPU), a GPU, a neural network processing unit (NPU), a data processing unit (DPU), a field programmable gate array (FPGA), etc.) in the first computing node. Hereinafter, the execution by the first computing node is taken as an example. The first computing node receives a control instruction sent by a user terminal, determines an image rendering task corresponding to the control instruction, and the image rendering task includes a plurality of subtasks; the first computing node determines a computing node corresponding to each subtask based on the affinity between the plurality of subtasks and the computing nodes; wherein the plurality of subtasks correspond to at least two types of computing nodes, and different types of computing nodes include different types of processors; and the first computing node schedules the computing node corresponding to each subtask to execute the corresponding subtask.

[0008] Optionally, the first computing node schedules the computing node corresponding to each subtask to execute the corresponding subtask, which can specifically be that the first computing node executes the corresponding subtask, and / or schedules the computing node corresponding to other subtasks to execute the corresponding subtask.

[0009] Through the above method, when the first computing node executes the image rendering task according to the control instruction sent by the user terminal, the first computing node can reasonably allocate the corresponding computing node to each subtask in the image rendering task according to the affinity between the subtasks and the computing nodes, and the task execution efficiency can be improved when the affinity between the computing nodes and the subtasks is high. In addition, the first computing node allocates at least two types of computing nodes for the image rendering task, and executes the corresponding subtasks through the at least two types of computing nodes, and the image rendering efficiency can be improved based on the heterogeneous architecture of the multiple types of computing nodes.

[0010] In a possible implementation, the image rendering task includes at least one subtask including a first function. For the first function, the first computing node can determine the computing node corresponding to the first function based on the affinity between the first function and the computing node.

[0011] Through the above method, for the first function (such as an operator), the first computing node can determine the computing node corresponding to the first function according to the affinity between the first function and the computing node. Based on this method, the first computing node can dynamically adjust the computing node corresponding to the first function during the execution of the subtasks by the computing nodes, and further improve the image rendering efficiency.

[0012] In a possible implementation, the first computing node determines the computing node corresponding to each subtask based on the load information of the plurality of computing nodes.

[0013] In the manner described above, the first computing node can reasonably allocate the corresponding computing node for each subtask according to the current load information of each computing node, so that the computing power of each computing node is more balanced, thereby improving the efficiency of each computing node in performing image rendering.

[0014] In a possible implementation, the first computing node determines a type of computing node corresponding to each subtask based on the affinity between the plurality of subtasks and the computing nodes; for each subtask, if the type of computing node corresponding to the subtask includes a plurality of computing nodes, the computing node corresponding to the subtask is determined according to the load information of the plurality of computing nodes.

[0015] In the manner described above, the first computing node can more reasonably allocate the corresponding computing node for each subtask in combination with the affinity between the subtask and the computing node and the load information of each computing node, so that the subtask and the corresponding computing node are more suitable and the computing power of each computing node is more balanced, thereby improving the efficiency of each computing node in performing image rendering.

[0016] In a possible implementation, the first computing node determines a plurality of candidate computing nodes corresponding to each subtask based on the load information of the plurality of computing nodes, and the plurality of candidate computing nodes are computing nodes of different types; for each subtask, the computing node corresponding to the subtask is determined from the plurality of candidate computing nodes based on the affinity between the subtask and each candidate computing node.

[0017] In the manner described above, the first computing node can more reasonably allocate the corresponding computing node for each subtask in combination with the affinity between the subtask and the computing node and the load information of each computing node, so that the subtask and the corresponding computing node are more suitable and the computing power of each computing node is more balanced, thereby improving the efficiency of each computing node in performing image rendering.

[0018] In a possible implementation, a direct link is included between the computing nodes of different types, and the direct link is used to transmit data between the computing nodes of different types.

[0019] In the manner described above, the direct link is included between the computing nodes of different types, thereby improving the data transmission efficiency in the image rendering process.

[0020] In a possible implementation, the computing node includes at least one of a CPU, a GPU, an NPU, a DPU, and an FPGA.

[0021] In this way, the image rendering task can be executed by multiple different types of computing nodes.

[0022] In a possible implementation, the multiple subtasks include first type subtasks and second type subtasks, the first type subtasks include at least one subtask for implementing raster rendering, and the second type subtasks include at least one subtask for implementing neural rendering; the first type subtasks correspond to GPUs, and at least one subtask in the second type subtasks corresponds to an NPU.

[0023] In this way, the first computing node allocates GPUs for raster rendering subtasks and allocates NPU for neural rendering subtasks, so that the affinity between the subtasks and the computing nodes is higher, thereby improving the efficiency of image rendering.

[0024] In a second aspect, the present application provides a computing system, which can include multiple computing nodes; a first computing node in the multiple computing nodes, configured to determine an image rendering task corresponding to a control instruction sent by a user terminal, the image rendering task including multiple subtasks; determine a computing node corresponding to each subtask based on the affinity between the multiple subtasks and the computing nodes; wherein the multiple subtasks correspond to at least two types of computing nodes, and different types of computing nodes include different types of processors; each computing node in the computing system is configured to execute the corresponding subtask.

[0025] In a possible implementation, at least one subtask included in the image rendering task includes a first function, and the first computing node is further configured to determine a computing node corresponding to the first function based on the affinity between the first function and the computing node.

[0026] In a possible implementation, the first computing node is further configured to determine a computing node corresponding to each subtask based on load information of the multiple computing nodes.

[0027] In a possible implementation, the first computing node is further configured to: determine a type of computing node corresponding to each subtask based on the affinity between the multiple subtasks and the computing nodes; and for each subtask, if the type of computing node corresponding to the subtask includes multiple computing nodes, determine a computing node corresponding to the subtask according to load information of the multiple computing nodes.

[0028] In a possible implementation, the first computing node is further configured to: determine multiple candidate computing nodes corresponding to each subtask based on load information of the multiple computing nodes, the multiple candidate computing nodes being different types of computing nodes; and for each subtask, determine a computing node corresponding to the subtask from the multiple candidate computing nodes based on the affinity between the subtask and each candidate computing node.

[0029] In a possible implementation, a pass-through link is included between the two different types of computing nodes, and the pass-through link is configured to transmit data between the two different types of computing nodes.

[0030] In a possible implementation, the computing node includes at least one of a CPU, a GPU, an NPU, a DPU, and an FPGA.

[0031] In a possible implementation, the plurality of sub-tasks includes first-type sub-tasks and second-type sub-tasks, the first-type sub-tasks include at least one sub-task for implementing raster rendering, and the second-type sub-tasks include at least one sub-task for implementing neural rendering; the first-type sub-tasks correspond to the GPU, and at least one sub-task in the second-type sub-tasks corresponds to the NPU.

[0032] In a third aspect, the present application also provides a computing device having the functions of the above-mentioned method examples of the first aspect, and the beneficial effects can be referred to the description of the first aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. In a possible design, the computing device can include an information transmission module, a computing power allocation module, and a scheduling module; wherein:

[0033] The information transmission module is configured to receive a control instruction sent by a user terminal, determine an image rendering task corresponding to the control instruction, and the image rendering task includes a plurality of sub-tasks.

[0034] The computing power allocation module is configured to determine, based on the affinity between the plurality of sub-tasks and the computing nodes, a computing node corresponding to each sub-task; wherein the plurality of sub-tasks correspond to at least two types of computing nodes, and the different types of computing nodes include different types of processors.

[0035] The scheduling module is configured to schedule the computing node corresponding to each sub-task to execute the corresponding sub-task.

[0036] The functions that can be achieved by each module are specifically described in the detailed description of the method examples of the first aspect, and will not be described here.

[0037] In a fourth aspect, the present application also provides a computing device including at least one processor and at least one memory; wherein one or more memories store one or more computer programs including instructions, and when the instructions are executed by one or more processors, the computing device executes the method provided by any possible implementation of the above-mentioned first aspect.

[0038] In a fifth aspect, the present application also provides a communication system including a user terminal and a computing system provided by any possible implementation of the above-mentioned second aspect.

[0039] In a sixth aspect, the present application provides a computer readable storage medium, which, when executed by a computing device, causes the computing device to perform the method provided by any possible implementation of the first aspect. The storage medium stores computer program instructions. The storage medium includes, but is not limited to, volatile memory, such as random access memory (RAM), and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid state drive (SSD).

[0040] In a seventh aspect, the present application provides a computer program product, which includes computer program instructions, and when executed by a computing device, causes the computing device to perform the method provided by any possible implementation of the first aspect. The computer program product can be a software package, which can be downloaded and installed on a computer when the method provided by any possible implementation of the first aspect is needed.

[0041] In an eighth aspect, the present application also provides a computer chip, which is connected with a memory, and is configured to read and execute computer program instructions stored in the memory, and perform the method provided by any possible implementation of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0042] FIG. 1 is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0043] FIG. 2A is a schematic diagram of a structure of a computing system provided by an embodiment of the present application;

[0044] FIG. 2B is a schematic diagram of a structure of a computing system provided by an embodiment of the present application;

[0045] FIG. 2C is a schematic diagram of a structure of a computing system provided by an embodiment of the present application;

[0046] FIG. 3A is a schematic diagram of a structure of a computing system provided by an embodiment of the present application;

[0047] FIG. 3B is a schematic diagram of a structure of a computing system provided by an embodiment of the present application;

[0048] FIG. 3C is a schematic diagram of a structure of a computing system provided by an embodiment of the present application;

[0049] FIG. 4 is a schematic diagram of a structure of a computing system provided by an embodiment of the present application;

[0050] FIG. 5 is a schematic diagram of a flow of an image rendering method provided by an embodiment of the present application;

[0051] FIG. 6 is a schematic diagram of a load sensing principle according to an embodiment of the present application;

[0052] FIG. 7 is a schematic diagram of an image rendering process according to an embodiment of the present application;

[0053] FIG. 8 is a schematic diagram of an image rendering process performed by a cloud server according to an embodiment of the present application;

[0054] FIG. 9 is a schematic diagram of a computing device according to an embodiment of the present application;

[0055] FIG. 10 is a schematic diagram of a computing device according to an embodiment of the present application;

[0056] FIG. 11 is a schematic diagram of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0058] Before introducing the specific solutions provided by the embodiments of the present application, some terms in the present application are explained to facilitate understanding by those skilled in the art; it should be understood that the explanation of some terms is only used to explain the specific embodiments of the present application, and is not intended to limit the present application.

[0059] 1. Neural rendering:

[0060] Neural rendering is a rendering method based on deep neural networks and physical engines, which can create new images or video clips from existing scenes. Users can freely define scene attributes, such as lighting, camera parameters, pose, geometry, shape, and model structure, etc. Based on the neural rendering technology, controllable and realistic 3D scene models can be rendered. Neural rendering is a rendering generation technology that generates new scenes based on existing image and video data.

[0061] 2. Rasterization-based rendering:

[0062] Rasterization rendering is the process of converting a three-dimensional model into a two-dimensional image, also known as rasterization. It is a basic technique in computer graphics, which decomposes a three-dimensional model into a series of pixels and calculates the color value of each pixel to generate a two-dimensional image. Rasterization rendering is suitable for most computer graphics applications, including games, movie special effects, etc. The basic steps of rasterization rendering include:

[0063] a. Modeling: use vertices, edges and faces to represent the geometric structure of a three-dimensional object.

[0064] b. Transformation: use matrix operations to perform translation, rotation, scaling, etc. on three-dimensional objects, and convert them from the world coordinate system to the camera coordinate system.

[0065] c. Clipping: use a view frustum to remove objects or parts of objects that are not within the field of view.

[0066] d. Projection: use perspective projection or orthogonal projection to convert three-dimensional objects from the camera coordinate system to the screen coordinate system.

[0067] e. Rasterization: use scan line algorithm or triangle filling algorithm to convert triangle patches to pixel points and calculate the color value of each pixel point.

[0068] f. Shading: use shader programs to perform lighting calculations, texture mapping, fogging effects, etc. on each pixel point to enhance the realism and aesthetics of the image.

[0069] 3. Rendering pipeline:

[0070] The rendering pipeline is also called the rendering pipeline. The main function of the rendering pipeline is to determine the process of generating or drawing a two-dimensional image under the conditions of a given virtual camera, three-dimensional objects, light sources, lighting modes, and textures. The rendering pipeline is not fixed, and due to differences in software and hardware and platforms, there may be some differences in details. Broadly speaking, it contains four stages: application stage, geometry processing stage, rasterization stage, and pixel processing stage. Each stage is parallel processing.

[0071] 4. Graphics processor:

[0072] Used for processing computer graphics and image data, such as image / graphic rendering, image / graphic processing, etc. The graphics processor can be a GPU, which is used in graphics cards. The GPU has a large number of processing units and can quickly process a large amount of graphics and image data, greatly improving the computing speed and efficiency. In the field of AI, the GPU can be used to train neural networks or make inferences based on neural networks. The image processor can refer to an image processing chip, or a board card that integrates an image processing chip, which is plugged into the mainboard and communicates with the CPU, image processor, etc. through the bus.

[0073] 5. Neural network processor:

[0074] NPU is a special processor designed specifically for the computational needs of neural network models, aiming to efficiently and low-power execute machine learning, especially deep learning tasks. The core idea of NPU is to simulate the working principle of human brain neural network, through large-scale parallel processing units (similar to neurons) and efficient interconnection structure (similar to synapses), to realize the acceleration of large-scale matrix operation, convolution operation and other complex calculations in deep neural network. NPU is mainly applied in artificial intelligence related fields, such as autonomous driving, image recognition, speech recognition, natural language understanding, recommendation system, etc., especially in edge computing devices and AI servers in data centers. Unlike traditional general-purpose processors, NPU integrates highly customized computing units, storage structures and data flow scheduling mechanisms at the hardware level, which can efficiently handle feature extraction, weight update, activation function calculation and other operations in deep learning models. This design makes NPU show extremely high energy efficiency ratio when processing artificial intelligence tasks, that is, it provides far more computing performance than general-purpose processors under the same power consumption, especially in the inference stage (such as image recognition, speech recognition, natural language processing, etc.).

[0075] 6. Data processor:

[0076] The data processor is a data-centric, input-output (IO) intensive chip that can improve the efficiency of the computing system and reduce the cost of the computing system. It mainly faces the computing of data center infrastructure applications and can flexibly offload virtualization, network, storage, security and other service loads to reduce the load of the central processor. The data processor is used to offload part of the functions of the central processor, for example, high-performance data processing functions that are not suitable for the central processor in the network, storage or operating system can be offloaded to the data processor to release the computing power of the central processor. The data processor can be a DPU, an infrastructure processing unit (IPU), a system on chip (SoC), an iNIC or a smartNIC, etc. with offloading functions. Among them, the iNIC or smartNIC can be understood as an intelligent network card. The data processor can refer to a data processing chip, or a board card integrating a data processing chip, which is plugged into the mainboard and communicates with CPU, graphics processor and other components through bus.

[0077] In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b or c can represent a, b, c, a and b, a and c, b and c, or a, b and c, where each of a, b and c can be an element or a set containing one or more elements.

[0078] In the present application, "example", "in some embodiments", "in another embodiment" and the like are used to mean by way of example, illustration or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific way.

[0079] In the present application, "of", "corresponding" and "corresponding" can be used interchangeably at times. It should be pointed out that when the distinction is not emphasized, the meanings expressed are consistent. In the embodiments of the present application, communication and transmission can be used interchangeably at times. It should be pointed out that when the distinction is not emphasized, the meanings expressed are consistent. For example, transmission can include sending and / or receiving, and can be a noun or a verb.

[0080] It should be noted that the "first", "second" and the like in the embodiments of the present application are only used for the purpose of distinguishing description, and cannot be understood as indicating or implying relative importance, nor can it be understood as indicating or implying order.

[0081] The image rendering method provided by the embodiments of the present application can be applied to the system architecture as shown in FIG. 1, which includes at least one user terminal 10 and a computing system 20. In FIG. 1, at least one user terminal 10 is taken as an example including two user terminals. At least one user terminal 10 can access the computing system 20 through a communication network.

[0082] The user terminal 10 sends a control instruction to the computing system 20 in response to a user triggered operation; for example, taking a game APP running on the user terminal 10 as an example, the user can trigger a game related operation (such as upslide, downslide, screen rotation, etc.) in the display interface of the running game application (application, APP), the user terminal 10 generates a control instruction related to the operation in response to the operation triggered by the user in the display interface of the running game APP, and sends the control instruction to the computing system 20 through the communication network.

[0083] After the computing system 20 receives the control instruction sent by the user terminal 10, the received control instruction is decoded to obtain the image rendering control parameter corresponding to the control instruction; for example, the image rendering control parameter includes light quantity, material, viewing angle, etc. The computing system 20 determines the image rendering task corresponding to the control instruction according to the image rendering control parameter, and executes the image rendering task. The computing system 20 can obtain the rendered image data by executing the image rendering task; optionally, the image data can be encoded image data. The computing system 20 sends the rendered image data to the user terminal 10 through the communication network. Correspondingly, after the user terminal 10 receives the rendered image data sent by the computing system 20, the rendered image is displayed in the interface of the user terminal 10; optionally, the user terminal 10 decodes the received image data, and displays the rendered image in the interface according to the decoded image data.

[0084] The user terminal 10 in the system architecture of the embodiment of the application can be a mobile phone, a tablet computer, a notebook computer, a wearable device, a smart screen device, etc.

[0085] The computing system provided by the embodiment of the application will be described below in conjunction with the drawings. The computing system 20 includes a plurality of computing nodes. The computing system 20 of the embodiment of the application can be a heterogeneous architecture, and the plurality of computing nodes include different types of computing nodes; wherein the different types of computing nodes can include different types of processors, or the different types of computing nodes can be different types of processors. Optionally, the processor of the embodiment of the application can include CPU, GPU, NPU, DPU, FPGA, etc.

[0086] In a possible implementation, the computing node is a node comprising a processor. For example, the computing node can comprise a processor and other units / components / devices, such as the computing node comprising a processor and a memory. In this implementation, the computing system 20 can comprise multiple different types of computing nodes, and there can be one or more computing nodes of each type in the computing system 20. Optionally, each computing node can comprise one or more processors, and the multiple processors in one computing node can be of the same type or of different types. Taking the computing system 20 shown in FIG. 2A as an example, the computing system 20 comprises computing node 1, computing node 2, and computing node 3, where computing node 1 comprises a CPU, computing node 2 comprises a GPU, and computing node 3 comprises an NPU. Alternatively, taking the computing system 20 shown in FIG. 2B as an example, the computing system 20 comprises computing node 1, computing node 2, and computing node 3, where computing node 1 comprises a DPU, computing node 2 comprises a GPU, and computing node 3 comprises an NPU. Alternatively, taking the computing system 20 shown in FIG. 2C as an example, the computing system 20 comprises computing node 1, computing node 2, and computing node 3, where computing node 1 comprises a CPU and a DPU, computing node 2 comprises a GPU, and computing node 3 comprises an NPU. It should be noted that each type of computing node in FIG. 2A, FIG. 2B, and FIG. 2C is taken as an example, and in practice, there can be multiple computing nodes of each type.

[0087] In another possible implementation, the computing node is a processor. In this implementation, the computing system 20 comprises multiple different types of processors, and there can be one or more processors of each type in the computing system 20. Taking the computing system 20 shown in FIG. 3A as an example, the computing system 20 comprises a CPU, a GPU, and an NPU. Alternatively, taking the computing system 20 shown in FIG. 3B as an example, the computing system 20 comprises a DPU, a GPU, and an NPU. Alternatively, taking the computing system 20 shown in FIG. 3C as an example, the computing system 20 comprises a CPU, a DPU, a GPU, and an NPU. Each type of processor in FIG. 3A, FIG. 3B, and FIG. 3C is taken as an example, and in practice, there can be multiple processors of each type.

[0088] The computing system 20 can be a single computing device or a computing device cluster that can include multiple computing devices. Each computing device can be understood as a computer or a server, etc. The server can be a locally deployed server or a cloud server. When the computing system is a computing device cluster composed of multiple computing devices, different computing nodes can be deployed on the same computing device or on different computing devices. For example, when the computing system is a computing device cluster composed of two computing devices and the computing system includes computing node 1, computing node 2 and computing node 3, computing node 1 and computing node 2 can be deployed on one computing device and computing node 3 can be deployed on another computing device. For another example, when the computing system is a computing device cluster composed of three computing devices and the computing system includes computing node 1, computing node 2 and computing node 3, computing node 1, computing node 2 and computing node 3 can be deployed on one computing device respectively.

[0089] When multiple computing nodes are deployed on the same computing device, the multiple computing nodes can be connected through a peripheral component interconnect Express (PCIe) bus or an extended industry standard architecture (EISA) bus. For example, the computing system 20 includes a CPU, a GPU and an NPU, and when the CPU, the GPU and the NPU are deployed on one computing device, the CPU, the GPU and the NPU can be connected through a PCIe bus or an EISA bus.

[0090] Optionally, when the plurality of computing nodes are deployed on the same computing device, a pass-through link is included between the different types of computing nodes, which can also be referred to as a peer to peer (P2P) link, and the pass-through link is used to transmit data between two different types of computing nodes. For example, as shown in FIG. 4, the computing system 20 includes a plurality of different types of computing nodes, and each type of computing node can include a plurality of computing nodes; for example, a plurality of computing nodes including a CPU and dynamic random access memory (DRAM), a plurality of computing nodes including a GPU and high bandwidth memory (HBM), and a plurality of computing nodes including an NPU and HBM. Data transmission can be performed between the CPU and the GPU through a PCIe bus, data transmission can be performed between the CPU and the NPU through a PCIe bus, and a pass-through link is included between the GPU and the NPU. The data processing result of the GPU can be transmitted to the NPU through the pass-through link, and correspondingly, the data processing result of the NPU can also be transmitted to the GPU through the pass-through link. Based on this, the data processing result of the GPU does not need to be forwarded through the CPU, and can be directly transmitted to the NPU through the pass-through link; and the data processing result of the NPU does not need to be forwarded through the CPU, and can be directly transmitted to the GPU through the pass-through link; in this way, the data transmission efficiency between the GPU and the NPU can be reduced, and the data transmission delay can be reduced.

[0091] When the plurality of computing nodes are deployed on different computing devices, the computing nodes on different computing devices can perform data transmission through a communication network.

[0092] Based on the above-described computing system, the computing system can be used to perform an image rendering process. As shown in FIG. 5, the computing system of the embodiment of the present application performs an image rendering process, and the process can specifically include the following steps:

[0093] Step 501: A first computing node in the computing system determines an image rendering task corresponding to a control instruction sent by a user terminal. The image rendering task includes a plurality of sub-tasks.

[0094] In the embodiment of the present application, a user can operate a user terminal, and an APP can be run or a web page can be displayed on a display interface of the user terminal. The user can trigger an operation related to page display in the display interface of the user terminal. The user terminal responds to the operation of the user and sends a control instruction to the computing system; optionally, the control instruction can be used to represent a parameter related to page refreshing, that is, the control instruction can represent a page that will be presented in the display interface of the user terminal based on the user operation.

[0095] In a possible implementation, the computing system can include an information interaction unit; the user terminal sends the control instruction to the information interaction unit in the computing system in response to the operation of the user. For example, when the computing system is deployed in the cloud, the computing system can include a cloud container, which can also be referred to as a cloud mobile container; the cloud container receives the control instruction sent by the user terminal, decodes the control instruction, and obtains the image rendering control parameter. For example, taking the running of a game APP on the user terminal as an example, the user can trigger a game-related operation (such as an upslide, a downslide, a screen rotation, or the like) in the display interface of the running game APP; the user terminal generates a control instruction related to the operation triggered by the user in the display interface of the running game APP, and sends the control instruction to the cloud container. After receiving the control instruction, the cloud container decodes the control instruction to obtain an image rendering control parameter corresponding to the control instruction; for example, the image rendering control parameter includes light quantity, material, viewing angle, or the like. The cloud container sends the decoded image rendering control parameter to the first computing node in the computing system.

[0096] The first computing node obtains the image rendering control parameter, and determines an image rendering task corresponding to the control instruction according to the image rendering control parameter. The computing system can complete the operation corresponding to the control instruction sent by the user terminal by executing the image rendering task, and obtain a rendering image corresponding to the control instruction.

[0097] Optionally, after determining the image rendering task, the first computing node can split the image rendering task into a plurality of subtasks. The plurality of subtasks can constitute a rendering pipeline of the image rendering task. For example, the image rendering task can include at least two of the following subtasks: rasterization, super-resolution, color gamut conversion, noise reduction, optical flow, frame interpolation, and encoding.

[0098] Step 502: The first computing node in the computing system determines a computing node corresponding to each subtask.

[0099] In step 502, the first computing node performs a computing power scheduling operation, and allocates a corresponding computing node for each subtask in the image rendering task.

[0100] In the embodiment of the application, the plurality of subtasks in the image rendering task correspond to at least two types of computing nodes, and the different types of computing nodes include different types of processors. For example, the plurality of subtasks in the image rendering task correspond to at least two types of processors; for example, the plurality of subtasks correspond to a CPU, a GPU, and an NPU, that is, the CPU, the GPU, and the NPU in the computing system jointly execute the image rendering task.

[0101] The scheme in which the first computing node determines a computing node corresponding to each subtask will be described in detail below.

[0102] Option 1: The first computing node determines, based on the affinity between the plurality of sub-tasks and the computing nodes, a computing node corresponding to each sub-task.

[0103] Optionally, the first computing node can be a computing node including a CPU or a DPU in the computing system, or the first computing node is a CPU or a DPU in the computing system.

[0104] In the embodiments of the present application, the affinity between the sub-task and the computing node can be used to represent whether the sub-task and the computing node are affinity. The affinity between the sub-task and the computing node can be understood as that the sub-task is suitable for running on the computing node. Correspondingly, the non-affinity between the sub-task and the computing node can be understood as that the sub-task is not suitable for running on the computing node.

[0105] After determining the plurality of sub-tasks included in the image rendering task, the first computing node respectively judges the affinity between each sub-task and various types of computing nodes in the computing system. When determining the computing node corresponding to each sub-task, the first computing node assigns the sub-task to the affinity computing node. For example, the sub-task is encoding, and the encoding sub-task is more affinity to the NPU. Therefore, the first computing node can determine that the encoding sub-task corresponds to the NPU.

[0106] In the embodiments of the present application, the first computing node can determine the affinity between each sub-task and various types of computing nodes in the computing system according to a plurality of different ways, which will be introduced as follows:

[0107] Implementation 1: The first computing node detects whether each sub-task and various types of computing nodes in the computing system are affinity. According to the detection result, the computing node in the computing system which is affinity with the sub-task is determined.

[0108] In the implementation, the first computing node can detect whether each sub-task and the computing node are affinity according to the following ways:

[0109] For each sub-task, the first computing node can obtain the time length occupied by the sub-task when executed on the corresponding computing node. According to the time length occupied by the sub-task, it is judged whether the sub-task and the computing node currently executing the sub-task are affinity. For example, if the time length occupied by the sub-task is greater than a first set threshold, it is determined that the sub-task and the computing node currently executing the sub-task are not affinity. If the time length occupied by the sub-task is not greater than the first set threshold, it is determined that the sub-task and the computing node currently executing the sub-task are affinity.

[0110] Optionally, in the case that the first function is included in the subtask, the first computing node can determine the affinity between the subtask and the computing node according to the affinity between the first function and the computing node. The first computing node can determine whether the first function is affinity with the computing node currently executing the first function according to the time length occupied by the first function on the corresponding computing node; for example, if the time length occupied by the first function is greater than a second set threshold, it is determined that the first function is not affinity with the computing node currently executing the first function; if the time length occupied by the first function is not greater than the second set threshold, it is determined that the first function is affinity with the computing node currently executing the first function. The first set threshold and the second set threshold can be the same or different.

[0111] It should be understood that if multiple first functions are included in the subtask, the first computing node can determine the affinity between the subtask and the computing node according to the affinity between the multiple first functions and the computing node. For example, if the number of first functions affinity with the computing node in the subtask is greater than the number of first functions not affinity with the computing node, the first computing node can determine that the subtask is affinity with the computing node. Alternatively, if multiple first functions are included in the subtask, the first computing node can determine the affinity between the subtask and the computing node according to the affinity between a target first function in the multiple first functions and the computing node; the first computing node can select the target first function from the multiple first functions according to a set rule.

[0112] As a possible implementation, in the case that the first function is not included in the subtask, the first computing node can determine that the subtask is affinity with the set computing node; for example, in the case that the first function is not included in the subtask, the first computing node can determine that the subtask is affinity with the CPU or the computing node including the CPU.

[0113] Optionally, the first function can be an operator. For example, an artificial intelligence (AI) operator.

[0114] Implementation 2: The first computing node can obtain a mapping relationship between the subtasks and whether the subtasks are affinity with each type of computing node pre-configured; according to the mapping relationship, determine the computing node corresponding to each subtask included in the image rendering task.

[0115] Optionally, the mapping relationship can be pre-configured according to the design of the subtask and the hardware design of the computing node.

[0116] In implementation, the mapping relationship can record the type of the computing node to which each subtask is affinity, for example, the mapping relationship includes the affinity between the encoding subtask and the NPU and the affinity between the raster rendering subtask and the GPU. For each subtask in the image rendering task, the first computing node selects the computing node to which the subtask is affinity from the multiple types of computing nodes included in the computing system according to the mapping relationship.

[0117] It should be noted that after the first computing node assigns the corresponding computing node to each subtask, the first computing node can detect the affinity between the subtask and the corresponding computing node during the execution of the corresponding subtask by each computing node in the computing system, and thus update the mapping relationship.

[0118] Option 2: The first computing node determines the corresponding computing node for each subtask according to the load information of the multiple computing nodes in the computing system.

[0119] In the embodiments of the present application, the first computing node can obtain the current load information of each computing node in the computing system, and assign the corresponding computing node to each subtask based on the current load information of each computing node.

[0120] Optionally, the first computing node can obtain the load information of each computing node by a load sensing unit.

[0121] As shown in the load sensing principle diagram in FIG. 6, the first computing node can call the load sensing unit to obtain the tasks currently executed by each computing node from the task manager in the device manager and the number of threads currently executed by each computing node from the thread manager; for each computing node, the load sensing unit estimates the computing power load of the computing node and performs performance analysis on the computing node to obtain the load information of the computing node.

[0122] For example, when assigning the corresponding computing node to each subtask, the first computing node can select a computing node with smaller load from the multiple computing nodes in the computing system to execute the corresponding subtask.

[0123] Alternatively, the first computing node can assign the corresponding computing node to each subtask according to the current load information of each computing node in the computing system and the computing power required by the subtask.

[0124] For example, when assigning the corresponding computing node to each subtask, the first computing node can select a computing node matching the subtask from the computing system for the subtask according to the computing power required by the subtask.

[0125] Scheme three: the first computing node determines the computing node corresponding to each subtask based on the affinity of the plurality of subtasks and the computing nodes, and the load information of the plurality of computing nodes.

[0126] In this scheme, the first computing node determines the computing node corresponding to each subtask in combination with the affinity of the subtask and the computing node, and the load information of the computing node.

[0127] For scheme three, the embodiments of the present application provide a plurality of different implementation manners.

[0128] Implementation manner 1:

[0129] The first computing node determines a type of computing node corresponding to each subtask based on the affinity of the plurality of subtasks and the computing nodes; for each subtask, if the type of computing node corresponding to the subtask includes a plurality of computing nodes, the computing node corresponding to the subtask is determined according to the load information of the plurality of computing nodes.

[0130] In implementation manner 1, two steps can be specifically included:

[0131] Step 1a, the first computing node first determines a type of computing node corresponding to each subtask according to the affinity of the plurality of subtasks and the computing nodes.

[0132] For each subtask in the image rendering task, the first computing node determines the affinity of the subtask and each type of computing node in the computing system, and selects an affinity type of computing node for the subtask from the computing system. For example, a type of computing node can be a computing node including a GPU, or a computing node including an NPU, or a computing node including a CPU, or a computing node including a DPU; or a type of computing node can be at least one GPU, or at least one NPU, or at least one CPU, or at least one DPU.

[0133] It should be noted that in this step 1, the manner in which the first computing node determines the affinity of the subtask and each type of computing node can refer to the affinity determination manner introduced in scheme one above, which will not be repeated here.

[0134] Step 1b, for each subtask, when the type of computing node corresponding to the subtask includes a plurality of same type of computing nodes, the first computing node determines the computing node corresponding to the subtask according to the load information of the plurality of computing nodes.

[0135] After the first computing node determines a type of computing node corresponding to each subtask in the image rendering task based on step 1, the following operations are performed for each subtask respectively:

[0136] The first computing node obtains load information of each computing node in a type of computing nodes corresponding to the subtask, and selects the computing node corresponding to the subtask from a plurality of computing nodes included in the type of computing nodes according to the load information of each computing node in the type of computing nodes.

[0137] For example, the first computing node determines that the type of computing nodes for the subtask A is a plurality of GPUs, and then selects a GPU with lower load from the plurality of GPUs as the GPU corresponding to the subtask A according to the load information of each GPU, that is, executes the subtask A on the selected GPU.

[0138] It should be noted that the manner in which the first computing node obtains the load information of each computing node in the type of computing nodes corresponding to each subtask can be the load information obtaining manner described in the above scheme 2, which will not be repeated here.

[0139] Implementation 2:

[0140] The first computing node determines a plurality of candidate computing nodes corresponding to each subtask based on the load information of the plurality of computing nodes, and the plurality of candidate computing nodes are computing nodes of different types; for each subtask, determines the computing node corresponding to the subtask from the plurality of candidate computing nodes based on the affinity between the subtask and each candidate computing node.

[0141] In the implementation 1, two steps can be specifically included:

[0142] Step 2a, the first computing node determines a plurality of candidate computing nodes corresponding to each subtask based on the load information of the plurality of computing nodes.

[0143] Among them, the plurality of candidate computing nodes corresponding to each subtask are computing nodes of different types.

[0144] The first computing node obtains current load information of each computing node in the computing system. For each type of computing node, the first computing node can select a computing node with smaller load to form a plurality of candidate computing nodes. For example, the computing system includes computing nodes of CPU type, computing nodes of GPU type, and computing nodes of NPU type. For example, the computing nodes of CPU type include CPU 0 and CPU 1, the computing nodes of GPU type include GPU 0, GPU 1, and GPU 2, and the computing nodes of NPU type include NPU 0, NPU 1, and NPU 2. The first computing node selects CPU 0 with smaller load from CPU 0 and CPU 1, selects GPU 1 with smaller load from GPU 0, GPU 1, and GPU 2, and selects NPU 1 with smaller load from NPU 0, NPU 1, and NPU 2 according to the load information of each computing node. Therefore, the plurality of candidate computing nodes determined by the first computing node include CPU 0, GPU 1, and NPU 1.

[0145] It should be noted that the first computing node can obtain the load information of each computing node in the manner described in the load information obtaining manner in the second solution, which will not be repeated here.

[0146] In step 2b, for each subtask, the affinity between the subtask and each candidate computing node is determined, and the computing node corresponding to the subtask is determined from the plurality of candidate computing nodes.

[0147] For each subtask in the image rendering task, the first computing node determines the affinity between the subtask and each candidate computing node in the plurality of candidate computing nodes, and selects the computing node corresponding to the subtask from the plurality of candidate computing nodes according to the affinity.

[0148] For example, the computing system includes computing nodes of CPU type, computing nodes of GPU type, and computing nodes of NPU type. For example, the computing nodes of CPU type include CPU 0 and CPU 1, the computing nodes of GPU type include GPU 0, GPU 1, and GPU 2, and the computing nodes of NPU type include NPU 0, NPU 1, and NPU 2. The first computing node determines the plurality of candidate computing nodes to include CPU 0, GPU 1, and NPU 1 according to the load information of each computing node. If the subtask is super resolution, the super resolution is affinity with NPU 1 in CPU 0, GPU 1, and NPU 1, and the super resolution corresponds to NPU 1. If the subtask is raster rendering, the raster rendering is affinity with GPU 1 in CPU 0, GPU 1, and NPU 1, and the raster rendering corresponds to GPU 1.

[0149] The first computing node in the embodiment of the present applicationapplicationemploy any of the above implementation manners to determine the computing node corresponding to each subtask in the image rendering task.

[0150] In step 503, each computing node in the computing system executes the corresponding subtask.

[0151] In the embodiment of the present application, after determining the computing node corresponding to each subtask in the image rendering task, the first computing node schedules the computing node corresponding to each subtask to execute the subtask corresponding to the computing node.

[0152] It should be understood that when a subtask in the plurality of subtasks corresponds to the first computing node, the first computing node executes the corresponding subtask.

[0153] Optionally, the plurality of subtasks in the image rendering task includes a first type of subtask and a second type of subtask; wherein the first type of subtask includes at least one subtask for implementing raster rendering, and the second type of subtask includes at least one subtask for implementing neural rendering.

[0154] In the embodiment of the present application, when the computing node is a computing node including a processor, the first type of subtask corresponds to a computing node including a GPU, and at least one subtask in the second type of subtask corresponds to a computing node including an NPU. For example, the first type of subtask includes raster rendering, and the GPU computing node is used to execute the raster rendering; if the subtasks for implementing neural rendering include a plurality of subtasks, such as super resolution, optical flow, frame interpolation, color gamut conversion, and noise reduction, then the NPU computing node is used to execute the super resolution, optical flow, frame interpolation, color gamut conversion, and noise reduction, or the NPU computing node is used to execute the super resolution, optical flow, and frame interpolation, and the GPU computing node is used to execute the color gamut conversion and noise reduction.

[0155] When the computing node is a processor, the first type of subtask corresponds to a GPU, and at least one subtask in the second type of subtask corresponds to an NPU. For example, the first type of subtask includes raster rendering, and the GPU is used to execute the raster rendering; if the subtasks for implementing neural rendering include a plurality of subtasks, such as super resolution, optical flow, frame interpolation, color gamut conversion, and noise reduction, then the NPU is used to execute the super resolution, optical flow, frame interpolation, color gamut conversion, and noise reduction, or the NPU is used to execute the super resolution, optical flow, and frame interpolation, and the GPU is used to execute the color gamut conversion and noise reduction.

[0156] In the process of executing the corresponding subtask by each computing node in the computing system, the first computing nodeapplicationfurther detect the affinity between the first function in each subtask and the corresponding computing node. According to the detection result of the affinity, the first computing nodeapplicationjudge whether to adjust the computing node corresponding to the first function.

[0157] Optionally, the image rendering task comprises at least one subtask, and the at least one subtask comprises a first function, and the first computing node determines a computing node corresponding to the first function based on affinity between the first function and the computing node.

[0158] In an implementation, for each first function in each subtask, the first computing node detects affinity between the first function and a computing node performing the subtask, and if the detection result indicates that the first function is not affinity with the corresponding computing node, the first computing node adjusts the computing node corresponding to the first function. Based on this manner, the first computing node can dynamically adjust the computing node corresponding to the first function in the subtask, thereby improving the data processing efficiency of the computing system.

[0159] After each computing node in the computing system performs a corresponding subtask, the computing system generates image data after rendering; the computing system can send the image data after rendering to the user terminal.

[0160] Optionally, after the computing system generates the image data after rendering, the computing system can encode the image data after rendering and send the encoded image data after rendering to the user terminal.

[0161] Correspondingly, after the user terminal receives the image data after rendering sent by the computing system, the user terminal decodes the image data after rendering; and displays the decoded image data after rendering, so that the user terminal displays the image after rendering in the display interface.

[0162] In the following, an image rendering scheme of the embodiments of the present application will be introduced with an example.

[0163] As shown in FIG. 7, taking a computing node as an example, the computing system includes at least one CPU, at least one GPU, and at least one NPU, and a cloud container, and a pass-through link is included between the GPU and the NPU. Among them, the first computing node is a first CPU in the at least one CPU. The user terminal sends a control instruction to the cloud container in response to a user triggered operation; the cloud container decodes the control instruction to obtain image rendering control parameters corresponding to the control instruction. The cloud container sends the image rendering control parameters corresponding to the control instruction to the first CPU, and correspondingly, the first CPU receives the image rendering control parameters. The first CPU determines an image rendering task including a plurality of sub-tasks according to the image rendering control parameters. The first CPU determines the processor corresponding to each sub-task based on the affinity between the plurality of sub-tasks and each CPU, each GPU, and each NPU, and / or based on the load information of each CPU, each GPU, and each NPU, that is, arranges each sub-task in the image rendering task and allocates a processor to each sub-task. Among them, the first CPU calls the load sensing unit to sense the load information of each CPU, each GPU, and each NPU. For example, the image rendering task includes raster rendering, super resolution, color gamut conversion, noise reduction, optical flow, frame interpolation, and encoding, the raster rendering corresponds to the GPU, the optical flow, the super resolution, the frame interpolation, the noise reduction, the color gamut conversion, and the encoding correspond to the NPU, and other computing power in the image rendering task can be executed by the CPU.

[0164] Each CPU, each GPU, and each NPU in the computing system respectively executes the corresponding sub-task, and in the process of executing the corresponding sub-task by each CPU, each GPU, and each NPU, the first CPU can detect the affinity between the first function in the sub-task and the corresponding computing node. If the detection result indicates that the first function is not affinity with the corresponding processor, the first CPU adjusts the processor corresponding to the first function.

[0165] After each CPU, each GPU, and each NPU in the computing system execute the corresponding sub-task, the rendered image data is obtained. The rendered image data can be sent to the user terminal by the cloud container; the user terminal displays the rendered image on the display interface after receiving the rendered image data.

[0166] Taking a cloud server in which a computing system is deployed as an example, as shown in FIG. 8, the hardware structure of the cloud server can include a CPU, a GPU, and an NPU. After a user terminal sends a control instruction, the cloud server executes an image rendering process through the CPU, the GPU, and the NPU. The GPU and the NPU include a pass-through link (such as a P2P channel). The base image and Mesa in the cloud container are public components, providing a basic running environment for the cloud container. The screenshot engine in the cloud container is used to execute a raster rendering process. The AI engine is used to execute a neural rendering process in the image rendering process. The host kernel of the cloud server includes a P2P management module, which is used to interface with the P2P management module in the cloud container to achieve direct transmission of GPU and NPU data. The NPU driver in the host kernel is used to drive the AI engine to run on the NPU to execute the neural rendering process.

[0167] Based on the same technical concept as the method embodiments described above, the embodiments of the present application also provide a computing device for executing the method performed by the first computing node in the method embodiments shown in FIG. 5. As shown in FIG. 9, the computing device 900 includes an information transmission module 901, a computing power allocation module 902, and a scheduling module 903.

[0168] The information transmission module 901 can be configured to receive a control instruction sent by a user terminal, determine an image rendering task corresponding to the control instruction, and determine a plurality of subtasks in the image rendering task.

[0169] The computing power allocation module 902 can be configured to determine a computing node corresponding to each subtask based on the affinity of the plurality of subtasks and the computing nodes. The plurality of subtasks correspond to at least two types of computing nodes, and different types of computing nodes include different types of processors.

[0170] The scheduling module 903 can be configured to schedule the computing node corresponding to each subtask to execute the corresponding subtask.

[0171] Optionally, the information transmission module 901 can provide the function of information transmission between the computing device 900 and an external device, including receiving information input by the external device or outputting information to the external device. For example, the external device can be a user terminal, and the information transmission module 901 can be configured to receive a control instruction input by the user terminal. Alternatively, the information transmission module 901 can include an information input module and an information output module. The information input module is configured to receive information input by the external device, and the information output module is configured to output information to the external device. For example, the external device can be a user terminal, and the information input module can be configured to receive a control instruction input by the user terminal.

[0172] It should be noted that in other embodiments, the information transmission module 901 can be configured to perform any of the steps of the image rendering method, the computing power allocation module 902 can be configured to perform any of the steps of the image rendering method, and the scheduling module 903 can be configured to perform any of the steps of the image rendering method. The steps to be implemented by the information transmission module 901, the computing power allocation module 902, and the scheduling module 903 can be specified as needed, and the entire function of the computing device can be implemented by the information transmission module 901, the computing power allocation module 902, and the scheduling module 903 respectively implementing different steps in the image rendering method.

[0173] For example, in the above description, the information transmission module 901 can be configured to receive a control instruction sent by a user terminal and determine an image rendering task corresponding to the control instruction. In other embodiments, the information transmission module 901 can be configured to receive a control instruction sent by a user terminal, and the computing power allocation module 902 is configured to determine an image rendering task corresponding to the control instruction.

[0174] As a possible implementation, the image rendering task includes at least one subtask including a first function, and the computing power allocation module 902 is further configured to determine a computing node corresponding to the first function based on the affinity between the first function and the computing node.

[0175] As a possible implementation, the computing power allocation module 902 is further configured to determine a computing node corresponding to each subtask based on the load information of the plurality of computing nodes.

[0176] As a possible implementation, the computing power allocation module 902 is further configured to determine a type of computing node corresponding to each subtask based on the affinity between the plurality of subtasks and the computing nodes, and for each subtask, if the type of computing node corresponding to the subtask includes a plurality of computing nodes, determine a computing node corresponding to the subtask according to the load information of the plurality of computing nodes.

[0177] As a possible implementation, the computing power allocation module 902 is further configured to determine a plurality of candidate computing nodes corresponding to each subtask based on the load information of the plurality of computing nodes, and the plurality of candidate computing nodes are computing nodes of different types; for each subtask, determine a computing node corresponding to the subtask from the plurality of candidate computing nodes based on the affinity between the subtask and each candidate computing node.

[0178] As a possible implementation, the information transmission module 901 is configured to decode the received control instruction to obtain rendering control parameters of the image rendering task, and determine the image rendering task including a plurality of subtasks according to the rendering control parameters.

[0179] Optionally, a direct link is included between the two different types of computing nodes, and the direct link is used to transmit data between the two different types of computing nodes.

[0180] Optionally, the computing node comprises at least one of a CPU, a GPU, an NPU, a DPU, and an FPGA.

[0181] Optionally, the plurality of sub-tasks comprise first-type sub-tasks and second-type sub-tasks, the first-type sub-tasks comprise at least one sub-task for implementing raster rendering, and the second-type sub-tasks comprise at least one sub-task for implementing neural rendering; the first-type sub-tasks correspond to the GPU, and at least one sub-task in the second-type sub-tasks corresponds to the NPU.

[0182] As a possible implementation, as shown in FIG. 10, the computing device 900 of the embodiment of the present application can further comprise a raster rendering module 904, a neural rendering module 905, and an encoding module 906.

[0183] The raster rendering module 904 is configured to perform raster rendering based on the raster rendering; the neural rendering module 905 is configured to perform neural rendering on the image based on a deep neural network model, for example, to perform enhancement processing on the image based on the deep neural network model, so as to improve the image quality; and the encoding module 906 is configured to encode the rendered image to obtain encoded rendered image data.

[0184] As a possible implementation, the information transmission module 901 is further configured to send the encoded rendered image data to the user terminal.

[0185] It should be understood that the code for executing the functions of the raster rendering module 904, the neural rendering module 905, and the encoding module 906 of the embodiment of the present application can run on the computing nodes corresponding to the various sub-tasks introduced above to realize the functions of the raster rendering module 904, the neural rendering module 905, and the encoding module 906. For example, the code for realizing the function of the raster rendering module 904 can run on the GPU, and the code for realizing the functions of the neural rendering module 905 and the encoding module 906 can run on the NPU.

[0186] It should be noted that in other embodiments, the raster rendering module 904 can be configured to perform any step in the image rendering method, the neural rendering module 905 can be configured to perform any step in the image rendering method, and the encoding module 906 can be configured to perform any step in the image rendering method. The steps to be realized by the raster rendering module 904, the neural rendering module 905, and the encoding module 906 can be specified as needed, and the entire function of the computing device can be realized by the raster rendering module 904, the neural rendering module 905, and the encoding module 906 respectively realizing different steps in the image rendering method.

[0187] The division of the modules in the embodiments of the present application is illustrative, and is only a logical function division. In actual implementation, another division manner can be used. In addition, each function module in each embodiment of the present application can be integrated in one processor, or can be a separate physical existence, or two or more modules can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software function module.

[0188] When the integrated module is realized in the form of a software function module and sold or used as an independent product, the integrated module can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing an end device (which can be a personal computer, a mobile phone, or a network device, etc.) or a processor to perform all or part of the steps of the method according to the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0189] The embodiments of the present application also provide a computing device 1100 as shown in FIG. 11. The computing device 1100 includes a bus 1101, at least one processor 1102, and a memory 1103. The processor 1102 and the memory 1103 communicate through the bus 1101.

[0190] When the computing system in the embodiments of the present application is a single computing device, the computing device 1100 can be a computing device. When the computing system in the embodiments of the present application is a computing device cluster, the computing device 1100 can be a computing device in the computing device cluster. For example, the computing device can be a server (including but not limited to a local server or a cloud server).

[0191] Optionally, the computing device 1100 can also include a network card 1104.

[0192] The processor 1102 included in the computing device 1100 according to the embodiments of the present application can be a CPU, a GPU, an NPU, a DPU, an FPGA, or the like. The processor 1102 can also be another general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a discrete gate or transistor logic, a discrete hardware component, or the like.

[0193] The memory 1103 can be a dynamic random access memory (DRAM). In addition to the DRAM, the memory 1103 can also be another random access memory, such as a static random access memory (SRAM), or the like. In addition, the memory 1103 can also be a read only memory (ROM). For the read only memory, for example, it can be a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), or the like. The memory 1103 can also be a FLASH memory medium, a hard disk drive (HDD), a solid state disk (SSD), or the like.

[0194] The memory 1103 stores computer program instructions, and the processor 1102 invokes the computer program instructions to perform the steps performed by the computing node in the method described in the foregoing FIG. 5. The memory 1103 can also include software modules required by other running processes, such as the modules in the computing device 900 shown in FIG. 9 or FIG. 10.

[0195] The embodiments of the present application also provide a communication system including a user terminal and a computing system. The functions of the user terminal and the computing system can be referred to the foregoing description.

[0196] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computer device, the at least one computer device performs the method performed by the computing node in the foregoing embodiment of FIG. 5, which is described in the description of each step of FIG. 5 and will not be described here.

[0197] The computer readable storage medium can be any available media or data storage device that can be accessed by the computing node and includes one or more of volatile memory, nonvolatile memory, or a combination of both. The computer readable storage medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium, (e.g., a compact disk, a DVD), or a semiconductor medium, (e.g., a solid state disk), etc. The computer readable storage medium includes instructions for the computing node to perform the methods described above in the embodiment of FIG. 5, as described in connection with the steps of FIG. 5, which are not repeated here.

[0198] In the embodiments described above, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium accessible by a computer or a data storage device including one or more available media integrated into a server, data center, etc. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0199] The various illustrative logical blocks, modules, circuits, and algorithms described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general purpose processor can be a microprocessor, optionally the general purpose processor can also be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other similar configuration.

[0200] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is tangible. In addition, the storage medium can be connected to the processor, so that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC.

[0201] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks or in the block or blocks of the block diagrams.

[0202] Although the present application has been described in connection with specific embodiments thereof, it will be evident for those skilled in the art that various modifications and combinations are possible without departing from the spirit and scope of the application. Accordingly, the description and drawings are to be regarded as illustrative in nature and not as restrictive. The scope of the application is indicated by the appended claims, and all changes which come within the meaning and range of equivalents are intended to be embraced therein. It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. An image rendering method, characterized by, The method comprises: receiving a control instruction sent by a user terminal, determining an image rendering task corresponding to the control instruction, the image rendering task comprising a plurality of subtasks; determining a computing node corresponding to each of the subtasks based on the affinity between the plurality of subtasks and the computing nodes; wherein the plurality of subtasks correspond to at least two types of computing nodes, and different types of computing nodes comprise different types of processors; scheduling the computing node corresponding to each of the subtasks to execute the corresponding subtask.

2. The method of claim 1, wherein, The image rendering task comprises at least one subtask comprising a first function, and the method further comprises: determining a computing node corresponding to the first function based on the affinity between the first function and the computing nodes.

3. The method of claim 1 or 2, wherein, The method further comprises: determining a computing node corresponding to each of the subtasks based on the load information of the plurality of computing nodes.

4. The method of claim 1 or 2, wherein, The method further comprises: determining a type of computing node corresponding to each of the subtasks based on the affinity between the plurality of subtasks and the computing nodes; for each of the subtasks, if the type of computing node corresponding to the subtask comprises a plurality of computing nodes, determining the computing node corresponding to the subtask according to the load information of the plurality of computing nodes.

5. The method of claim 1 or 2, wherein, The method further comprises: determining a plurality of candidate computing nodes corresponding to each of the subtasks based on the load information of the plurality of computing nodes, the plurality of candidate computing nodes being different types of computing nodes; for each of the subtasks, determining the computing node corresponding to the subtask from the plurality of candidate computing nodes based on the affinity between the subtask and each of the candidate computing nodes.

6. The method of any one of claims 1-5, wherein, The two different types of computing nodes comprise a direct link therebetween, and the direct link is used to transmit data between the two different types of computing nodes.

7. The method of any of claims 1-6, wherein, The computing node comprises at least one of a CPU, a GPU, an NPU, a DPU, and an FPGA.

8. The method of claim 7, wherein, The plurality of subtasks comprise a first type of subtask and a second type of subtask, the first type of subtask comprising at least one subtask for implementing raster rendering, and the second type of subtask comprising at least one subtask for implementing neural rendering; the first type of subtask corresponds to the GPU, and at least one subtask of the second type of subtask corresponds to the NPU.

9. A computing system, comprising: The system comprises a plurality of computing nodes; a first computing node in the plurality of computing nodes is configured to determine an image rendering task corresponding to a control instruction sent by a user terminal, the image rendering task comprising a plurality of subtasks; determine a computing node corresponding to each of the subtasks based on the affinity between the plurality of subtasks and the computing nodes; wherein the plurality of subtasks correspond to at least two types of computing nodes, and different types of computing nodes comprise different types of processors; the computing node is configured to execute the corresponding subtask.

10. The system of claim 9, wherein, The image rendering task comprises at least one subtask comprising a first function, and the first computing node is further configured to: determine a computing node corresponding to the first function based on the affinity between the first function and the computing nodes.

11. The system of claim 9 or 10, wherein, The first computing node is further configured to: determine a computing node corresponding to each of the subtasks based on the load information of the plurality of computing nodes.

12. The system of claim 9 or 10, wherein, The first computing node is further configured to: determine a type of computing node corresponding to each of the sub-tasks based on the affinity between the sub-tasks and the computing nodes; for each of the sub-tasks, if the type of computing node corresponding to the sub-task includes multiple computing nodes, determine the computing node corresponding to the sub-task according to load information of the multiple computing nodes.

13. The system of claim 9 or 10, wherein, The first computing node is further configured to: determine multiple candidate computing nodes corresponding to each of the sub-tasks based on load information of the multiple computing nodes, the multiple candidate computing nodes being different types of computing nodes; for each of the sub-tasks, determine the computing node corresponding to the sub-task from the multiple candidate computing nodes based on the affinity between the sub-task and each of the candidate computing nodes.

14. The system of any of claims 11-13, wherein, A direct link is included between two different types of computing nodes, and the direct link is used to transmit data between the two different types of computing nodes.

15. The system of any of claims 11-14, wherein, The computing node includes at least one of a CPU, a GPU, an NPU, a DPU, and an FPGA.

16. The system of claim 15, wherein, The multiple sub-tasks include a first type of sub-task and a second type of sub-task, the first type of sub-task including at least one sub-task for implementing raster rendering, and the second type of sub-task including at least one sub-task for implementing neural rendering; the first type of sub-task corresponds to the GPU, and at least one sub-task of the second type of sub-task corresponds to the NPU.

17. A computing device, comprising: The apparatus includes: an information transmission module configured to receive a control instruction sent by a user terminal, determine an image rendering task corresponding to the control instruction, and determine that the image rendering task includes multiple sub-tasks; a computing power allocation module configured to determine a computing node corresponding to each of the sub-tasks based on the affinity between the multiple sub-tasks and the computing nodes; wherein the multiple sub-tasks correspond to at least two types of computing nodes, and different types of computing nodes include different types of processors; a scheduling module configured to schedule the computing node corresponding to each of the sub-tasks to execute the corresponding sub-task.

18. A computing device, comprising: The apparatus includes at least one processor and at least one memory; wherein the one or more memories store one or more computer programs, and the one or more computer programs include instructions that, when executed by the one or more processors, cause the computing apparatus to perform the method of any one of claims 1-8.

19. A communication system, characterized by The system includes a user terminal and the computing system of any one of claims 9-16.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when executed on an electronic device, causes the electronic device to perform the method of any one of claims 1-8.

Images