Image rendering method, electronic device, storage medium, and computer program product

Abstract

Embodiments of the present application provide an image rendering method, an electronic device, a storage medium and a product, and belong to the field of image processing. The method comprises: a second processor processing texture data of a pixel point in a first texture map of a first resolution, and writing a plurality of first texture data units obtained into a first shared memory; a third processor generating a second texture map of the first resolution based on the plurality of first texture data units in the first shared memory; performing super-resolution rendering on the second texture map to obtain a third texture map of a second resolution, and writing texture data of a pixel point in the third texture map into a second shared memory to obtain a plurality of second texture data units, the second resolution being higher than the first resolution; the second processor processing the plurality of second texture data units in the second shared memory to obtain a fourth texture map of the second resolution; and generating an image of the second resolution based on the fourth texture map, thereby improving the image display effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image rendering method, an electronic device, a storage medium and a computer program product. BACKGROUND

[0002] With the rise of large 3D (3 Dimensions) mobile games, users have increasingly high requirements for the quality of 3D mobile games. In order to meet the quality requirements of users, the GPU (Graphics Processing Unit, image processor) of the electronic device needs to render high-resolution images. However, the GPU consumes a lot of power when rendering high-resolution images, which in turn causes heating problems. In order to solve the heating problem caused by high power consumption of the GPU, how to render images has become a problem that technicians in the field are concerned about.

[0003] The related technology uses a CPU (Central Processing Unit, central processing unit) + GPU + NPU (Neural-network Processing Unit, neural network processor) architecture to render images. Specifically, it includes: the CPU applies for a HardwareBuffer, creates an EGLImage according to the HardwareBuffer, transmits the virtual address of the HardwareBuffer into the EGLImage as a data carrier, and binds the EGLImage with texture data; when rendering a first texture map of any image at a first resolution, the GPU writes the texture data of the pixel points in the first texture map into the HardwareBuffer through the EGLImage; the NPU reads the texture data of the pixel points from the HardwareBuffer, writes the read texture data of the pixel points into a texture matrix corresponding to the first resolution, obtains a second texture map at the first resolution, and calls an AI super-resolution model to perform super-resolution rendering on the second texture map to obtain a third texture map at a second resolution, and then writes the texture data of the pixel points in the third texture map into the HardwareBuffer, wherein the second resolution is higher than the first resolution; the GPU reads the texture data of the pixel points from the HardwareBuffer through the EGLImage, and writes the read texture data of the pixel points into a texture matrix corresponding to the second resolution, obtains a fourth texture map at the second resolution, and then binds the fourth texture map back to the original rendering process to obtain the image at the second resolution.

[0004] Considering the memory alignment problem, the actual memory space applied by the HardwareBuffer is different from the set memory space size, resulting in that the number of pixel point texture data actually stored by the HardwareBuffer in each row is different from the set number. When the pixel point texture data in the first texture map is written into the HardwareBuffer, at least one column of blank pixel points exists in each row of the HardwareBuffer. The NPU reads the pixel point texture data in the HardwareBuffer row by row and writes the pixel point texture data into the texture matrix corresponding to the first resolution column by column. Due to the existence of the blank pixel points, the pixel points in the second texture map are disordered, and the pixel points in the third texture map obtained by performing super-resolution rendering on the second texture map are also disordered. Since the pixel points in the third texture map are already disordered, at least one column of blank pixel points also exists in each row of the HardwareBuffer when the HardwareBuffer stores the pixel point texture data in the third texture map, further increasing the disorder degree of the pixel points in the fourth texture map. Therefore, the original image content cannot be rendered based on the fourth texture map, and the image display effect is poor, and there are problems such as screen flashing. SUMMARY

[0005] Embodiments of the present application provide an image rendering method, an electronic device, a storage medium and a product, which can avoid disorder of pixel points in a texture map and improve image display effect.

[0006] In a first aspect, an image rendering method is provided. The method is applied to an electronic device, and the electronic device includes a first processor, a second processor and a third processor. The first processor is configured to create a first shared memory and a second shared memory. Data in the first shared memory and the second shared memory is stored in a row. The method includes the following steps.

[0007] When a first texture map at a first resolution of a frame image is rendered, the second processor processes texture data of pixel points in the first texture map to obtain a plurality of first texture data units, and sequentially writes the plurality of first texture data units into the first shared memory.

[0008] The third processor generates a second texture map at the first resolution based on the plurality of first texture data units stored in the first shared memory. The pixel points at each position in the first texture map and the second texture map are the same. The third texture map at a second resolution is obtained by performing super-resolution rendering on the second texture map. Texture data of pixel points in the third texture map is sequentially written into the second shared memory to obtain a plurality of second texture data units. The second resolution is higher than the first resolution.

[0009] The second processor processes the plurality of second texture data units stored in the second shared memory to obtain a fourth texture map of a second resolution, and the third texture map and the fourth texture map have the same pixel at each position; and the second processor generates the image of the second resolution based on the fourth texture map.

[0010] In a first possible implementation manner of the first aspect, before the second processor processes the texture data of each pixel in the first texture map to obtain the plurality of first texture data units, the method further includes:

[0011] The second processor creates a compute shader, and creates a first shader storage buffer object (SSBO) and a second SSBO in the compute shader, the first SSBO provides a first binding point, and the second SSBO provides a second binding point;

[0012] The second processor establishes a first binding relationship between the first SSBO, the compute shader, and the first shared memory based on the first binding point;

[0013] The second processor establishes a second binding relationship between the second SSBO, the compute shader, and the second shared memory based on the second binding point.

[0014] In a second possible implementation manner of the first aspect, the texture data of the pixel in the first texture map is first texture data, the first texture data unit includes all or part of second texture data of two adjacent pixels in the first texture map, the data format of the first texture data is different from that of the second texture data, and the second processor processes the texture data of the pixel in the first texture map to obtain the plurality of first texture data units, including:

[0015] The compute shader calculates a right shift number of the first texture data of the first pixel according to the pixel identifier of the first pixel, and the first pixel is any pixel in the first texture map;

[0016] The compute shader calculates a left shift number of the first texture data of a second pixel according to the right shift number of the first texture data of the first pixel, and the second pixel is the next pixel nearest to the first pixel in the first texture map;

[0017] The compute shader generates a first texture data unit of a preset type according to the first texture data of the first pixel and the right shift number thereof, and the first texture data of the second pixel and the left shift number thereof.

[0018] In a third possible implementation of the first aspect, the calculation of the left shift number of the first texture data of the second pixel point by the calculation shader according to the right shift number of the first texture data of the first pixel point comprises:

[0019] The calculation shader obtains a first remainder value by calculating a ratio of a product of a pixel identifier of the first pixel point and a first value and a second value, the first value being a number of channels included in the first texture data, and the second value being a number of channels included in the second texture data;

[0020] The calculation shader calculates a product of the first remainder value and a third value to obtain a first product value, the third value being a number of bits occupied by each channel when the first texture data is represented in the preset type;

[0021] The calculation shader takes the first product value as the right shift number of the first texture data of the first pixel point.

[0022] In a fourth possible implementation of the first aspect, the calculation of the left shift number of the first texture data of the second pixel point by the calculation shader according to the right shift number of the first texture data of the first pixel point comprises:

[0023] The calculation shader calculates a difference between a fourth value and the right shift number of the first texture data of the first pixel point to obtain a first difference value, the fourth value being a maximum number of bits by which the first texture data of the second pixel point can be left shifted;

[0024] The calculation shader takes the first difference value as the left shift number of the first texture data of the second pixel point.

[0025] In a fifth possible implementation of the first aspect, the first texture data is a value of an RGBA channel, the second texture data is a value of an RGB channel, and the preset type is a uint type, and the calculation of the first texture data unit in the preset type by the calculation shader according to the first texture data of the first pixel point and the right shift number thereof, the first texture data of the second pixel point and the left shift number thereof comprises:

[0026] The calculation shader modifies a value of an A channel in the RGBA channel of the first pixel point and the RGBA channel of the second pixel point to 0;

[0027] The calculation shader converts the modified value of the RGBA channel of the first pixel point to a first uint value and converts the modified value of the RGBA channel of the second pixel point to a second uint value;

[0028] The computing shader right shifts the first uint value according to a right shift number of an RGBA channel of the first pixel point to obtain a third uint value;

[0029] The computing shader left shifts the second uint value according to a left shift number of an RGBA channel of the second pixel point to obtain a fourth uint value.

[0030] The computing shader adds the third uint value and the fourth uint value of the same number of bits to obtain a first texture data unit of the uint type.

[0031] In a sixth possible implementation of the first aspect, the second processor sequentially writes the plurality of first texture data units into the first shared memory, including:

[0032] Based on the first binding relationship, the computing shader sequentially writes the plurality of first texture data units into the first SSBO and sequentially writes the plurality of first texture data units into the first shared memory through the first SSBO.

[0033] In a seventh possible implementation of the first aspect, the third processor generates a second texture map of the first resolution based on the plurality of first texture data units stored in the first shared memory, including:

[0034] The third processor reads texture data of a pixel point from the plurality of first texture data units stored in the first shared memory and generates the second texture map based on a number of rows and a number of columns of pixel points indicated by the first resolution.

[0035] In an eighth possible implementation of the first aspect, the third processor performs super-resolution rendering on the second texture map to obtain a third texture map of a second resolution, including:

[0036] The third processor calls an AI (Artificial Intelligence) super-resolution model to perform super-resolution rendering on the second texture map to obtain the third texture map.

[0037] In a ninth possible implementation of the first aspect, the second texture data unit is of a preset type, the second texture data unit includes all or part of second texture data of two adjacent pixel points in the third texture map, texture data of a pixel point in the fourth texture map is first texture data, a data format of the first texture data is different from a data format of the second texture data, and the second processor processes the plurality of second texture data units stored in the second shared memory to obtain a fourth texture map of a second resolution, including:

[0038] The compute shader caches, based on the second binding relationship, a plurality of second texture data units stored in the second shared memory into the second SSBO, and processes the plurality of second texture data units in the second SSBO to obtain first texture data of each pixel point at a second resolution;

[0039] The fourth texture map is generated based on the first texture data of each pixel point at the second resolution.

[0040] In a tenth possible implementation of the first aspect, the processing, by the compute shader, the plurality of second texture data units in the second SSBO to obtain first texture data of each pixel point at a second resolution comprises:

[0041] The compute shader determines, according to a pixel identifier of a third pixel point, a minimum index value of a second texture data unit for storing second texture data of the third pixel point in the second SSBO, the third pixel point being any pixel point in the plurality of second texture data units;

[0042] The compute shader calculates, according to the pixel identifier of the third pixel point, a right shift number of second texture data in the first target data unit, the first target data unit being the second texture data unit corresponding to the minimum index value;

[0043] When the right shift number of second texture data in the first target data unit is not 0, the compute shader calculates a left shift number of second texture data in a second target data unit, the second target data unit being a next second texture data unit in the second SSBO that is most adjacent to the first target data unit;

[0044] The compute shader determines, according to the second texture data in the first target data unit, the right shift number corresponding to the first target data unit, the second texture data in the second target data unit, and the left shift number corresponding to the second target data unit, the first texture data of the third pixel point.

[0045] In an eleventh possible implementation of the first aspect, the determining, by the compute shader, according to a pixel identifier of a third pixel point, a minimum index value of a second texture data unit for storing second texture data of the third pixel point in the second SSBO comprises:

[0046] The calculation shader calculates a quotient of a product of a pixel index of the third pixel point and a second value and a first value, and obtains a remainder value as the minimum index value, the first value being a number of channels included in the first texture data, and the second value being a number of channels included in the second texture data.

[0047] In a twelfth possible implementation of the first aspect, the calculation shader calculates, according to the pixel index of the third pixel point, a right shift number of the second texture data stored in the first target data unit, including:

[0048] The calculation shader obtains a second remainder value by calculating a quotient of a product of a pixel index of the third pixel point and a second value and a first value, the first value being a number of channels included in the first texture data, and the second value being a number of channels included in the second texture data;

[0049] The calculation shader calculates a product of the second remainder value and a third value to obtain a second product value, the third value being a number of bits occupied by each channel when the first texture data is represented in the preset type;

[0050] The calculation shader takes the second product value as a right shift number of the second texture data stored in the first target data unit.

[0051] In a thirteenth possible implementation of the first aspect, the calculation shader calculates a left shift number of the second texture data stored in the second target data unit, including:

[0052] The calculation shader calculates a difference between a fifth value and a right shift number corresponding to the first target data unit to obtain a second difference value, the fifth value being a maximum number of bits that the second texture data can be left shifted in the second target data unit;

[0053] The calculation shader takes the second difference value as a left shift number of the second texture data stored in the second target data unit.

[0054] In a fourteenth possible implementation of the first aspect, the first texture data is a value of an RGBA channel, the second texture data is a value of an RGB channel, and the preset type is a uint type, and the calculation shader determines the first texture data of the third pixel point according to the second texture data in the first target data unit, a right shift number corresponding to the first target data unit, the second texture data in the second target data unit, and a left shift number corresponding to the second target data unit, including:

[0055] The computing shader right shifts the second texture data in the first target data unit according to a right shift number corresponding to the first target data unit to obtain a fifth uint value;

[0056] The computing shader left shifts the second texture data in the second target data unit according to a left shift number corresponding to the second target data unit to obtain a sixth uint value;

[0057] The computing shader adds the fifth uint value and the sixth uint value to obtain a seventh uint value, and the seventh uint value includes all values of RGB channels of the third pixel point.

[0058] The computing shader converts the seventh uint value into values of RGBA channels, and modifies a value of an A channel in the RGBA channels to 1 to obtain the first texture data of the third pixel point.

[0059] In a fifteenth possible implementation manner of the first aspect, the method further includes:

[0060] When the right shift number of the second texture data in the first target data unit is 0, the computing shader takes an eighth uint value corresponding to the second texture data in the first target data unit as the seventh uint value.

[0061] In a second aspect, an electronic device is provided, including a processor and a memory, the processor including a first processor, a second processor and a third processor; the memory stores at least one program code; the at least one program code is used to be called and executed by the first processor, the second processor and the third processor to implement the image rendering method in the first aspect.

[0062] In a third aspect, a computer readable storage medium is provided, and the computer readable storage medium stores at least one computer program, and the at least one computer program is executed by the first processor, the second processor and the third processor to implement the image rendering method in the first aspect.

[0063] In a fourth aspect, a computer program product is provided, and the computer program product includes a computer program, and the computer program is executed by the first processor, the second processor and the third processor to implement the image rendering method in the first aspect.

[0064] The technical scheme provided by the embodiments of the present application has the following beneficial effects:

[0065] The first processor in the embodiment of the present application applies two shared memories, which are respectively a first shared memory and a second shared memory. Unlike the storage form of the existing shared memory, the data in the first shared memory and the second shared memory are stored in a row. When the first texture map of the first resolution is rendered, the second processor sequentially writes a plurality of first texture data units obtained by processing the texture data of the pixel points in the first texture map into the first shared memory. Since the data in the first shared memory is stored in a row, there will be no blank pixel points between the respective pixel points of the first texture map stored in the first shared memory. Even if the memory space of the first shared memory is different from the set memory space size, when the third processor reads the blank pixel points from the first shared memory, these blank pixel points will not be written into the second texture map, so as to ensure that the pixel points at each position in the second texture map are the same as those in the first texture map without being disordered. After the super-resolution rendering of the second texture map obtains a third texture map, the third processor sequentially writes a plurality of second texture data units obtained by processing the third texture map into the second shared memory. Since the data in the second shared memory is stored in a row, there will be no blank pixel points between the respective pixel points of the third texture map stored in the second shared memory. Even if the memory space of the second shared memory is different from the set memory space size, when the second processor reads the blank pixel points from the second shared memory, these blank pixel points will not be written into the fourth texture map, so as to ensure that the pixel points at each position in the fourth texture map are the same as those in the third texture map without being disordered. Since the pixel points in the fourth texture map are not disordered, the rendering of the image based on the fourth texture map can render the original image content, and since the fourth texture map is a high-resolution texture map, the image is clearer and the display effect is better. BRIEF DESCRIPTION OF DRAWINGS

[0066] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and serve to explain the present application. In the drawings:

[0067] Figure 1 A flowchart of a rendering of an AI super-resolution technology;

[0068] Figure 2 A schematic diagram of an image rendering architecture of a related technology;

[0069] Figure 3 A schematic diagram of a low-resolution image;

[0070] Figure 4 A schematic diagram of an image rendered by a related technology;

[0071] Figure 5A schematic diagram of an image rendering architecture provided by an embodiment of the present application;

[0072] Figure 6 A structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0073] Figure 7 A flowchart of an image rendering method provided by an embodiment of the present application;

[0074] Figure 8 A schematic diagram of a conversion process of converting a value of an RGBA channel of a pixel point into a value of an RGB channel provided by an embodiment of the present application;

[0075] Figure 9 A flowchart of writing a value of an RGBA channel of a pixel point into a first shared memory provided by an embodiment of the present application;

[0076] Figure 10 A schematic diagram of a conversion process of converting a value of an RGB channel of a pixel point into a value of an RGBA channel provided by an embodiment of the present application;

[0077] Figure 11 A flowchart of converting a value of an RGB channel of a pixel point into a value of an RGBA channel provided by an embodiment of the present application;

[0078] Figure 12 A schematic diagram of an image rendered by an image rendering method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in detail with the embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0080] It can be understood that the terms “each”, “multiple” and “any” used in the embodiments of the present application include two or more, each refers to each of the corresponding multiple, and any refers to any one of the corresponding multiple. For example, multiple words include 10 words, and each word refers to each of the 10 words, and any word refers to any one of the 10 words. In addition, “first”, “second” and the like in the embodiments of the present application are used to distinguish different messages, devices, modules and the like, and do not represent the order and do not limit that “first” and “second” are different types.

[0081] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0082] With the development of computer technology, users have higher and higher requirements for the image quality displayed on electronic devices, which requires the GPU of the electronic device to render high-resolution images. However, rendering high-resolution images consumes a lot of power of the GPU, which causes heat and other problems, further affecting the performance of the electronic device. To solve the problem of heat caused by high power consumption of the GPU and improve the performance of the electronic device, a CPU+GPU+NPU architecture can be used for image rendering. When using this architecture for image rendering, the CPU is responsible for generating logical control instructions, the GPU is responsible for rendering low-resolution texture maps, and the NPU is responsible for super-resolution rendering of the low-resolution texture maps. Since the NPU with higher computing power and lower energy consumption is used for super-resolution rendering, and the GPU only needs to render low-resolution texture maps, the computing power of the GPU is saved.

[0083] Figure 1 A flowchart of image rendering based on CPU+GPU+NPU is shown, see Figure 1 During image rendering, the CPU can intercept image rendering instructions through OpenGL or Vulkan, etc. so that the GPU can render low-resolution texture maps of the image. When the GPU finishes rendering the low-resolution texture maps, the NPU calls an AI super-resolution model to process the texture data of each pixel point in the low-resolution texture maps to obtain high-resolution texture maps, and then the GPU binds the high-resolution texture maps back to the original rendering process, thereby rendering high-resolution images.

[0084] Although using the above architecture for image rendering can reduce the power consumption of the GPU and improve the performance of the electronic device to some extent, since the NPU needs to perform super-resolution rendering on the low-resolution texture maps rendered by the GPU, and the NPU and the CPU cannot access the memory of the GPU, it is necessary to copy the texture data of each pixel point in the low-resolution texture maps rendered by the GPU to the memory of the CPU, so that the NPU can read the texture data of each pixel point in the low-resolution texture maps from the memory of the CPU. A large amount of texture data will be copied between the memory of the GPU and the memory of the CPU, and the copying of these data will consume a lot of resources and reduce the performance of the electronic device.

[0085] To reduce the resource consumption between the CPU and the GPU due to data copying, a related art provides an image rendering method. Referring to Figure 2 The method uses the way of EGLImage to perform image rendering. Specifically, before image rendering, the CPU creates a HardwareBuffer, obtains a virtual address of the HardwareBuffer through a lock function, and creates an EGLImage according to the HardwareBuffer, and then passes the virtual address of the HardwareBuffer into the EGLImage as a data carrier, so as to realize the binding of the EGLImage and the HardwareBuffer. At the same time, the CPU binds the EGLImage and the texture data of the pixel points in the texture map, so that the GPU can access the HardwareBuffer through the EGLImage, thereby reducing the resource consumption between the CPU and the GPU due to data copying. During image rendering, the GPU calls glBlitFrameBuffer to write the texture data of each pixel point in the first texture map of the first resolution into the HardwareBuffer through the EGLImage. The NPU reads the texture data of each pixel point in the first map from the HardwareBuffer, writes the read texture data of the pixel points into a texture matrix corresponding to the first resolution, obtains a second texture map of the first resolution, calls an AI super-resolution model to perform super-resolution rendering on the second texture map, obtains a third texture map of the second resolution, and then writes the texture data of each pixel point in the third texture map into the HardwareBuffer. The GPU reads the texture data of each pixel point in the third texture map from the HardwareBuffer through the EGLImage, writes the read texture data of the pixel points into a texture matrix corresponding to the second resolution, obtains a fourth texture map of the second resolution, and calls glBindtexture to bind the fourth texture map back to the original rendering process, so as to render an image of the second resolution.

[0086] Although the use of EGLImage can reduce the resource consumption caused by data copying between CPU and GPU, the memory space of HardwareBuffer applied according to the set memory space is different from the actual memory space, and the number of pixel point texture data that can be written in each row of HardwareBuffer is different from the set number, because the memory alignment is required when applying HardwareBuffer. Moreover, the size of EGLImage is greater than that of HardwareBuffer, because EGLImage is created based on the size of HardwareBuffer, and therefore the number of pixel point texture data in EGLImage is also different from the set number. When GPU writes the texture data of each pixel point in the first texture map to HardwareBuffer through EGLImage, there are at least one column of blank pixel points in each row of HardwareBuffer. NPU reads the texture data of each pixel point from HardwareBuffer row by row, and writes the read pixel point texture data into the texture matrix of the first resolution in sequence. When the texture data of a row of pixel points read from HardwareBuffer is written into the texture matrix, the next row of pixel point texture data is read from HardwareBuffer, and this process continues until the texture matrix is filled. Since there are at least one column of blank pixel points in each row of HardwareBuffer, these blank pixel points are also written into the texture matrix, resulting in disorder of pixel points in the generated second texture map. Since the pixel points in the second texture map are disordered, the pixel points in the third texture map obtained by AI super-resolution model performing super-resolution rendering on the second texture map are also disordered. Since the pixel points in the third texture map are already disordered, and the memory space of HardwareBuffer is different from the set memory space, there are also at least one column of blank pixel points in each row of HardwareBuffer when storing the texture data of pixel points in the third texture map. When GPU reads the texture data of pixel points in the third texture map from HardwareBuffer row by row through EGLImage, and writes the read pixel point texture data into the texture matrix of the second resolution, some blank pixel points will also be written into the texture matrix, further increasing the disorder degree of pixel points in the fourth texture map. Since the pixel points in the fourth texture map are disordered, when image rendering is performed based on the fourth texture map, the original image content cannot be rendered, and when the rendered image is displayed, there are problems such as screen flashing, and the image display effect is poor. Figure 3 a low-resolution image is shown, Figure 4 a high-resolution image obtained by rendering the image using the related art method is shown, it can be seen that Figure 4The middle image has a screen problem, and the display effect is poor.

[0087] To reduce resource consumption caused by data copying between the CPU and the GPU and improve the display effect of the rendered image, an embodiment of the present application provides an image rendering method, which is described below with reference to Figure 5 The method also uses the CPU+GPU+NPU architecture to perform image rendering.

[0088] The CPU is configured to apply two shared memories, i.e., a first shared memory and a second shared memory. The CPU can call the AHardwareBuffer_allocate function to apply the first shared memory and the second shared memory. The first shared memory and the second shared memory can be HardwareBuffer, which is a memory type provided by the Android system and can be shared between processes. The CPU and the heterogeneous computing unit (e.g., GPU, NPU, etc.) in the electronic device can access the memory, but the HardwareBuffer has certain requirements for the data format of the data written, i.e., the data format of the written data is AHARDWAREBUFFER_FORMAT_BLOB. The first shared memory is configured to store the intermediate rendering result of the GPU, i.e., the texture data of each pixel point in the first texture map of the first resolution rendered by the GPU. The second shared memory is configured to store the processing result of the NPU, i.e., the texture data of each pixel point in the third texture map of the high resolution obtained after the NPU performs super-resolution rendering. The first shared memory and the second shared memory can be applied according to the set memory space, e.g., the first shared memory is applied according to the storage space occupied by the first texture map of the first resolution, and the second shared memory is applied according to the storage space occupied by the third texture map of the second resolution. Unlike the data storage form of the HardwareBuffer applied in the related art, the data in the first shared memory and the second shared memory is stored in a row. With this data storage method, the texture data of the pixel points can be continuously written into the shared memory, and even if the memory space of the first shared memory and the second shared memory is different from the size of the applied memory space, no blank pixel points will be written into the texture map, avoiding the disorder of the pixel points in the texture map. The CPU is further configured to generate logical control instructions and send the generated logical control instructions to the GPU and the NPU to control the GPU and the NPU to perform corresponding operations.

[0089] The GPU is configured to create a compute shader and access the first shared memory and the second shared memory through the compute shader to reduce the copying of data between the CPU and the GPU. To access the first shared memory and the second shared memory, the GPU is further configured to create a first SSBO and a second SSBO in the compute shader, the first SSBO providing a first binding point and the second SSBO providing a second binding point. The compute shader of the GPU is bound to the first SSBO based on the first binding point and calls a glBufferStorageExternalEXT function to bind the first SSBO to the first shared memory, thereby establishing a first binding relationship between the compute shader, the first SSBO and the second shared memory. The compute shader of the GPU is bound to the second SSBO based on the second binding point and calls the glBufferStorageExternalEXT function to bind the second SSBO to the second shared memory, thereby establishing a second binding relationship between the compute shader, the second SSBO and the second shared memory. Based on the established first binding relationship, the GPU can write the texture data of each pixel point in the first texture map of the first resolution rendered to the first shared memory. Based on the established second binding relationship, the GPU can read the texture data of each pixel point in the third texture map of the second resolution from the second shared memory, generate a fourth texture map of the second resolution, and bind the fourth texture map back to the original rendering process, thereby rendering an image of high resolution.

[0090] The NPU is configured to read the texture data of the pixel points from the first shared memory, generate a second texture map of the first resolution, and call an AI super-resolution model to perform super-resolution rendering on the second texture map to obtain a third texture map of the second resolution, and then write the texture data of each pixel point in the third texture map to the second shared memory.

[0091] Further, considering that the texture data input and output of the AI super-resolution model is numerical value of RGB channel, and the texture data rendered by the GPU in the embodiment of the present application is numerical value of RGBA channel. To meet the input requirements of the AI super-resolution model, when the compute shader of the second processor writes the texture data of each pixel point in the first texture map of the first resolution into the first shared memory, it also converts the texture data of each pixel point from numerical value of RGBA channel to numerical value of RGB channel, and then writes it into the first SSBO, and further writes it into the first shared memory through the first SSBO. Since the texture data of the pixel points in the third texture map of the second resolution rendered by the AI super-resolution model is numerical value of RGB channel, and the GPU rendering needs numerical value of RGBA channel, therefore, when the NPU writes the texture data of each pixel point in the third texture map into the second shared memory, the compute shader converts the texture data of each pixel point in the third texture map read from the second shared memory into numerical value of RGBA channel after writing it into the second SSBO, and finally binds the numerical value of RGBA channel of each pixel point back to the original rendering process.

[0092] Further, since the first SSBO and the second SSBO support uint type data, therefore, when the compute shader writes the texture data of each pixel point in the first texture map into the first SSBO, it also converts the texture data of each pixel point in the first texture map into uint value, and subsequent reading and writing of the texture data of the pixel points are all in uint value.

[0093] Figure 6 A structural schematic diagram of an electronic device 100 is shown. The electronic device 100 can be a mobile phone, a personal computer (PC), a tablet computer, an AR (Augmented Reality) device, a VR (Virtual Reality) device, a vehicle-mounted computer, a wearable device, a smart home device, etc.

[0094] The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0095] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0096] The processor 110 can include one or more processing units, for example, the processor 110 can include an application processor (AP), a modem processor, a GPU, an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or an NPU, etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0097] The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching instructions and executing instructions.

[0098] The processor 110 can also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can hold instructions or data that the processor 110 has just used or is using repeatedly. If the processor 110 needs to use the instructions or data again, it can call them directly from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0099] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a Subscriber Identity Module interface, and / or a Universal Serial Bus (USB) interface, etc.

[0100] The I2C interface is a bidirectional synchronous serial bus that includes a Serial Data Line (SDA) and a Derail Clock Line (SCL). In some embodiments, the processor 110 can include multiple sets of I2C buses. The processor 110 can be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 can be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to realize the touch function of the electronic device 100.

[0101] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple sets of I2S buses. The processor 110 can be coupled to the audio module 170 through the I2S bus to realize communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can deliver audio signals to the wireless communication module 160 through the I2S interface to realize the function of answering a phone through a Bluetooth headset.

[0102] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 can be coupled with the wireless communication module 160 through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface, realizing the function of answering a phone call through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0103] The UART interface is a general-purpose serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is usually used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface, realizing the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface, realizing the function of playing music through a Bluetooth headset.

[0104] The MIPI interface can be used to connect the processor 110 and peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes the camera serial interface (CSI), the display screen serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface, realizing the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface, realizing the display function of the electronic device 100.

[0105] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 and the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0106] The USB interface 130 is an interface that conforms to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, to transmit data between the electronic device 100 and a peripheral device, or to connect a headset to play audio through the headset. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0107] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 100. In some other embodiments of the present application, the electronic device 100 can also use different interface connection modes or a combination of multiple interface connection modes in the above embodiments.

[0108] The charging management module 140 is configured to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input through a wireless charging coil of the electronic device 100. The charging management module 140 can charge the battery 142 and also supply power to the electronic device through the power management module 141.

[0109] The power management module 141 is configured to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to supply power to the processor 110, the internal memory 121, the display screen 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 can also be configured to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), and the like. In some other embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0110] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, and the like.

[0111] The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0112] The mobile communication module 150 can provide a solution including 2G / 3G / 4G / 5G, etc. wireless communication applied to the electronic device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform filtering, amplification, etc. on the received electromagnetic waves, and transfer to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor, and radiate as electromagnetic waves through the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be disposed in the same device as at least part of the modules of the processor 110.

[0113] The modem processor can include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is configured to demodulate a received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the microphone 170B, etc.), or displays an image or a video through the display screen 194. In some embodiments, the modem processor can be a separate device. In other embodiments, the modem processor can be independent of the processor 110, and disposed in the same device as the mobile communication module 150 or other functional modules.

[0114] The wireless communication module 160 can provide a solution for wireless communication including Wireless Local Area Networks (WLAN) (e.g., Wireless Fidelity (Wi-Fi) network), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR) technology, etc. applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives an electromagnetic wave via the antenna 2, frequency-modulates and filters the electromagnetic wave signal, and transmits the processed signal to the processor 110. The wireless communication module 160 can also receive a signal to be transmitted from the processor 110, frequency-modulate and amplify the signal, and radiate the signal as an electromagnetic wave via the antenna 2.

[0115] In some embodiments, the antenna 1 and the mobile communication module 150 of the electronic device 100 are coupled, and the antenna 2 and the wireless communication module 160 are coupled, so that the electronic device 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include Global Positioning System (GPS), Global Navigation Satellite System (GNSS), Beidou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0116] The electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0117] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), an Active-Matrix Organic Light Emitting Diode (AMOLED), a Flex Light-Emitting Diode (FLED), a Miniled, a Micro Led, a Micro-oLed, a Quantum Dot Light Emitting Diodes (QLED), or the like. In some embodiments, the electronic device 100 can include one or N display screens 194, where N is a positive integer greater than 1.

[0118] The electronic device 100 can implement the photographing function through the ISP, the camera 193, a video codec, a GPU, the display screen 194, and an application processor, and the like.

[0119] The ISP is configured to process data fed back by the camera 193. For example, when taking a photo, light is transmitted to the camera photosensitive element through the lens after the shutter is opened, and the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also optimize the algorithm for noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be disposed in the camera 193.

[0120] The camera 193 is configured to capture still images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a Charge Coupled Device (CCD) or a Complementary Metal-Oxide-Semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, the electronic device 100 can include one or N cameras 193, where N is a positive integer greater than 1.

[0121] The digital signal processor is used to process digital signals, in addition to being able to process digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0122] The video codec is used to compress or decompress digital video. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0123] The NPU is a neural network (Neural-Network, NN) computing processor. By drawing on the structure of a biological neural network, such as drawing on the transmission mode between human brain neurons, it can quickly process input information and continuously self-learn. Through the NPU, the electronic device 100 can realize intelligent cognition applications such as image recognition, face recognition, voice recognition, and text understanding.

[0124] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to realize data storage functions. For example, music, video, and other files are saved in the external memory card.

[0125] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required by a function (such as a sound playing function, an image playing function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phonebook, etc.), etc. In addition, the internal memory 121 can include a high-speed random access memory and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a Universal Flash Storage (UFS), etc. The processor 110 executes various function applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory disposed in the processor.

[0126] The electronic device 100 can realize audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc. For example, music playing, recording, etc.

[0127] The keys 190 include a power key, a volume key, and the like. The keys 190 can be mechanical keys or touch keys. The electronic device 100 can receive a key input and generate a key signal input related to user settings and function control of the electronic device 100.

[0128] The motor 191 can generate a vibration prompt. The motor 191 can be used for incoming call vibration prompts or touch vibration feedback. For example, touch operations applied to different applications (e.g., taking pictures, playing audio, and the like) can correspond to different vibration feedback effects. Touch operations applied to different regions of the display screen 194 can also correspond to different vibration feedback effects. Different application scenarios (e.g., time reminders, received messages, alarms, games, and the like) can also correspond to different vibration feedback effects. The touch vibration feedback effects can also be customizable.

[0129] The indicator 192 can be an indicator light that can be used to indicate a charging state, a power change, or can be used to indicate messages, missed calls, notifications, and the like.

[0130] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. The same SIM card interface 195 can simultaneously insert multiple cards. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external storage cards. The electronic device 100 interacts with a network through a SIM card to implement functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0131] Embodiments of the present application provide an image rendering method, which can be executed by an electronic device, which can be an electronic device as shown in the Figure 6 The electronic device includes a first processor, a second processor, and a third processor, and the like. Taking the first processor as a CPU, the second processor as a GPU, and the third processor as an NPU, and taking the first shared memory applied for by the first processor as a first HardwareBuffer and the second shared memory applied for by the first processor as a second HardwareBuffer as examples. Referring to Figure 7 The method flow provided by the embodiments of the present application includes:

[0132] 701、When rendering the first texture map of any frame image at the first resolution is completed, the GPU processes the texture data of the pixel points in the first texture map to obtain a plurality of first texture data units, and sequentially writes the plurality of first texture data units into the first HardwareBuffer.

[0133] The image rendering of the embodiments of the present application involves two resolutions, namely a first resolution and a second resolution. The first resolution is the resolution adopted by the GPU for image rendering, and the second resolution is the resolution adopted by the NPU for super-resolution rendering. The first resolution and the second resolution can be determined according to the performance of the electronic device itself and the display requirements of the running application program, and it is only necessary to ensure that the first resolution is lower than the second resolution. In the running process of the application program, the CPU receives the display instruction sent by the application program, and based on the display instruction, the CPU sends a rendering instruction to the GPU, which includes the first resolution and the like. In response to the rendering instruction, the GPU renders the image data of the application program. When the first texture map of any frame image at the first resolution is rendered, the compute shader of the GPU processes the texture data of each pixel point in the first texture map to obtain a plurality of first texture data units.

[0134] The texture map can be a UV map. The so-called UV map refers to an image rendering method in which each point on the image is accurately mapped to the surface of a model object, and image smoothing interpolation processing is performed on the gap positions between the points. UV is the abbreviation of texture map coordinates, which defines the position information of each point on the image, and these points are related to the three-dimensional model to determine the position of the surface texture map. The number of pixel points in the first texture map can be determined according to the first resolution. For example, if the first resolution is W*H, the number of pixel points in the first texture map can be W*H.

[0135] In the embodiments of the present application, the texture data of the pixel points can be first texture data or second texture data, the data formats of the first texture data and the second texture data are different, the first texture data can be a value represented by an RGBA channel (hereinafter referred to as a value of an RGBA channel), and the second texture data can be a value represented by an RGB channel (hereinafter referred to as a value of an RGB channel), wherein the R channel, the G channel and the B channel are three color channels, and the A channel is a parameter representing the opacity of the image. The texture data of each pixel point in the first texture map is first texture data, i.e., a value of an RGBA channel. Since the AI super-resolution model called by the NPU requires the texture data input and output to be second texture data, i.e., a value of an RGB channel, in order to meet the requirements of the AI super-resolution model, the compute shader needs to convert the first texture data of each pixel point in the first texture map into second texture data, i.e., convert a value of an RGBA channel into a value of an RGB channel.

[0136] Further, the data type of the first texture data unit is a preset type, which can be determined according to the data types supported by the first SSBO and the second SSBO. In the embodiment of the application, the data types supported by the first SSBO and the second SSBO are uint types, and therefore the data type of the first texture data unit can be a uint type, and each first texture data unit can correspond to a uint value. Since the data type of the first texture data of each pixel point in the first texture map is not a uint type, in order to meet the data storage requirements of the first SSBO and the second SSBO, the compute shader needs to convert the first texture data of each pixel point into a uint value when converting the first texture data of each pixel point in the first texture map into the second texture data. Generally, a uint value is 4 bytes, a total of 32 bits, and the value of each channel in the RGBA channel and the RGB channel is 8 bits, that is, 8 bits per channel. When the first texture data of a pixel point is converted into the second texture data, since the value of the RGB channel is less than a uint value, and data is read and written in units of uint values, therefore, the second texture data of two adjacent pixel points needs to be combined into a uint value, that is, each first texture data unit includes all or part of the second texture data of two adjacent pixel points in the first texture map.

[0137] Specifically, the compute shader processes the texture data of the pixel points in the first texture map to obtain a plurality of first texture data units, including the following steps:

[0138] 7011. The compute shader calculates the right shift number of the first texture data of the first pixel point according to the pixel identifier of the first pixel point.

[0139] The first pixel point is any pixel point in the first texture map. The pixel identifier is used to indicate the position of the pixel point in the first texture map, and can be in the form of a number. When setting the pixel identifiers of the pixel points in the first texture map, the first pixel point on the left side of the first row can be numbered from the first pixel point on the left side of the first row, and the pixel identifier of the first pixel point on the left side of the first row is set to 0. When setting the pixel identifiers of the pixel points in the first row, the pixel identifier of the first pixel point on the right side of the second row is set. When setting the pixel identifiers of the pixel points in the second row, the pixel identifier of the first pixel point on the left side of the third row is set, and so on, until the last pixel point in the first texture map.

[0140] The compute shader can use the following steps to calculate the right shift number of the first texture data of the first pixel point according to the pixel identifier of the first pixel point.

[0141] 70111、The computing shader obtains a first remainder value by calculating a product of the pixel identifier of the first pixel point and the first value, and a ratio of the second value.

[0142] The first value is a number of channels included in the first texture data, and the second value is a number of channels included in the second texture data. When the first texture data is a value of RGBA channels and the second texture data is a value of RGB channels, the first value can be 4, and the second value can be 3.

[0143] 70112、The computing shader calculates a product of the first remainder value and a third value to obtain a first product value.

[0144] The third value is a number of bits occupied by each channel when the first texture data is represented by a preset type. When the preset type is uint, the third value can be 8.

[0145] 70113、The computing shader takes the first product value as a right shift number of the first texture data of the first pixel point.

[0146] When calculating the right shift number of the first texture data of the first pixel point according to the pixel identifier of the first pixel point, the computing shader can use the following formula:

[0147] The right shift number of the first texture data of the pixel point = the pixel identifier of the pixel point x 4 % 3 x 8

[0148] The % represents a remainder of a ratio.

[0149] Referring to Figure 8 The first texture map includes pixel points with Id (i.e., pixel identifier) of 0, 1, 2, and 3. For the pixel point with Id = 0, the right shift number of the first texture data of the pixel point with Id = 0 can be obtained by substituting the pixel point into the above formula. For the pixel point with Id = 1, the right shift number of the first texture data of the pixel point with Id = 1 can be obtained by substituting the pixel point into the above formula. For the pixel point with Id = 2, the right shift number of the first texture data of the pixel point with Id = 2 can be obtained by substituting the pixel point into the above formula. For the pixel point with Id = 3, the right shift number of the first texture data of the pixel point with Id = 3 can be obtained by substituting the pixel point into the above formula.

[0150] 7012、The computing shader calculates a left shift number of the first texture data of the second pixel point according to the right shift number of the first texture data of the first pixel point.

[0151] The second pixel point is the next pixel point on the first texture map that is nearest neighbor to the first pixel point. If the pixel identifiers of the pixel points are continuous, the second pixel point can be the pixel point corresponding to the pixel identifier that is 1 greater than the pixel identifier of the first pixel point. For example, if the pixel identifier of the first pixel point is 1, the second pixel point is the pixel point with the pixel identifier of 2. The fourth numerical value is the maximum number of bits by which the first texture data of the second pixel point can be shifted to the left. When the preset type is uint, the fourth numerical value can be 24. The computing shader calculates a difference between the fourth numerical value and the number of bits by which the first texture data of the first pixel point is shifted to the right, to obtain a first difference value, and uses the first difference value as the number of bits by which the first texture data of the second pixel point is shifted to the left.

[0152] The computing shader calculates the number of bits by which the first texture data of the second pixel point is shifted to the left according to the number of bits by which the first texture data of the first pixel point is shifted to the right, and can use the following formula:

[0153] The number of bits by which the first texture data of the second pixel point is shifted to the left = 24 - the number of bits by which the first texture data of the first pixel point is shifted to the right

[0154] Referring to Figure 8 , the first texture map includes pixel points with Ids of 0, 1, 2, and 3. When the first pixel point is the pixel point with Id = 0 and the second pixel point is the pixel point with Id = 1, the number of bits by which the first texture data of the first pixel point is shifted to the right is 0, and the number of bits by which the first texture data of the second pixel point is shifted to the left is 24. When the first pixel point is the pixel point with Id = 1 and the second pixel point is the pixel point with Id = 1, the number of bits by which the first texture data of the first pixel point is shifted to the right is 8, and the number of bits by which the first texture data of the second pixel point is shifted to the left is 16. When the first pixel point is the pixel point with Id = 2 and the second pixel point is the pixel point with Id = 3, the number of bits by which the first texture data of the first pixel point is shifted to the right is 16, and the number of bits by which the first texture data of the second pixel point is shifted to the left is 8.

[0155] 7013. The computing shader generates the first texture data unit according to the first texture data of the first pixel point and the number of bits by which the first texture data of the first pixel point is shifted to the right, and the first texture data of the second pixel point and the number of bits by which the first texture data of the second pixel point is shifted to the left.

[0156] The first texture data is a numerical value of an RGBA channel, the second texture data is a numerical value of an RGB channel, and the preset type is uint. The computing shader generates the first texture data unit according to the first texture data of the first pixel point and the number of bits by which the first texture data of the first pixel point is shifted to the right, and the first texture data of the second pixel point and the number of bits by which the first texture data of the second pixel point is shifted to the left, including the following steps:

[0157] 70131. The computing shader modifies the numerical value of the A channel in the RGBA channel of the first pixel point and the RGBA channel of the second pixel point to 0.

[0158] 70132、The computing shader converts the modified value of the RGBA channel of the first pixel point into a first uint value, and converts the modified value of the RGBA channel of the second pixel point into a second uint value.

[0159] The uint value converted by the RGBA channel in the embodiment of the present application is ordered in the order from right to left, that is, the R channel occupies the 1-8 bits of the uint value, the G channel occupies the 9-16 bits of the uint value, the B channel occupies the 17-24 bits of the uint value, and the A channel occupies the 25-32 bits of the uint value.

[0160] 70133、The computing shader right shifts the first uint value according to the right shift number of the RGBA channel of the first pixel point to obtain a third uint value.

[0161] For example, the first uint value of the first pixel point is RGBA, if it is right shifted by 8 bits, the third uint value obtained is GBA0; if it is right shifted by 16 bits, the third uint value obtained is BA00.

[0162] 70134、The computing shader left shifts the second uint value according to the left shift number of the RGBA channel of the second pixel point to obtain a fourth uint value.

[0163] For example, the second uint value of the second pixel point is RGBA, if it is left shifted by 8 bits, the fourth uint value obtained is 0RGB; if it is left shifted by 16 bits, the fourth uint value obtained is 00RG.

[0164] 70135、The computing shader adds the values of the same number of bits of the third uint value and the fourth uint value to obtain a first texture data unit of the uint type.

[0165] For the process of processing the texture data of the pixel points in the first texture map by the computing shader to obtain a plurality of first texture data units, for the convenience of understanding, the following will be described in combination with Figure 8 .

[0166] Referring to Figure 8, the first texture map includes pixel points with Ids of 0, 1, 2, and 3, and the value of the A channel of the RGBA channel of the pixel with Id 0, 1, 2, and 3 is set to 0. When the first pixel point is the pixel point with Id 0 and the second pixel point is the pixel point with Id 1, the right shift number of the first pixel point is 0, and the left shift number of the second pixel point is 24. Since the right shift number of the first pixel point is 0, the first pixel point does not need to be moved and is still RGBA. The values of the RGBA channel of the second pixel point are moved 24 bits to the left to obtain 000R. The values of the same number of bits of RGBA and 000R are added to obtain RGBR. The RGBR is a first texture data unit of uint type, which includes the values of the RGB channel of the pixel point with Id 0 and the value of the R channel of the pixel point with Id 1. When the first pixel point is the pixel point with Id 1 and the second pixel point is the pixel point with Id 2, the right shift number of the first pixel point is 8, and the left shift number of the second pixel point is 16. The values of the RGBA channel of the first pixel point are moved 8 bits to the right to obtain GBA0. The values of the RGBA channel of the second pixel point are moved 16 bits to the left to obtain 00GR. The values of the same number of bits of GBA0 and 00GR are added to obtain GBRG. The GBRG is a first texture data unit of uint type, which includes the values of the GB channel of the pixel point with Id 1 and the values of the RG channel of the pixel point with Id 2. When the first pixel point is the pixel point with Id 2 and the second pixel point is the pixel point with Id 3, the right shift number of the first pixel point is 16, and the left shift number of the second pixel point is 16. The values of the RGBA channel of the first pixel point are moved 16 bits to the right to obtain BA00. The values of the RGBA channel of the second pixel point are moved 16 bits to the left to obtain 0RGB. The values of the same number of bits of BA00 and 0RGB are added to obtain BRGB. The BRGB is a first texture data unit of uint type, which includes the value of the B channel of the pixel point with Id 2 and the values of the RGB channel of the pixel point with Id 3.

[0167] In a possible implementation, when the first texture data of each pixel point in the first texture map is processed to obtain a plurality of first texture data units, the computing shader can sequentially write the plurality of first texture data units into the first SSBO based on the first binding relationship, and sequentially write the plurality of first texture data units into the first shared memory through the first SSBO.

[0168] In another possible implementation, each time a first texture data unit is generated, the computing shader writes the generated first texture data unit into the first SSBO and writes the first texture data unit into the first shared memory through the first SSBO.

[0169] By adopting the method provided in the embodiments of the present application, the first texture data of each pixel point in the first texture map is converted into a plurality of first texture data units and written into a row of the first shared memory, and there is no blank pixel point between each pixel point, thereby avoiding the subsequent problem of screen flashing.

[0170] Further, in order to facilitate management of the plurality of first texture data units, the computing shader can set an index value for the plurality of first texture data units, and the index value can be in the form of a number. When setting the index value for the plurality of first texture data units, the computing shader can set the index value in the order of generation of the first texture data units, for example, the index value set for the first generated first texture data unit is 0, the index value set for the second generated first texture data unit is 0, and so on, and the index value set for the Mth generated first texture data unit is M, where M is greater than or equal to 0.

[0171] Figure 9 The process of the computing shader writing the texture data of the pixel points in the first texture map into the first SSB is shown, referring to Figure 9 , the computing shader calculates the right shift number of the first texture data of the current pixel point (i.e., the first pixel point) and the left shift number of the first texture data of the next pixel point (i.e., the second pixel point), then obtains the values of the RGBA channels of the current pixel point and the values of the RGBA channels of the next pixel point, and further converts the values of the RGBA channels of the current pixel point into uint values and converts the values of the RGBA channels of the next pixel point into uint values. The computing shader right shifts the values of the RGBA channels of the current pixel point according to the right shift number corresponding to the current pixel point, left shifts the values of the RGBA channels of the next pixel point according to the left shift number corresponding to the next pixel point, then adds the values of the same number of bits of the uint values of the current pixel point after the bit operation and the uint values of the next pixel point after the bit operation, and writes the obtained uint value (i.e., the first texture data unit) into the first SSB.

[0172] 702、The NPU generates a second texture map of a first resolution based on the plurality of first texture data units stored in the first HardwareBuffer, performs super-resolution rendering on the second texture map to obtain a third texture map of a second resolution, and sequentially writes the texture data of the pixel points in the third texture map into the second HardwareBuffer to obtain a plurality of second texture data units.

[0173] In the embodiment of the present application, after the GPU writes the first texture data of the pixel points in the first texture map to the first HardwareBuffer, the GPU can send a notification message to the CPU. When receiving the notification message, the CPU can send a super-resolution rendering instruction to the NPU. In response to the super-resolution rendering instruction, the NPU can read the second texture data of each pixel point from the plurality of first texture data units stored in the second HardwareBuffer, generate a second texture map of the first resolution based on the number of rows and the number of columns of the pixel points indicated by the first resolution, and then perform super-resolution rendering on the second texture map to obtain a third texture map, and then sequentially write the texture data of the pixel points in the third texture map to the second HardwareBuffer to obtain a plurality of second texture data units. The second texture map has the same pixel points at each position as the first texture map, but the data format of the pixel points is different. The texture data of the pixel points on the first texture map is the first texture data, and the texture data of the pixel points on the second texture map is the second texture data. The texture data of each pixel point in the third texture map is the second texture data.

[0174] In the embodiment of the present application, since the first texture data unit stores all or part of the second texture data of two adjacent pixel points, when the NPU reads the second texture data of each pixel point from the plurality of first texture data units stored in the second HardwareBuffer, the NPU can read according to the pixel identifier. Specifically, the NPU can determine at least one first texture data unit storing the second texture data of the pixel point according to the pixel identifier of the pixel point, and obtain the stored second texture data from each first texture data unit. If the second texture data of the pixel point is stored in one first texture data unit, the second texture data of the pixel point can be obtained from the first texture data unit. If the second texture data of the pixel point is stored in two first texture data units, the second texture data of the pixel point can be obtained from the two first texture data units, respectively. Then, the partial values of the second texture data of the pixel point obtained from the two first texture data units are combined into the full value of the second texture data.

[0175] In the embodiment of the present application, when the NPU generates the second texture map of the first resolution based on the number of rows and the number of columns of the pixel points indicated by the first resolution, the texture data of the pixel points read from the first shared memory can be sequentially filled into the texture matrix corresponding to the first resolution according to the pixel identifier, to obtain the second texture map of the first resolution. For example, if the first resolution is 10*10, the NPU can write the pixel point with a pixel identifier of 0 read from the first texture data unit into the first row and the first column of the texture matrix, write the pixel point with a pixel identifier of 1 into the first row and the second column of the texture matrix,..., write the pixel point with a pixel identifier of 9 into the first row and the tenth column of the texture matrix, and write the pixel point with a pixel identifier of 10 into the second row and the first column of the texture matrix, and so on, until the texture matrix is filled, thereby obtaining the second texture map.

[0176] In the embodiment of the present application, when the NPU performs super-resolution rendering on the second texture map to obtain the third texture map, an AI super-resolution model can be called to perform super-resolution rendering on the second texture map to obtain the third texture map. After the third texture map is generated, the NPU can write the second texture data of each pixel point in the third texture map into the second HardwareBuffer to obtain a plurality of second texture data units, each of which corresponds to a uint value. Since the texture data of each pixel point in the third texture map generated by the AI super-resolution model is the second texture data, i.e., the numerical value of the RGB channel, which is less than a uint value, and data is read and written in units of uint values, it is necessary to form a uint value with all or part of the second texture data of the next pixel point. At this time, each second texture data unit will include all or part of the second texture data of two adjacent pixel points in the third texture map.

[0177] Further, in order to facilitate management of the plurality of second texture data units, the NPU will also set an index value for the plurality of second texture data units. The index value can be in the form of a number, which is used to represent the storage location of the second texture data unit in the second shared memory. When setting the index value for the plurality of second texture data units, the NPU can set the index value according to the generation order of the second texture data units, for example, the index value set for the first generated second texture data unit is 0, the index value set for the second generated second texture data unit is 0, and so on, and the index value set for the Nth generated second texture data unit is N, where N is greater than or equal to 0.

[0178] 703、The GPU processes the plurality of second texture data units stored in the second HardwareBuffer to obtain a fourth texture map of the second resolution, and generates an image of the second resolution based on the fourth texture map.

[0179] In the embodiment of the present application, after the NPU writes the second texture data of each pixel point in the third texture map into the second HardwareBuffer, the NPU can send a notification message to the CPU. After the CPU receives the notification message, the CPU sends a rendering instruction to the GPU. In response to the rendering instruction, the compute shader of the GPU reads a plurality of second texture data units from the second HardwareBuffer, and caches the plurality of second texture data units into the second SSBO based on the second binding relationship. Then, the compute shader processes the plurality of second texture data units in the second SSBO to obtain the first texture data of each pixel point at the second resolution. Further, based on the first texture data of each pixel point at the second resolution, the fourth texture map is generated. The fourth texture map is the same as the pixel point at each position in the third texture map, but the data format of the texture data of the pixel point is different. The texture data of the pixel point in the third texture map is the first texture data, and the texture data of the pixel point in the fourth texture map is the second texture data.

[0180] Specifically, the compute shader processes the plurality of second texture data units in the second SSBO to obtain the first texture data of each pixel point at the second resolution, including the following steps:

[0181] 7031、The compute shader determines the minimum index value of the second texture data unit for storing the second texture data of the third pixel point in the second SSBO according to the pixel identifier of the third pixel point.

[0182] The third pixel point is any pixel point in the plurality of second texture data units. The compute shader calculates the quotient of the product of the pixel identifier of the third pixel point and the second value and the first value, and takes the obtained quotient as the minimum index value.

[0183] Specifically, when the compute shader determines the minimum index value of the second texture data unit for storing the second texture data of the third pixel point in the second SSBO according to the pixel identifier of the third pixel point, the following formula can be used:

[0184] Minimum index value = pixel identifier of third pixel point x 3 ÷ 4

[0185] For example, if the pixel identifier of the third pixel point is 0, the minimum index value of the second texture data unit for storing the second texture data of the third pixel point in the second SSBO is 0, that is, all or part of the second texture data of the third pixel point is stored in the second texture data unit with the index of 0; if the pixel identifier of the third pixel point is 1, the minimum index value of the second texture data unit for storing the second texture data of the third pixel point in the second SSBO is 0, that is, all or part of the second texture data of the third pixel point is stored in the second texture data unit with the index of 0; if the pixel identifier of the third pixel point is 2, the minimum index value of the second texture data unit for storing the second texture data of the third pixel point in the second SSBO is 1, that is, all or part of the second texture data of the third pixel point is stored in the second texture data unit with the index of 1.

[0186] 7032、The computing shader calculates the right shift number of the second texture data in the first target data unit according to the pixel identifier of the third pixel point.

[0187] The first target data unit is the second texture data unit corresponding to the minimum index value. The computing shader obtains the second remainder value by calculating the product of the pixel identifier of the third pixel point and the second value and dividing the product by the first value, and obtains the second product value by calculating the product of the second remainder value and the third value, and then takes the second product value as the right shift number of the second texture data stored in the first target data unit.

[0188] Specifically, when the computing shader calculates the right shift number of the second texture data in the first target data unit according to the pixel identifier of the third pixel point, the following formula can be used:

[0189] The right shift number of the second texture data in the first target data unit = third pixel point id x 3 % 4 x 8

[0190] For example, for the third pixel point with the pixel identifier of 0, the right shift number of the second texture data in the first target data unit of the third pixel point is 0; for the third pixel point with the pixel identifier of 1, the right shift number of the second texture data in the first target data unit of the third pixel point is 24; for the third pixel point with the pixel identifier of 2, the right shift number of the second texture data in the first target data unit of the third pixel point is 16.

[0191] 7033、When the right shift number of the second texture data in the first target data unit is not 0, the computing shader calculates the left shift number of the second texture data in the second target data unit.

[0192] The second target data unit is a next second texture data unit in the second SSBO that is most adjacent to the first target data unit. If the second texture data units are sequentially numbered, the second target data unit is a second texture data unit corresponding to an index value obtained by adding 1 to a minimum index value. When the right shift number of the second texture data in the first target data unit is not 0, it indicates that the second texture data in the first target data unit needs to be offset, and the computing shader needs to calculate the left shift number of the second texture data in the second target data unit. Specifically, the computing shader calculates a difference between the fifth value and the right shift number corresponding to the first target data unit to obtain a second difference value, and then uses the second difference value as the left shift number of the second texture data stored in the second target data unit. The fifth value is a maximum number of bits by which the second texture data can be left shifted in the second target data unit, and the fifth value can be 32.

[0193] Specifically, when calculating the left shift number of the second texture data in the second target data unit, the computing shader can use the following formula:

[0194] The left shift number of the (index+1)th uint value = 32 - the right shift number of the indexth uint value

[0195] The left shift number of the (index+1)th uint value represents the left shift number of the second texture data in the second target data unit, and the right shift number of the indexth uint value represents the right shift number of the second texture data in the first target data unit.

[0196] For example, for a third pixel point with a pixel identifier of 1, the right shift number of the second texture data in the first target data unit of the third pixel point is 24, and the left shift number of the second texture data in the second target data unit is 8. For a third pixel point with a pixel identifier of 2, the right shift number of the second texture data in the first target data unit of the third pixel point is 16, and the left shift number of the second texture data in the second target data unit is 16.

[0197] 7034、The computing shader determines the first texture data of the third pixel point according to the second texture data in the first target data unit, the right shift number corresponding to the first target data unit, the second texture data in the second target data unit, and the left shift number corresponding to the second target data unit.

[0198] The first texture data is a value of an RGBA channel, the second texture data is a value of an RGB channel, and the preset type is uint. When the computing shader determines the first texture data of the third pixel point according to the second texture data in the first target data unit, the right shift number corresponding to the first target data unit, the second texture data in the second target data unit and the left shift number corresponding to the second target data unit, the computing shader can right shift the second texture data in the first target data unit according to the right shift number corresponding to the first target data unit, to obtain a fifth uint value, and left shift the second texture data in the second target data unit according to the left shift number corresponding to the second target data unit, to obtain a sixth uint value, then add the values of the same number of bits of the fifth uint value and the sixth uint value to obtain a seventh uint value, the seventh uint value includes all values of the RGB channel of the third pixel point, then convert the seventh uint value into a value of the RGBA channel, modify the value of the A channel in the RGBA channel to 1, and obtain the first texture data of the third pixel point.

[0199] In another embodiment of the present application, when the right shift number of the second texture data in the first target data unit is 0, the computing shader can take the eighth uint value corresponding to the second texture data in the first target data unit as the seventh uint value.

[0200] Referring to Figure 10 For the three uint type second texture data units with indexes 0, 1 and 2, the second texture data unit corresponding to the index 0 is RGBR value, including the RGB value of the pixel point with Id=0 and the R value of the pixel point with Id=1; the second texture data unit corresponding to the index 1 is GBRG value, including the GB value of the pixel point with Id=1 and the RG value of the pixel point with Id=2; and the second texture data unit corresponding to the index 2 is BRGB value, including the B value of the pixel point with Id=2 and the RGB value of the pixel point with Id=3. When the pixel point with Id=0 is the third pixel point, the minimum index value corresponding to the third pixel point is 0, the first target data unit is the second texture data unit with index 0, according to the Id=0 of the third pixel point, it is calculated that the right shift number of the second texture data in the second texture data unit with index 0 is 0, then the RGBR value corresponding to the second texture data unit with index 0 is taken as the uint value corresponding to the pixel point with Id=0, then the uint value is converted into RGBA value, and the value of the A channel is modified to 1, to obtain the RGBA value of the pixel point with Id=0.

[0201] When the pixel point with Id=1 is the third pixel point, the minimum index value corresponding to the third pixel point is 0, the first target data unit is the second texture data unit with the index value of 0, according to the Id=1 of the third pixel point, it is calculated that the right shift number of the second texture data in the second texture data unit with the index value of 0 is 24, then the second target data unit is the second texture data unit with the index value of 1, the left shift number of the second texture data in the second target data unit is 8, then according to the right shift number of the second texture data in the second texture data unit with the index value of 0 being 24, the RGBR value corresponding to the second texture data unit with the index value of 0 is moved right by 24 bits to obtain the R000 value, and according to the left shift number of the second texture data in the second texture data unit with the index value of 1 being 8, the GBRG value in the second texture data unit with the index value of 1 is moved left by 8 bits to obtain the 0GBR value, then the values at the same bit number of the R000 value and the 0GBR value are added to obtain the RGBR value as the uint value corresponding to the pixel point with Id=1, then the uint value is converted into the RGBA value, and the value at the last bit of the A channel is modified to 1 and the values at other bits are modified to 0 to obtain the RGBA value of the pixel point with Id=1.

[0202] When the pixel point with Id=2 is the third pixel point, the minimum index value corresponding to the third pixel point is 1, the first target data unit is the second texture data unit with the index value of 1, according to the Id=2 of the third pixel point, it is calculated that the right shift number of the second texture data in the second texture data unit with the index value of 2 is 16, then the second target data unit is the second texture data unit with the index value of 2, the left shift number of the second texture data in the second target data unit is 16, then according to the right shift number of the second texture data in the second texture data unit with the index value of 1 being 16, the GBRG value corresponding to the second texture data unit with the index value of 1 is moved right by 16 bits to obtain the RG00 value, and according to the left shift number of the second texture data in the second texture data unit with the index value of 2 being 16, the BRGB value in the second texture data unit with the index value of 2 is moved left by 16 bits to obtain the 00BR value, then the values at the same bit number of the RG00 value and the 00BR value are added to obtain the RGBR value as the uint value corresponding to the pixel point with Id=2, then the uint value is converted into the RGBA value, and the value at the last bit of the A channel is modified to 1 and the values at other bits are modified to 0 to obtain the RGBA value of the pixel point with Id=2.

[0203] Based on the first texture data of each pixel point at the second resolution, the shader determines the position of each pixel point in the texture matrix corresponding to the second resolution according to the pixel identification of each pixel point, and then writes the first texture data of each pixel point into the texture matrix to obtain the fourth texture map at the second resolution.

[0204] Further, based on the obtained fourth texture map, the GPU binds the fourth texture map back to the original rendering process to obtain an image of the second resolution.

[0205] To facilitate understanding of the processing process of the GPU on the plurality of second texture data units in the second SSBO, the following describes the processing process of the GPU on the plurality of second texture data units in the second SSBO with reference to Figure 11 .

[0206] Referring to Figure 11 , the compute shader calculates, according to the pixel identifier of the current pixel point (i.e., the third pixel point), the index (the minimum index value) of the second texture data corresponding to the current pixel point in the second SSBO, and the right shift number of the second texture data in the indexth second texture data unit in the second SSBO. If the right shift number is 0, the uint value corresponding to the indexth second texture data unit is taken as the uint value of the pixel point. If the right shift number is not 0, the left shift number of the second texture data in the index+1th second texture data unit in the second SSBO is calculated, and then the second data in the indexth second texture data unit is right shifted according to the right shift number corresponding to the indexth second texture data unit, and the second data in the index+1th second texture data unit is left shifted according to the left shift number corresponding to the index+1th second texture data unit, so that the bit shift results of the values in the indexth second texture data unit and the values in the index+1th second texture data unit are the same, and the uint value corresponding to the pixel point is obtained. Then, the uint value corresponding to the current pixel point is converted into an RGBA value, and then the RGBA value of the pixel point is written into the image.

[0207] The method provided in the embodiments of the present application does not write blank pixel points into the texture map, avoids disorder of the pixel points in the texture map, and thus, based on the generated high-resolution texture map, not only can the picture content of the image be accurately rendered, but also the image clarity is improved due to the high resolution, and the image display effect is better. For example, Figure 12 , the image rendered by the method provided in the embodiments of the present application has a higher clarity and a better image display effect than the image shown in Figure 3 . Figure 12

[0208] In addition, the embodiments of the present application use the same compute shader to process the input and output of the AI super-resolution model, reduce the number of drawcalls, reduce the power consumption of the electronic device, and improve the performance of the electronic device.

[0209] ​In addition, the method provided by the embodiments of the present application is suitable for various scenarios of image rendering based on GPU, including game playing scenarios, augmented reality application scenarios, and the like.

[0210] It should also be noted that the terms "comprising", "containing", or any other variant thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0211] The above only describes the embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. An image rendering method, characterized by, The method is applied to an electronic device including a first processor, a second processor and a third processor, the first processor is configured to create a first shared memory and a second shared memory, data in the first shared memory and the second shared memory is stored in a line, the method comprises: When a first texture map of any frame image is rendered at a first resolution, the second processor processes texture data of a pixel point in the first texture map to obtain a plurality of first texture data units, and sequentially writes the plurality of first texture data units into the first shared memory; The third processor generates a second texture map of the first resolution based on the plurality of first texture data units stored in the first shared memory, the pixel point at each position in the first texture map and the second texture map is the same, performs super-resolution rendering on the second texture map to obtain a third texture map of a second resolution higher than the first resolution, and sequentially writes texture data of a pixel point in the third texture map into the second shared memory to obtain a plurality of second texture data units; The second processor processes the plurality of second texture data units stored in the second shared memory to obtain a fourth texture map of the second resolution, the pixel point at each position in the third texture map and the fourth texture map is the same, and generates the image of the second resolution based on the fourth texture map.

2. The method of claim 1, wherein, Before the second processor processes the texture data of each pixel point in the first texture map to obtain a plurality of first texture data units, the method further comprises: The second processor creates a compute shader, and creates a first shader storage buffer object (SSBO) and a second SSBO in the compute shader, the first SSBO provides a first binding point, and the second SSBO provides a second binding point; The second processor establishes a first binding relationship among the first SSBO, the compute shader and the first shared memory based on the first binding point; The second processor establishes a second binding relationship among the second SSBO, the compute shader and the second shared memory based on the second binding point.

3. The method of claim 2, wherein, The texture data of a pixel point in the first texture map is first texture data, the first texture data unit includes all or part of second texture data of two adjacent pixel points in the first texture map, the data format of the first texture data and the second texture data is different, and the second processor processes the texture data of a pixel point in the first texture map to obtain a plurality of first texture data units, including: The compute shader calculates a right shift number of the first texture data of a first pixel point according to a pixel identifier of the first pixel point, and the first pixel point is any pixel point in the first texture map. The computing shader calculates a left shift number of first texture data of a second pixel point according to a right shift number of the first texture data of the first pixel point, the second pixel point being a next nearest neighbor pixel point on the first texture map to the first pixel point; The computing shader generates a first texture data unit of a preset type according to the first texture data of the first pixel point and the right shift number thereof, the first texture data of the second pixel point and the left shift number thereof.

4. The method of claim 3, wherein, The computing shader calculates a left shift number of first texture data of a second pixel point according to a right shift number of the first texture data of the first pixel point, the second pixel point being a next nearest neighbor pixel point on the first texture map to the first pixel point; The computing shader obtains a first remainder value by calculating a product of a pixel identifier of the first pixel point and a first value and a ratio of a second value, the first value being a number of channels included in the first texture data, and the second value being a number of channels included in the second texture data; The computing shader calculates a product of the first remainder value and a third value to obtain a first product value, the third value being a number of bits occupied by each channel when the first texture data is represented in the preset type; The computing shader takes the first product value as the right shift number of the first texture data of the first pixel point.

5. The method of claim 3, wherein, The computing shader calculates a left shift number of first texture data of a second pixel point according to a right shift number of the first texture data of the first pixel point, the second pixel point being a next nearest neighbor pixel point on the first texture map to the first pixel point; The computing shader calculates a first difference value by calculating a difference between a fourth value and the right shift number of the first texture data of the first pixel point, the fourth value being a maximum number of bits by which the first texture data of the second pixel point can be left shifted; The computing shader takes the first difference value as the left shift number of the first texture data of the second pixel point.

6. The method of claim 3, wherein, The first texture data is a value of an RGBA channel, the second texture data is a value of an RGB channel, the preset type is a uint type, and the computing shader generates a first texture data unit of the preset type according to the first texture data of the first pixel point and the right shift number thereof, the first texture data of the second pixel point and the left shift number thereof, including: The computing shader modifies a value of an A channel in the RGBA channel of the first pixel point and the RGBA channel of the second pixel point to 0; The computing shader converts the modified value of the RGBA channel of the first pixel point into a first uint value and converts the modified value of the RGBA channel of the second pixel point into a second uint value; The computing shader right shifts the first uint value according to the right shift number of the RGBA channel of the first pixel point to obtain a third uint value; The computing shader left shifts the second uint value according to the left shift number of the RGBA channel of the second pixel point to obtain a fourth uint value; The computing shader adds values of the same bits in the third uint value and the fourth uint value to obtain a first texture data unit of the uint type.

7. The method of claim 3, wherein, The second processor sequentially writes the plurality of first texture data units into the first shared memory, including: Based on the first binding relationship, the compute shader sequentially writes the plurality of first texture data units into the first SSBO and sequentially writes the plurality of first texture data units into the first shared memory through the first SSBO.

8. The method of claim 1, wherein, The third processor generates the second texture map of the first resolution based on the plurality of first texture data units stored in the first shared memory, including: The third processor reads the texture data of the pixel points from the plurality of first texture data units stored in the first shared memory, and generates the second texture map based on the number of rows and columns of the pixel points indicated by the first resolution.

9. The method of claim 1, wherein, The third processor performs super-resolution rendering on the second texture map to obtain a third texture map of a second resolution, including: The third processor calls an AI super-resolution model to perform super-resolution rendering on the second texture map to obtain the third texture map.

10. The method of claim 2, wherein, The second texture data unit is of a preset type, and includes all or part of second texture data of two adjacent pixel points in the third texture map. The texture data of the pixel points in the fourth texture map is first texture data. The data format of the first texture data is different from that of the second texture data. The second processor processes the plurality of second texture data units stored in the second shared memory to obtain a fourth texture map of a second resolution, including: The compute shader caches the plurality of second texture data units stored in the second shared memory into the second SSBO based on the second binding relationship, and processes the plurality of second texture data units in the second SSBO to obtain first texture data of each pixel point at a second resolution; Based on the first texture data of each pixel point at the second resolution, the fourth texture map is generated.

11. The method of claim 10, wherein, The compute shader processes the plurality of second texture data units in the second SSBO to obtain first texture data of each pixel point at a second resolution, including: The compute shader determines a minimum index value of a second texture data unit storing second texture data of a third pixel point in the second SSBO according to a pixel identifier of the third pixel point, the third pixel point being any pixel point in the plurality of second texture data units; The compute shader calculates a right shift number of second texture data in a first target data unit according to the pixel identifier of the third pixel point, the first target data unit being the second texture data unit corresponding to the minimum index value; When the right shift number of second texture data in the first target data unit is not 0, the compute shader calculates a left shift number of second texture data in a second target data unit, the second target data unit being the next second texture data unit in the second SSBO most adjacent to the first target data unit; The computing shader determines the first texture data of the third pixel point according to second texture data in the first target data unit, a right shift number corresponding to the first target data unit, second texture data in the second target data unit, and a left shift number corresponding to the second target data unit.

12. The method of claim 11, wherein, The computing shader determines the minimum index value of the second texture data unit for storing the second texture data of the third pixel point in the second SSBO according to the pixel identifier of the third pixel point, and the minimum index value comprises: The computing shader calculates a quotient value obtained by multiplying the pixel identifier of the third pixel point by a second value and dividing by a first value, and the first value is a number of channels included in the first texture data, and the second value is a number of channels included in the second texture data.

13. The method of claim 11, wherein, The computing shader calculates the right shift number of the second texture data stored in the first target data unit according to the pixel identifier of the third pixel point, and the right shift number comprises: The computing shader obtains a second remainder value by calculating a quotient value obtained by multiplying the pixel identifier of the third pixel point by a second value and dividing by a first value, and the first value is a number of channels included in the first texture data, and the second value is a number of channels included in the second texture data. The computing shader calculates a second product value by multiplying the second remainder value by a third value, and the third value is a number of bits occupied by each channel when the first texture data is represented in the preset type. The computing shader takes the second product value as the right shift number of the second texture data stored in the first target data unit.

14. The method of claim 11, wherein, The computing shader calculates the left shift number of the second texture data stored in the second target data unit, and the left shift number comprises: The computing shader calculates a second difference value by calculating a difference between a fifth value and the right shift number corresponding to the first target data unit, and the fifth value is a maximum number of bits that the second texture data can be left shifted in the second target data unit. The computing shader takes the second difference value as the left shift number of the second texture data stored in the second target data unit.

15. The method of claim 11, wherein, The first texture data is a value of an RGBA channel, the second texture data is a value of an RGB channel, the preset type is a uint type, and the computing shader determines the first texture data of the third pixel point according to second texture data in the first target data unit, a right shift number corresponding to the first target data unit, second texture data in the second target data unit, and a left shift number corresponding to the second target data unit. The computing shader right shifts the second texture data in the first target data unit according to the right shift number corresponding to the first target data unit, to obtain a fifth uint value. The computing shader left shifts the second texture data in the second target data unit according to the left shift number corresponding to the second target data unit, to obtain a sixth uint value. The computing shader adds values on the same number of bits of the fifth uint value and the sixth uint value to obtain a seventh uint value, the seventh uint value including all values of RGB channels of the third pixel point; The computing shader converts the seventh uint value into values of RGBA channels and modifies a value of an A channel in the RGBA channels to 1 to obtain first texture data of the third pixel point.

16. The method of claim 15, wherein, The method further includes: When the right shift number of the second texture data in the first target data unit is 0, the computing shader takes an eighth uint value corresponding to the second texture data in the first target data unit as the seventh uint value.

17. An electronic device, comprising: The processor includes a first processor, a second processor and a third processor; the memory stores at least one program code; and the at least one program code is called and executed by the first processor, the second processor and the third processor to implement the image rendering method in any one of claims 1 to 16.

18. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one computer program, and the at least one computer program is executed by the first processor, the second processor and the third processor to implement the image rendering method in any one of claims 1 to 16.

19. A computer program product, characterised in that, The computer program product includes a computer program, and the computer program is executed by the first processor, the second processor and the third processor to implement the image rendering method in any one of claims 1 to 16.

Images