Image processor, image rendering method, and electronic device

Abstract

Embodiments of the present application provide an image processor, an image rendering method, and an electronic device, applicable to the technical field of image processing. The image processor comprises: an acquisition module, configured to acquire a plurality of fragments; a first rendering module, configured to execute first rendering processing on the plurality of fragments, wherein the first rendering processing comprises: determining visibility information of the plurality of fragments on the basis of depth information of the plurality of fragments, and updating the visibility information on the basis of a culling instruction in a rendering task, wherein the visibility information is used for indicating at least one of the following: at least one visible fragment among the plurality of fragments or a visible pixel of at least one visible fragment; and a second rendering module, configured to execute second rendering processing on the at least one visible fragment on the basis of the visibility information to obtain a first image. The embodiments of the present application implement identification and culling of invisible fragments and / or invisible pixels by means of the above operations, reduce the processing amount and power consumption of image rendering, and improve the image processing precision.

Description

Image processor, image rendering method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on November 29, 2023, with application number 202311625360.6 and application name “An image processor, image rendering method and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image rendering processing technology, and in particular to an image processor, an image rendering method, and an electronic device. Background Art

[0003] An electronic device may include an image processor. The image processor performs image tasks and renders and generates an image to be displayed. Image rendering is an image processing process that presents a three-dimensional image on a two-dimensional plane. In this image processing process, different imaging parts of a three-dimensional image are expressed in the form of multiple fragments, and the multiple fragments are pipelined to perform fragment drawing processing and shading processing to obtain the final imaging image on the two-dimensional imaging surface. However, multiple fragments have three-dimensional characteristics, so in the final imaging image, there will inevitably be invisible fragments that are blocked and cannot be visualized. These invisible fragments are useless fragments of the imaging image, and performing fragment drawing processing and shading processing on them will greatly increase the processing capacity of the chip system for image rendering, thereby increasing the latency and power consumption of image rendering.

[0004] To reduce the processing overhead of non-visible fragments, one existing image rendering technique proposes performing depth testing on the rendered fragments after each fragment rendering process to obtain depth information during the pipeline rendering and shading of multiple fragments. This depth information is used to indicate the occlusion relationships between the multiple fragments. During the fragment rendering process for each subsequent fragment, occluded fragments are culled based on the depth information to reduce the processing overhead associated with the culled fragments. However, the implementation of this method depends on the drawing order of the multiple fragments in the rendering task. If the occluded fragment is drawn before the occluding fragment, this method cannot avoid the associated processing overhead of the occluded fragment. Furthermore, occlusion relationships do not necessarily mean that the occluded fragment is invisible. Therefore, existing techniques cannot effectively identify and cull invisible fragments and may reduce image processing accuracy.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide an image processor, an image rendering method, and an electronic device, which effectively realize the identification and elimination of invisible fragments and / or invisible pixels, reduce the processing volume and power consumption of image rendering, and improve image processing accuracy.

[0007] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0008] In a first aspect, an image processor is provided, comprising: an acquisition module for acquiring a plurality of fragments; a first rendering module for performing a first rendering process on the plurality of fragments, the first rendering process comprising: determining visibility information of the plurality of fragments based on depth information of the plurality of fragments, and updating the visibility information based on a culling instruction in a rendering task, the visibility information being used to indicate at least one of the following: at least one visible fragment among the plurality of fragments or visible pixels of the at least one visible fragment; and a second rendering module for performing a second rendering process on the at least one visible fragment based on the visibility information to obtain a first image.

[0009] First, in the traditional image rendering process, fragment drawing, depth testing, and rendering are performed on multiple fragments in a pipelined manner. Therefore, the traditional depth test is related to the processing order of multiple fragments in the process of pipeline rendering multiple fragments. Under this processing method, the occlusion relationship obtained by the depth test also depends on the drawing order between the multiple fragments. In the embodiment of the present application, the rendering process is divided into two stages: a first rendering process and a second rendering process, wherein the first rendering process is a pre-rendering process and the second rendering process is the rendering process actually used to generate the first image. In the first rendering process, the visibility information of multiple fragments is determined based on the depth information. That is, before the actual rendering process of the second rendering process is performed, the depth test is performed on the multiple fragments according to the global rendering result. The visibility information obtained by this depth test depends on the depth information obtained by combining the results of multiple fragments. Because the result of combining multiple fragments represents the result after the fragments have been drawn, this depth information is accurate information obtained based on the occlusion relationship under the global drawing order, and is not the depth information obtained based on the pipeline processing in the traditional image rendering process that does not consider the drawing order.

[0010] Second, the depth test is based on the visible information obtained from the occlusion relationship, but the occlusion relationship does not fully represent the visibility of the fragment. In actual image rendering, there are fragments that are occluded but still visible (for example, fragments occluded by translucent fragments). In traditional image rendering, the visibility information obtained based on the depth test cannot fully represent the actual visibility of the fragment. In the shading processing stage of image rendering, a culling instruction is executed based on the rendering processing when the fragment is shaded, and invisible pixels are culled based on the culling instruction according to information such as the color parameters of the shading stage. Therefore, in an embodiment of the present application, a culling instruction can also be executed in the first rendering processing stage, and the fragment pixels that are occluded but can still be shaded and displayed are determined based on the running results of the culling instruction, and the visibility information obtained in the depth test stage of the first rendering processing is updated based on the running results. The updated visibility information includes the precise visible fragments and visible pixels involved in the rendering task. According to the visibility information, at least one visible fragment among multiple fragments and / or the visible pixels on at least one visible fragment can be determined.

[0011] Third, after obtaining accurate visibility information in the first rendering processing stage, a second rendering processing is performed based on the second rendering module. In the second rendering processing, only at least one visible fragment needs to be rendered based on the visibility information to obtain the first image. In the above method, the first rendering processing is equivalent to rendering preprocessing, which mainly involves the related processing of depth information and culling instructions, wherein the processing amount generated by running the related programs about depth information and culling instructions is relatively small. However, in the second rendering processing, based on the visibility information, the processing of invisible fragments and invisible pixels can be avoided, thereby greatly reducing the processing amount of image rendering, and will not cause misprocessing of visible pixels, and improve the image processing accuracy.

[0012] In one possible implementation, the first rendering process further includes: performing a first depth test on pixels of the plurality of fragments to obtain depth information. In an embodiment of the present application, the first depth test may be performed on pixels of the plurality of fragments to obtain depth information for each pixel. Visibility information of the plurality of fragments is determined based on the depth information of the plurality of fragments.

[0013] In one possible implementation, before performing the first depth test, the first rendering process also includes: performing rasterization processing on multiple fragments. In an embodiment of the present application, rasterization processing is first performed on multiple fragments, and the pixels on the imaging surface that are not related to fragment imaging are eliminated through rasterization processing to obtain retained pixels. The pixels retained after rasterization processing are pixels used to display fragment pixels, and each retained pixel may be related to one or more fragment pixels. Then, a first depth test is performed on these retained pixels. The depth information of each retained pixel can be determined based on the first depth test. The depth information can indicate the fragment pixel located at the top on the corresponding pixel.

[0014] In one possible implementation, updating visibility information based on a culling instruction in a rendering task includes: culling invisible pixels from one or more fragments of the plurality of fragments according to the culling instruction to obtain remaining pixels of the at least one fragment. Performing a second depth test on the remaining pixels to obtain updated depth information. Updating visibility information based on the updated depth information. In an embodiment of the present application, a culling instruction is provided in the pixel rendering program of the rendering task. The culling instruction is used to cull fragment pixels that do not require shading during the shading phase of image rendering. The culling instruction may originate from an application program. These fragment pixels that do not require shading are invisible pixels. Therefore, the invisible pixels of one or more fragments of the plurality of fragments are further culled according to the culling instruction to obtain remaining pixels of the at least one fragment. These remaining pixels are the visible pixels involved in the shading phase. Because some visible pixels that require shading are not necessarily the topmost fragment pixels on the corresponding pixel point, these non-topmost pixels to be shaded are visible pixels that are ignored during the first depth test. Performing a second depth test on the remaining pixels can identify these non-topmost pixels to be shaded. Updated depth information is obtained based on the second depth test, and visibility information is updated according to the updated depth information, so that the updated visibility information includes visible fragments and / or visible pixels involved in the processing flow of the rendering task.

[0015] In one possible implementation, performing a second rendering process on at least one visible fragment based on visibility information includes: obtaining at least one visible fragment from a plurality of fragments based on the visibility information. Performing the second rendering process on the at least one visible fragment. In an embodiment of the present application, when performing the second rendering process, the second rendering module can effectively determine which fragments from the plurality of fragments are visible fragments based on the updated visibility information. By obtaining at least one visible fragment from the plurality of fragments based on the visibility information, processing of invisible fragments can be avoided, thereby minimizing the amount of rendering processing.

[0016] In one possible implementation, the second rendering process includes: obtaining visible pixels of at least one visible fragment based on visibility information. Performing shading on the visible pixels of the at least one visible fragment. In an embodiment of the present application, when performing the second rendering process, the second rendering module can effectively determine the visible pixels in the visible fragment based on the updated visibility information. Obtaining visible pixels in the at least one visible fragment based on the visibility information can avoid processing invisible fragments, thereby minimizing the amount of rendering processing.

[0017] In one possible implementation, before the shading process, the second rendering process further includes: fragment drawing processing and rasterization processing. In an embodiment of the present application, in the fragment drawing processing stage of the second rendering process, at least one visible fragment can be obtained from multiple fragments based on visibility information, fragment drawing can be performed on at least one visible fragment, and rasterization processing can be performed after the fragment drawing process. Rasterization processing can eliminate pixels on the imaging surface that are not related to the fragments. In the shading processing stage after the rasterization process, visible pixels can be obtained from at least one visible fragment based on visibility information, and shading processing operations can be performed on the visible pixels. The above operations can reduce the processing amount of invisible fragments and invisible pixels.

[0018] For example, since the previous first rendering process includes visibility operations such as depth testing, it is no longer necessary to perform depth test-related operations in the second rendering process stage.

[0019] In one possible implementation, in the second rendering process, the processing order of at least one visible fragment is different from the acquisition order of multiple fragments. In an embodiment of the present application, since visibility information has been obtained in the first rendering process, the visibility information includes relevant information of visible fragments and / or visible pixels in the full processing stage of the rendering task. Therefore, when rendering at least one visible fragment in the second rendering processing stage, when determining the visible content to be rendered, it is not necessary to perform the rendering processing completely in the order in which the fragments are acquired. The processing order of at least one visible fragment can be adjusted according to actual design requirements and application requirements.

[0020] In one possible implementation, the image processor further includes a program generation module. The program generation module is used to crop the rendering task to obtain a cropped program related to visibility. The first rendering module is further used to perform a first rendering process on a plurality of fragments by running the cropped program. In an embodiment of the present application, the rendering task is used to instruct the rendering process of a plurality of fragments to obtain a first image. Therefore, during the rendering process, it is inevitable to process visible fragments and visible pixels. By cropping the rendering task, a cropped program related to visibility in the rendering task is obtained. The amount of running the cropped program is small, but it can achieve accurate acquisition of visibility information in the full processing stage of the rendering task. The first rendering module runs the cropped program to perform the first rendering process, which can achieve the acquisition of visibility information with a smaller amount of processing.

[0021] In one possible implementation, the acquisition module is further used to obtain a cropped program related to visibility from the rendering task. The first rendering module is further used to perform a first rendering process on a plurality of fragments by running the cropped program. In an embodiment of the present application, in some cases the cropped program for the rendering task has been stored offline, and there is no need for the image processor itself to generate the cropped program. At this time, the acquisition module can directly obtain the cropped program from the rendering task. Then, the first rendering module runs the cropped program to perform the first rendering process, which can achieve the acquisition of visibility information with a smaller amount of processing.

[0022] In a second aspect, an embodiment of the present application further provides an image rendering method, comprising: obtaining a plurality of fragments. Performing a first rendering process on the plurality of fragments, the first rendering process comprising: determining visibility information of the plurality of fragments based on depth information of the plurality of fragments, and updating the visibility information based on a culling instruction in a rendering task, wherein the visibility information is used to indicate at least one of the following: at least one visible fragment among the plurality of fragments or visible pixels of at least one visible fragment. Performing a second rendering process on the at least one visible fragment based on the visibility information to obtain a first image.

[0023] In a possible implementation, the first rendering process further includes: performing a first depth test on pixels of the plurality of fragments to obtain depth information.

[0024] In a possible implementation, before performing the first depth test, the first rendering process further includes: performing rasterization processing on the plurality of fragments.

[0025] In one possible implementation, updating the visibility information according to a culling instruction in the rendering task includes: culling invisible pixels of one or more fragments from the plurality of fragments according to the culling instruction to obtain remaining pixels of at least one fragment; performing a second depth test on the remaining pixels to obtain updated depth information; and updating the visibility information according to the updated depth information.

[0026] In a possible implementation, performing the second rendering process on the at least one visible fragment according to the visibility information includes: obtaining at least one visible fragment from a plurality of fragments according to the visibility information, and performing the second rendering process on the at least one visible fragment.

[0027] In a possible implementation, the second rendering process includes: obtaining visible pixels of at least one visible fragment according to the visibility information, and performing a shading process on the visible pixels of the at least one visible fragment.

[0028] In a possible implementation, before the shading process, the second rendering process further includes: fragment drawing process and rasterization process.

[0029] In a possible implementation, in the second rendering process, the processing order of the at least one visible fragment is different from the acquisition order of the multiple fragments.

[0030] In a possible implementation, the method further includes: clipping the rendering task to obtain a clipped program related to visibility, and executing the first rendering process on the plurality of fragments by running the clipped program.

[0031] In a possible implementation, the method further includes: obtaining a visibility-related clipping program from the rendering task, and executing the first rendering process on the plurality of fragments by running the clipping program.

[0032] In a third aspect, embodiments of the present application further provide an image rendering device, comprising an image processing circuit and a memory. The image processing circuit is coupled to the memory. The memory stores a rendering task for a first image, the rendering task including a culling instruction. The image processing circuit executes the image rendering method according to the second aspect and any other method described above based on the rendering task.

[0033] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising a circuit board and an image processor as described in the first aspect above. The image processor is disposed on the circuit board.

[0034] In a fifth aspect, embodiments of the present application further provide a computer-readable storage medium comprising instructions. When the instructions are executed on the image processor described in the first aspect, the image processor executes the image rendering method described in the second aspect and any other method.

[0035] In a sixth aspect, an embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to execute the image rendering method according to the second aspect and any other method described above.

[0036] Regarding the technical principles and beneficial effects of the embodiments of the second, third, fourth, fifth and sixth aspects, please refer to the relevant description of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG1 is a schematic diagram of rendering fragments in a visual coordinate system;

[0038] FIG2 is a schematic diagram of an electronic device provided in an embodiment of the present application;

[0039] FIG3 is a schematic diagram of a chip system provided in an embodiment of the present application;

[0040] FIG4 is a schematic diagram of a first image processor provided in an embodiment of the present application;

[0041] FIG5 is a schematic diagram of a second image processor provided in an embodiment of the present application;

[0042] FIG6 is a schematic diagram of different drawing orders for fragments in an occlusion relationship provided by an embodiment of the present application;

[0043] FIG7 is a schematic diagram of another second image processor provided in an embodiment of the present application;

[0044] FIG8 is a first schematic diagram of a third image processor provided in an embodiment of the present application;

[0045] FIG9 is a second schematic diagram of another third image processor provided in an embodiment of the present application;

[0046] FIG10 is a third schematic diagram of yet another third image processor provided in an embodiment of the present application;

[0047] FIG11 is a fourth schematic diagram of yet another third image processor provided in an embodiment of the present application;

[0048] FIG12 is a flowchart of an image rendering method provided in an embodiment of the present application;

[0049] FIG13 is a schematic diagram of a running program of a rendering process provided by an embodiment of the present application;

[0050] FIG14 is a second flowchart of another image rendering method provided by an embodiment of the present application;

[0051] FIG15 is a third flowchart of another image rendering method provided in an embodiment of the present application;

[0052] FIG16 is a schematic diagram of generating visibility information in a first rendering processing stage according to an embodiment of the present application;

[0053] FIG17 is a fourth flowchart of another image rendering method provided in an embodiment of the present application;

[0054] FIG18 is a schematic diagram of rendering pixels of multiple fragments in the same pixel block according to an embodiment of the present application;

[0055] FIG19 is a schematic diagram of eliminating invisible pixels based on visibility information according to an embodiment of the present application;

[0056] FIG20 is a fifth flowchart of another image rendering method provided in an embodiment of the present application;

[0057] FIG21 is a schematic diagram of rendering visible pixels in a second rendering processing stage provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] It should be noted that the terms "first", "second", etc. involved in the embodiments of the present application are only used to distinguish features of the same type and cannot be understood as indicating relative importance, quantity, order, etc.

[0059] The terms "exemplary" or "for example" in the embodiments of this application are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0060] The terms "coupling" and "connection" involved in the embodiments of this application should be understood in a broad sense. For example, they may refer to a physical direct connection, or an indirect connection achieved through electronic devices, such as a connection achieved through metal wires, resistors, inductors, capacitors or other electronic devices.

[0061] First, some basic concepts involved in the embodiments of this application are explained:

[0062] A fragment is the basic input unit for image rendering. Typically, a fragment can include a single point, a line segment, or a polygon. Polygonal fragments are mostly triangle fragments.

[0063] Image rendering refers to the image processor performing image rendering processing on multiple input fragments based on the rendering task. Image rendering can include fragment drawing processing, rasterization processing and shading processing. Among them, as shown in Figure 1, fragment drawing processing refers to drawing polygonal fragments at the target imaging position through matrix calculation based on rotation, translation and scaling, that is, drawing polygonal fragments based on a certain position, angle and size in the visual coordinate system. Rasterization processing refers to traversing pixel points within the viewfinder range of the visual coordinate system. If the pixel point is covered by the polygonal fragment, it is retained. If the pixel point is not covered by the polygonal fragment, it is discarded. Shading processing refers to lighting rendering of the pixel points retained by rasterization to generate corresponding colors. In the shading processing stage, a discard instruction will be run to eliminate invisible pixel parts based on the color information of multiple fragments.

[0064] Depth testing (z-test), as shown in Figure 1, refers to the coordinate reference system used in computer graphics, with the screen's top, bottom, left, and right axes as the y and x axes, and the direction perpendicular to the screen as the z axis. Multiple fragments represent three-dimensional image information on a two-dimensional imaging surface. Therefore, different fragments have different positions on the z axis. Therefore, z is also called the depth of the fragment. Depth testing involves detecting and comparing the depth of fragments.

[0065] An embodiment of the present application provides an electronic device, as shown in FIG2 , wherein the electronic device 1000 includes a circuit board (not shown in the figure) and a chip system 100. The chip system 100 is arranged on the circuit board. As shown in FIG3 , the chip system 100 includes an image processor 10 and a memory 20. The image processor 10 and the memory 20 are coupled via a bus BUS. The memory 20 stores a rendering task regarding a first image and a plurality of fragments regarding the first image. The rendering task is used to instruct the image processor 10 to obtain a plurality of fragments from the memory 20, and to render an image based on the plurality of fragments to obtain a first image. The rendering process includes fragment drawing process, rasterization process, and shading process, etc. The rendering task may include culling instructions. The memory 20 may include various types of volatile memory or non-volatile memory. The image processor 10 may include, but is not limited to, an image processing unit (GPU).

[0066] For example, as shown in FIG3 , the chip system 100 may further include a processor 30. The processor 30 is coupled to the memory 20 and the image processor 10 via a bus BUS. The processor 30 is configured to write the aforementioned rendering task into the memory 20. Optionally, the processor 30 can run an operating system program or an application program. The processor may include, but is not limited to, a central processing unit (CPU), a microcontroller, or a microprocessor. The rendering task may be generated by the application program.

[0067] Exemplarily, the electronic device 1000 may be an image processing device, a desktop computer, a laptop computer, a tablet computer, a smart phone, a television, a smart display, or other device with an image processing function or an image display function.

[0068] In some possible implementations, the image processor 10 described in FIG3 may be a first image processor that performs image rendering on multiple fragments based on a rendering task. As shown in FIG4 , the first image processor 10A includes a first acquisition module 11A and a full rendering module 12A. The first acquisition module 11A is used to acquire multiple fragments. The full rendering module 12A is used to perform fragment drawing processing, rasterization processing, and shading processing on each of the multiple fragments in a pipeline manner according to the acquisition order of the multiple fragments to render a first image. However, because the multiple fragments have three-dimensional characteristics, each frame of the first image is a display of a three-dimensional object on a two-dimensional plane. In the final first image, there will inevitably be invisible fragment pixels that are blocked and cannot be visually displayed. These invisible fragment pixels are useless fragment pixels with respect to the imaged image. In the first image processor 10A of the embodiment shown in FIG4 , when fragment drawing and shading are sequentially performed on multiple pixels of multiple input fragments, excessive invisible pixels are processed, significantly increasing the image rendering processing load, thereby increasing image rendering latency and power consumption. This high processing load and power consumption make it difficult to implement image rendering on a mobile electronic device 1000 while maintaining high image quality.

[0069] To reduce processor and power consumption associated with processing invisible fragment pixels, as in the embodiment of FIG. 4 , in some possible implementations, the image processor 10 shown in FIG. 3 may be a second image processor based on depth testing. As shown in FIG. 5 , the second image processor 10B includes a second acquisition module 11B and a test rendering module 12B. The second acquisition module 11B is configured to acquire multiple fragments. The test rendering module 12B is configured to pipeline fragment rendering, rasterization, depth testing, and shading for each of the multiple fragments, based on the order in which the multiple fragments were acquired, to render a first image. Specifically, first, during the fragment rendering process, the currently drawn fragment is determined to be occluded based on existing depth information. Occluded fragment pixels are then culled, and only the unoccluded fragment pixels are subjected to fragment rendering. The aforementioned depth information is used to indicate occlusion relationships between multiple fragments. Second, during the rasterization process, pixels on the imaging surface that are not imaged by fragment pixels are culled based on the already drawn fragment pixels, resulting in retained pixels. Each retained pixel may be associated with one or more fragment pixels. Finally, in the depth test process: after the rasterization process, the fragment occlusion relationship under the current fragment drawing process progress is determined, and the depth information is recorded and updated according to the currently determined fragment occlusion relationship. The updated depth information is used for subsequent fragment drawing processes. In the embodiment shown in Figure 5 of the present application, the occluded fragment pixels can be eliminated based on the existing depth information during the fragment drawing process to reduce the drawing process of this part of the fragment pixels. At this time, the processing of the drawn pixel part of the fragment drawing process in the subsequent shading process will also avoid the processing of the occluded fragment pixel part. Therefore, through the embodiment recorded in Figure 5, the processing amount of useless redundant fragments can be reduced to a certain extent, thereby reducing the power consumption of the chip system 100.

[0070] However, the implementation shown in Figure 5 depends on the order in which the fragments are obtained and drawn. As shown in Figure 6, take the drawing of fragment 1 (represented by a black triangle) and fragment 2 (represented by a white triangle) as an example, where fragment 2 is the fragment obscured by fragment 1. In Figure 6 (a), fragment 1 is drawn before fragment 2. At this time, after the fragment drawing process is performed on fragment 1, the depth information after drawing is updated. Then, when the fragment drawing process is performed on fragment 2, it can be determined based on the updated depth information that some fragment pixel portions of fragment 2 are obscured by fragment 1. Then, during the fragment drawing process of fragment 2, the obscured fragment pixel portions of fragment 2 can be eliminated based on the depth information. Under the fragment drawing order of Figure 6 (a), the elimination of the obscured fragment pixel portions in fragment 2 based on the depth information can be achieved normally. As shown in Figure 6 (b), fragment 2 is drawn before fragment 1. At this point, the existing depth information does not contain information about fragment 1. Therefore, when fragment 2 is rendered, the occlusion relationship between fragments 1 and 2 cannot be determined based on the existing depth information. Fragment 2 must be rendered in its entirety. After fragment 2 is rendered, the depth information is updated based on fragment 2. Only after fragment 2's fragment rendering and depth testing are completed can fragment 1 be rendered. During the fragment rendering process for fragment 1, it can be determined based on the depth information that fragment 1 occludes fragment 2, so fragment 1 must be rendered in its entirety. Based on the above analysis, the embodiment shown in Figure 5 has two problems: Problem 1: Although this method can cull occluded fragment pixels based on depth information, this approach is limited by the fragment rendering order and cannot effectively cull fragments in the rendering order. Problem 2: The occlusion relationship of fragments is not completely equal to the fragment visibility. In image rendering, some occluded fragments may also be visible fragments. If the obscured but still visible pixel parts of the fragment are eliminated during the fragment drawing process, the processing accuracy of the image rendering will be reduced.

[0071] To improve the fragment culling accuracy of the embodiment of FIG. 5 and to eliminate more occluded fragment pixel portions, in some possible implementations, as shown in FIG. 7 , the second image processor 10B may further include a delay buffer 13B. The delay buffer 13B caches n fragments that have undergone fragment rendering. When a new fragment completes fragment rendering and rasterization, one fragment is selected from the n fragments based on the depth information for subsequent shading, etc. After the new fragment is cached in the delay buffer 13B, the depth information is updated based on the fragments stored in the delay buffer 13B. In this implementation, the updated depth information can delay the occlusion relationship between the n fragments in the buffer 13B, allowing the occlusion relationship within the span of n fragments to be determined independently of the fragment rendering order. However, this implementation still presents the following problems: Problem 1: n is related to the storage space size of the delay buffer 13B, and therefore, the size of n is limited. If the span between two fragments with an occlusion relationship in the drawing order is greater than n, the solution described in the embodiment shown in FIG7 cannot effectively identify the occlusion relationship. Question 2: The solution of the embodiment shown in FIG7 still needs to perform a complete fragment drawing process on the fragments to obtain depth information that characterizes the occlusion relationship. This also means that it is unavoidable to perform a complete fragment drawing process on invisible fragments, and this part of the processing volume is still unavoidable. Question 3: The occlusion relationship of the fragments is not completely equal to the visibility of the fragments. In image rendering, some occluded fragments may also be visible fragments. If the occluded but still visible fragment pixel part is eliminated during the fragment drawing process, the processing accuracy of the image rendering will be reduced.

[0072] To further reduce image rendering processing complexity and power consumption and improve image processing accuracy, in some possible implementations, the image processor 10 depicted in FIG3 may be a third image processor that obtains visibility information based on a first rendering process (i.e., rendering pre-processing) and then performs a second rendering process on visible fragments based on the visibility information. In this solution, instead of performing a single rendering process as in FIG4, FIG5, or FIG7, two processes, rendering pre-processing and rendering process, may be performed. As shown in FIG8, the third image processor 10C includes: a third acquisition module 11C for acquiring multiple fragments; a first rendering module 12C for performing a first rendering process on the multiple fragments, similar to the rendering pre-processing. The first rendering process includes determining visibility information for the multiple fragments based on their depth information and updating the visibility information based on culling instructions in the rendering task. The visibility information indicates at least one of the following: at least one visible fragment from the multiple fragments or visible pixels of the at least one visible fragment. A second rendering module 13C for performing a second rendering process on the at least one visible fragment based on the visibility information to obtain a first image. The second rendering process is used to generate the first image.

[0073] Exemplarily, in the first rendering process performed by the first rendering module 12C, the operation of determining the visibility information of multiple fragments based on the depth information of the multiple fragments can be an operation related to pixel culling based on depth information in the fragment drawing process. In an embodiment of the present application, in the traditional complete fragment drawing process, the occlusion relationship between the currently drawn fragment and the previously drawn fragment is determined based on the depth information, and the visibility information of the fragment is determined based on the occlusion relationship. Then, based on the visibility information, the pixels of the determined invisible fragments are culled in the fragment drawing process stage. Therefore, the operation of the invisible fragment pixel culling part in the fragment drawing process can be performed in the first rendering process. This part of the operation only accounts for a very small part of the processing amount of the complete fragment drawing process, but the operation based on this small part is sufficient to obtain accurate visibility information about the fragment drawing process stage.

[0074] For example, in the first rendering process performed by the first rendering module 12C, the operation of updating visibility information based on the culling instructions in the rendering task can be a pixel culling-related operation performed based on the culling instructions in traditional shading processing. In the embodiments of the present application, the shading process accounts for the vast majority of the processing load during the entire image rendering process. However, the shading process also involves color-related visibility information. Therefore, obtaining color-related visibility information has always been a major challenge in image rendering. In actual applications, the shading process requires culling invisible pixels based on the culling instructions in the rendering task. After executing the culling instructions, fragment pixels that do not require shading are culled based on the fragment's color parameters, etc. These fragment pixels that do not require shading are considered the invisible pixels of the fragment. The execution of the culling instructions is relatively small, but they are related to the pixel color display of the processed fragment during the shading process. Therefore, visibility information related to color, etc., can be obtained during the execution of the culling instructions. The execution of the culling instructions only accounts for a small portion of the processing load of the entire shading process. Therefore, in the embodiment of the present application as shown in FIG8 , only part of the content related to the culling instruction can be executed during the complete shading process in the first rendering process, so that accurate visibility information about the shading processing stage can be obtained with a very small amount of processing.

[0075] The traditional image rendering process includes pipeline execution of fragment drawing processing, rasterization processing and shading processing, etc. for each fragment in a plurality of fragments. In the image rendering process, the visibility of the fragment is reflected based on two levels: Level one, whether the fragment is blocked by other fragments. Level two, whether the fragment needs to be shaded (that is, whether the fragment is blocked by transparent or translucent fragments). However, the above-mentioned embodiments of Figures 5 and 7 only involve obtaining visibility information based on occlusion relationships based on depth information, that is, the embodiments of Figures 5 and 7 only focus on the visibility of level one, and do not focus on the visibility of level two, and cannot effectively achieve the elimination of invisible fragments. In addition, in the embodiments of Figures 5 and 7, depth testing is a step in the actual image rendering process, that is, when processing multiple fragments in a pipeline manner, depth testing is performed on each fragment in turn. In the embodiment of FIG5 , the occlusion relationship obtained by each depth test depends on the order in which the fragments are drawn. In the embodiment of FIG7 , within the range of the storage space size n of the delay buffer 13B, the occlusion relationship obtained by the depth test is independent of the order in which the fragments are drawn. However, for two fragments whose fragment drawing order spans greater than n, the depth test result still needs to depend on the fragment drawing order. In the embodiment of the present application, the image rendering operation is divided into two rendering stages, specifically including a first rendering process as a pre-rendering process and a second rendering process for actually rendering the image to generate the first image. In the first rendering process as a pre-rendering process, processing is performed on multiple fragments only with respect to depth information to obtain visibility information regarding the occlusion relationship of the multiple fragments. That is, before the depth information is set to the actual image rendering process of the second rendering process, processing is performed on the multiple fragments with respect to depth information to obtain depth information for the multiple fragments. This depth information is based on the results of depth tests performed on the multiple fragments, which takes all fragments into consideration. Therefore, the depth information obtained in the first rendering processing stage indicates the accurate occlusion relationship of multiple fragments, which does not depend on the drawing order of the fragments. Compared with the embodiments of Figures 5 and 7, the depth information obtained by the first rendering processing is more accurate. In addition, the depth information can only be used to obtain visibility information about the occlusion relationship. However, the occlusion relationship does not completely represent the visibility of the fragment. In actual image rendering, there are fragments that are obscured but still visible (for example, fragments obscured by translucent fragments). In traditional image rendering, the visibility information obtained based on the depth test cannot fully represent the true fragment visibility, and it ignores the color-related visibility information during the shading process. In the shading processing stage of image rendering, a culling instruction will be executed based on the rendering process when the fragment is shading, and invisible pixels will be culled based on the culling instruction according to information such as the color parameters of the shading stage.Therefore, in the first rendering processing stage, the visibility information about shading in the shading stage can also be obtained according to the culling instruction, and the visibility information obtained based on the depth information is updated based on the visibility information of the shading. Through the above operations, the visibility information obtained in the first rendering processing stage can accurately and effectively confirm which fragments among the multiple fragments are visible fragments, and confirm which pixel parts of each visible fragment are visible pixels. Then, it is only necessary to perform a second rendering process (i.e., actual image rendering) on ​​at least one visible fragment according to the visibility information to obtain the first image. The embodiment of the present application can realize the processing of the determined visible parts in the fragment drawing processing stage and the shading processing stage, more accurately and effectively avoid the rendering processing of the invisible parts, greatly reduce the processing amount and power consumption of the image rendering, and at the same time will not reduce the processing accuracy of the image rendering.

[0076] In some possible implementations, the visibility information may include pixel visibility information and / or fragment visibility information. The fragment visibility information is used to indicate at least one visible fragment among a plurality of fragments. The pixel visibility information is used to indicate visible pixels of the at least one visible fragment.

[0077] In some possible implementations, as shown in Figure 9, the first rendering process includes rasterization processing and a first depth test performed in sequence. In an embodiment of the present application, performing rasterization processing in the first rendering process can eliminate pixels on the imaging surface that are not related to fragment pixels. A first depth test is performed on the retained pixels to obtain depth information of multiple fragments on the retained pixels. Regarding the technical principles and technical effects of rasterization processing, the first depth test and depth information, reference can be made to the relevant descriptions of rasterization, depth testing and depth information in the aforementioned embodiments, which will not be repeated here.

[0078] In some possible implementations, as shown in FIG9 , updating the visibility information according to the culling instruction in the rendering task as described in the embodiment of FIG8 includes: culling invisible pixels of one or more fragments among a plurality of fragments according to the culling instruction to obtain the remaining pixels of at least one fragment. Performing a second depth test on the remaining pixels to obtain updated depth information. Updating the visibility information according to the updated depth information. Exemplarily, the rendering task can be generated by the processor 30 in the embodiment shown in FIG3 , for example, by the processor 30 executing an application. The processor 30 is provided with an execution function for the culling instruction in the generated rendering task. Therefore, the culling instruction can be obtained by the processor 30 executing the application.

[0079] In some possible implementations, performing the second rendering process on the at least one visible fragment according to the visibility information includes: obtaining at least one visible fragment from a plurality of fragments according to the visibility information, and performing the second rendering process on the at least one visible fragment.

[0080] In some examples, the second rendering process includes: obtaining visible pixels of at least one visible fragment according to the visibility information, and performing a shading process, ie, a pixel rendering process, on the visible pixels of the at least one visible fragment.

[0081] Exemplarily, as shown in FIG9 , the second rendering process includes fragment drawing, rasterization, and shading. In an embodiment of the present application, the second rendering module 13C obtains multiple fragments and visibility information. Based on the visibility information, at least one visible fragment is obtained from the multiple fragments. That is, the second rendering process, such as fragment drawing, is performed on the at least one visible fragment based on the fragment visibility information in the visibility information, and subsequent rasterization and shading operations are performed based on the fragment visibility information. Specifically, the shading process performed on each visible fragment includes: obtaining visible pixels of the at least one visible fragment based on the pixel visibility information in the visibility information, and performing pixel shading on the visible pixels. In this embodiment, the fragment drawing process in the second rendering process only needs to perform drawing on the visible pixel portion of the visible fragment, and rasterization and shading are performed based on the drawn visible pixel portion. The second rendering process can be performed specifically on visible fragments, and this execution is performed at the pixel level. Therefore, the second rendering process can effectively avoid rendering the invisible pixel portion of the fragment during the entire rendering process, greatly reducing the processing load of image rendering.

[0082] In some possible implementations, the first rendering module 12C may store visibility information in different ways.

[0083] In some examples, the first rendering module 12C stores the generated visibility information in the memory 20. The second rendering module 13C obtains the visibility information from the memory 20.

[0084] In some examples, the first rendering module 12C stores the generated visibility information in a cache circuit inside the image processor 10. Exemplarily, the cache circuit may be a buffer or a register.

[0085] Exemplarily, the cache circuit includes a first cache circuit and a second cache circuit; the first cache circuit is used to store fragment visibility information, and the second cache circuit is used to store pixel visibility information.

[0086] Exemplarily, the second buffer circuit may include multiple buffers, each of which is used to store incompatible pixel visibility information. For example, when multiple visible fragments exist on the same pixel block, the multiple visible fragments on the same pixel block may have corresponding pixel visibility information, and this pixel visibility information may be incompatible. In this case, different buffers among the multiple buffers may be used to store the incompatible pixel visibility information.

[0087] Exemplarily, the multiple buffers of the second buffer circuit may include a depth buffer, where the depth buffer is a buffer for storing depth information.

[0088] In some possible implementations, as shown in FIG10 , the third image processor 10C further includes: a program generation module 14C for cropping the rendering task to obtain a cropped program related to visibility. The first rendering module 12C is configured to perform a first rendering process on a plurality of fragments by running the cropped program. In an embodiment of the present application, the purpose of the first rendering process is to determine the visibility information of a plurality of fragment designs in the rendering task, so that the second rendering process can perform image rendering with minimized processing based on the precisely determined visibility information. Therefore, the third image processor 10C can crop the rendering task to remove the portion of the program in the rendering task that is not related to visibility, thereby obtaining a cropped program related to visibility. The processing volume of the cropped program is very small, and the first rendering module 12C can execute the first rendering process with very little processing overhead by running the cropped program to obtain visibility information.

[0089] In some possible implementations, as shown in FIG11 , the third acquisition module 11C is further configured to obtain a cropped program related to visibility from the rendering task. The first rendering module 12C is configured to perform a first rendering process on a plurality of fragments by directly running the cropped program. In an embodiment of the present application, the cropped program may have been set in the rendering task, or it may have been previously cropped by the third image processor 10C based on the embodiment shown in FIG10 when previously processing the rendering task. At this point, the third image processor 10C may obtain the existing cropped program based on the rendering task, and the first rendering module 12C may run the cropped program to obtain visibility information.

[0090] Based on the third image processor 10C shown in FIG. 8 , FIG. 9 , FIG. 10 and FIG. 11 , the following image rendering method including steps S100 to S300 as shown in FIG. 12 may be executed:

[0091] S100: Acquire multiple fragments.

[0092] In some possible implementations, as shown in Figures 8, 9, 10, and 11, the processor 30 of the chip system 100 writes a rendering task and multiple fragments into the memory 20. The rendering task is used to instruct image rendering of multiple fragments to generate a first image. The third acquisition module 11CC of the third image processor 10C acquires the rendering task from the memory 20 and obtains multiple fragments based on the rendering task.

[0093] In some possible implementations, step S100 further includes: when acquiring multiple fragments, further obtaining a clipped program related to visibility based on the rendering task. In some examples, as shown in FIG10 , a program generation module 14C based on the third image processor 10C clips the rendering task to obtain a clipped program related to visibility. In some examples, as shown in FIG11 , a third acquisition module 11C based on the third image processor 10C obtains a clipped program related to visibility based on the rendering task.

[0094] In some examples, the running program of the rendering task can be cropped, and the running program remaining after the cropping is used as the cropped program. For example, as shown in Figure 13 (a), it is a schematic diagram of the running program included in the rendering task obtained by the image processor 10. The running program recorded in Figure 13 (a) includes a program that fully performs operations such as fragment drawing processing, depth testing, rasterization processing and shading processing. Among them, fragment drawing processing, etc. can be performed based on the fragment processing program (vertex shader). Shading processing is performed based on the pixel rendering program (fragment shader). In the process of executing the fragment processing program, the depth-related posture (position) information of the fragment, the first rendering parameter and the second rendering parameter are obtained. The first rendering parameter is a rendering parameter related to visibility (such as color and other information), and the second rendering parameter is a rendering parameter unrelated to visibility. A depth test is performed based on the posture information to obtain the depth information of the relevant fragment. The pixel rendering program executes a culling instruction based on the depth information and the first rendering parameter to cull the invisible pixel portion, and performs a shading process on the remaining pixel portion after culling based on the second rendering parameter to generate a first image. By cropping the running program shown in Figure 13 (a), the cropped program required for the first rendering processing stage can be obtained. As shown in Figure 13 (b), a schematic diagram of the running program included in the cropped program is shown. The running program portion related to visibility in the fragment processing program is the running program portion that generates the posture information and the first rendering parameter. Therefore, the fragment processing program in Figure 13 (a) can be cropped, retaining only the running program related to the generation and processing of the posture information and the first rendering parameter. The running program portion related to visibility in the pixel rendering program is the execution portion of the culling instruction. Therefore, the pixel rendering program in Figure 13 (a) can be cropped, retaining the running program portion related to the execution of the culling instruction. The cropped program shown in FIG. 13( b ) , ie, the running program related to the first rendering process, can be obtained through the above cropping operation, thereby simplifying the processing process.

[0095] In some examples, the running program in Figure 13 (a) can be used as the running program for the second rendering process, or the running program part related to depth testing, etc. in Figure 13 (a) can be cropped and eliminated to obtain the running program related to the second rendering process.

[0096] S200: Perform a first rendering process on a plurality of fragments to obtain visibility information.

[0097] In some possible embodiments, the first rendering process includes determining visibility information of multiple fragments based on depth information of the multiple fragments, and updating the visibility information based on a culling instruction in the rendering task, where the visibility information is used to indicate at least one of the following: at least one visible fragment among the multiple fragments or visible pixels of at least one visible fragment, i.e., the fragment visibility information and pixel visibility information recorded in the relevant embodiment of Figure 10.

[0098] For example, the first rendering module of the third image processor 10C in Figures 8, 9, 10 and 11 can perform a first rendering process based on the cropped program obtained by the third acquisition module 11C. Based on the cropped fragment processing program and depth test program in Figure 13 (b), visibility information of multiple fragments can be determined based on the depth information of multiple fragments. Based on the cropped pixel rendering program in Figure 13 (b), visibility information can be updated according to the culling instructions in the rendering task. For the relevant principles of the cropped program, please refer to the relevant description in the aforementioned step S100, which will not be repeated here.

[0099] In some possible implementations, step S200 may include the following sub-operations of step S210 to step S220 as shown in FIG14 :

[0100] S210: Determine visibility information of multiple fragments according to depth information of the multiple fragments.

[0101] Exemplarily, step S200 may include the following sub-operations of step S211 to step S214 as shown in FIG15 :

[0102] First, the following steps S211 to S213 are executed in a pipeline manner on multiple fragments to obtain depth information of the multiple fragments:

[0103] S211 : Generate posture information of the currently processed fragment based on the existing depth information.

[0104] For example, the cropped fragment processing program in the cropped program shown in FIG13(b) can be run to perform fragment drawing on the currently processed fragment based on the currently available depth information, and generate corresponding pose information. For example, while the cropped fragment processing program in the cropped program shown in FIG13(b) is run to generate pose information, the first rendering parameter can also be generated.

[0105] S212: Perform rasterization processing on the currently processed fragment.

[0106] S213: Perform a first depth test on the fragment pixels of the currently processed fragment based on the pose information of the currently processed fragment to obtain the depth information of the currently processed fragment. After recording the depth information of the currently processed fragment, the process returns to S211 to process the next fragment. The pose processing of the next fragment may refer to the depth information of the previously processed fragment.

[0107] In some possible implementations, as shown in Figures 9, 10, and 11, after rasterization, the first rendering module 12C may perform a first depth test on the pixels of the currently processed fragment based on the pose information to obtain depth information of the currently processed fragment. For example, the first rendering module 12C obtains the pose information based on the fragment processing procedure shown in Figure 13(b), performs the first depth test on the pixels of multiple fragments, and thereby obtains depth information of the currently processed fragment. Existing depth information may be updated based on the depth information of the currently processed fragment.

[0108] After the operations of steps S211 to S213 are executed on multiple fragments in a pipeline manner, the depth information of the multiple fragments is obtained, and then the sub-operation of step S214 is executed:

[0109] S214: Determine visibility information of the multiple fragments according to the depth information of the multiple fragments.

[0110] Exemplarily, as shown in FIG16 , take the sequential drawing of fragment 0, fragment 1, and fragment 2 as an example: First, based on the operation of step S211, fragment 0 is drawn to obtain the posture information of fragment 0. Fragment 0 is the first fragment drawn, so the depth information of the previously processed fragment is not recorded in the current depth information. After fragment 0 is rasterized in step S212, the operation of step S213 is performed according to the posture information of fragment 0 to obtain the depth information of fragment 0. Based on the operation of step S213, it is determined that none of the pixels of fragment 0 are blocked. Secondly, based on the operation of step S211, fragment 1 is drawn to obtain the posture information of fragment 1. It is confirmed through the depth information of the previous fragment 0 that fragment 1 blocks part of the pixels of fragment 0. After the fragment 1 is subjected to the rasterization processing of step S212, the operation of step S213 is performed according to the posture information of the fragment 1 to obtain the depth information of the fragment 1, and the existing depth information is updated. The updated depth information is used to indicate that none of the pixels of the fragment 1 are blocked, and some of the pixels of the fragment 0 are not blocked. Then, based on the operation of step S211, the fragment 2 is drawn to obtain the posture information of the fragment 2. Based on the depth information of the previous fragments 0 and 1, it can be determined that the fragment 2 blocks a part of the pixels of the fragment 1, and also blocks the remaining visible pixels of the fragment 0. After the fragment 2 is subjected to the rasterization processing of step S212, the depth information of the fragment 2 is obtained based on the operation of step S213, and the existing depth information is updated. The updated depth information is used to indicate that none of the pixels of the fragment 2 are blocked, some of the pixels of the fragment 1 are not blocked, and the pixels of the fragment 0 are completely blocked. After executing steps S211 to S213 for multiple fragments, in step S214, the occlusion relationship of each fragment pixel can be determined based on the depth information of the multiple fragments, thereby determining the invisible pixels and visible pixels in the multiple fragments based on the occlusion relationship. Thus, pixel visibility information for the multiple fragments can be obtained.

[0111] S220: Execute culling instructions in the rendering task on the plurality of fragments to update visibility information.

[0112] For example, the third image processor 10C shown in Figures 8, 9, 10, and 11 can execute culling instructions in a rendering task based on the first rendering module 12C to update visibility information. In some examples, the rendering task can be generated by the processor 30 in the embodiment shown in Figure 3. For example, applications can be running in the processor 30, and different applications can call different task interfaces in the processor 30 to generate corresponding rendering tasks.

[0113] Exemplarily, step S220 may include the following sub-operations of step S221 to step S223 as shown in FIG17 :

[0114] S221 , according to a culling instruction, culling invisible pixels of one or more fragments among a plurality of fragments to obtain remaining pixels of at least one fragment.

[0115] Exemplarily, after executing the cropped fragment processing program, depth test and rasterization processing in the running program as shown in Figure 13 (b) in step S210, the cropped pixel rendering program in the running program as shown in Figure 13 (b) can be executed in step S221, specifically: executing the culling instruction in the cropped pixel rendering program. In some examples, the culling instruction can be executed based on the depth information and the first rendering parameter obtained in step S210. During the execution of the culling instruction, pixels that do not require shading processing will be culled based on whether the fragment requires shading processing, etc. Because the visibility information in step S210 only considers the impact of occlusion relationship on visibility, and the obscured fragment pixels may also need to be colored (for example, the pixel is partially obscured by transparent pixels or semi-transparent pixels of other fragments), the visibility information obtained in step S210 can ignore the visibility of the obscured pixels. At this time, in step S220, the invisible pixels can be re-determined based on a culling instruction from an application, such as an application run by the processor 30, and the invisible pixels of one or more fragments in the multiple fragments can be culled to obtain the remaining pixels of at least one fragment.

[0116] S222: Perform a second depth test on the remaining pixels to obtain updated depth information.

[0117] For example, in the third image processor 10C shown in Figures 9, 10, and 11, the first rendering module 12C can perform a second depth test on the remaining pixels after executing the culling instructions in the shading processing stage corresponding to the pixel rendering program. The second depth test can obtain updated depth information. The updated depth information includes information related to the pixel portion of the fragment that is occluded but needs to be rendered.

[0118] S223: Update visibility information according to the updated depth information.

[0119] For example, in the third image processor 10C shown in Figures 9, 10, and 11, the first rendering module 12C can update the pixel visibility information obtained in step S210 based on the updated depth information. The updated pixel visibility information includes pixel-level visibility information during the full image rendering process.

[0120] Exemplarily, as shown in Figure 18 (a), on the same 2×2 pixel block (including pixel 0, pixel 1, pixel 2 and pixel 3), fragment 0 is drawn first, and then fragment 1 is drawn on the basis of fragment 0, and fragment 1 is non-transparent. During the first depth test of step S210, pixel 2 of fragment 0 is recorded in the visibility information as a visible pixel, pixel 0, pixel 1 and pixel 3 of fragment 0 are invisible pixels, and pixel 0, pixel 1 and pixel 3 of fragment 1 are visible pixels. When the culling instruction is executed in the first rendering process, because fragment 1 is non-transparent, it is determined that the pixels blocked by fragment 0 need to be shaded on pixel 2 in the final pixel block. During the second depth test of step S220, it is confirmed that pixel 2 of fragment 0 is recorded as a visible pixel, and pixel 0, pixel 1 and pixel 3 of fragment 1 are visible pixels. As shown in Figure 18(b), fragment 0 is opaque and fragment 1 is semi-transparent. When fragment 1 occludes fragment 0, the shading effects of both fragment 0 and fragment 1 need to be displayed on the pixel block at the same time. In step S210 of the first rendering process, the visibility information obtained based on the occlusion relationship of the first depth measurement records the pixels of fragment 0 as invisible pixels and the pixels of fragment 1 as visible pixels. In the second depth test of step S220 of the first rendering process, the visibility information can be updated based on the culling instruction, and the pixels of both fragment 0 and fragment 1 are recorded as visible pixels.

[0121] For example, because the pixel visibility information includes information recording whether each pixel of each fragment of the first image is visible, any fragment with at least one visible pixel is considered a visible fragment. Fragment visibility information can be obtained based on the pixel visibility information. Furthermore, the fragment visibility information can also record the visible pixel portion of each visible fragment.

[0122] S300: Perform a second rendering process on at least one visible fragment according to visibility information to obtain a first image.

[0123] In some possible implementations, the second rendering process includes obtaining visible pixels of at least one visible fragment according to the visibility information, and performing pixel rendering on the visible pixels of the at least one visible fragment.

[0124] In some possible implementations, in the second rendering process, the processing order of at least one visible fragment is different from the acquisition order of multiple fragments. In the embodiment shown in Figure 4 of the present application, the order of fragment drawing depends on the acquisition order of the fragments. In the embodiment shown in Figure 5 of the present application, the drawing order of the n fragments cached by the delay buffer 13B may be different from the acquisition order, but the drawing order of the fragments other than n also needs to depend on the acquisition order. In the embodiment shown in Figure 8 of the present application, very accurate visibility information has been obtained in the first rendering processing stage. Therefore, in the second rendering processing stage, there is no need to perform fragment rendering processing based on the acquisition order of multiple fragments. In actual applications, the processing order of visible fragments can be adjusted according to design requirements.

[0125] In some possible implementations, depth testing may not be performed in the second rendering process. In the embodiment of the present application, since very accurate visibility information has been obtained in the first rendering process, in the second rendering process, pixels that do not need to be processed can be directly culled during the rendering process based on the visibility information, without having to rely on depth testing to perform pixel culling in the second rendering process.

[0126] In some possible implementations, image rendering adheres to the principle of rendering only one fragment per pixel block. However, in the embodiment shown in FIG. 18 , a pixel block may contain pixel visibility information for multiple fragments. When pixel visibility information for multiple visible fragments corresponding to the same pixel block is stored in the same cache circuit, incompatibility may arise. Multiple incompatible pixel visibility information may be stored in different buffers within the second cache circuit of the third image processor 10C.

[0127] For example, as shown in FIG19 , rendering processing is performed on eight fragments, fragments 0 through 7, within a pixel block. As shown in FIG19 (a), fragments 0, 3, 5, 6, and 7 are transparent (or semi-transparent) fragments. Fragments 1, 2, and 4 are opaque fragments. When the first rendering module 12C performs the first rendering process, because fragment 4 is a non-transparent fragment and obscures fragments 0, 1, 2, and 3, the pixel visibility information of fragment 4 is stored in the first buffer 1 in the second cache circuit. Since fragments 5, 6, and 7 are incompatible with fragment 4, the pixel visibility information of fragments 5, 6, and 7 can be stored in the second buffer 2 of the second cache circuit. In some examples, the buffer storing depth information of the third image processor 10C can be reused as the second buffer 2. For example, because fragment 4 is the topmost fragment among multiple opaque fragments, and fragment 7 is the topmost fragment among multiple transparent (or semi-transparent) fragments, and fragment 7 covers fragment 4, only the depth information of fragment 7 can be stored in the second buffer 2 as a parameter representing pixel visibility information.

[0128] As shown in FIG19( b), when the second rendering module 13C performs the second rendering process, the corresponding pixel visibility information is read from the first buffer 1 and the second buffer 2 respectively. According to the pixel visibility information of fragment 4 obtained from the first buffer 1, it can be determined that the pixels of fragment 0, fragment 1, fragment 2 and fragment 3 are invisible pixels. Therefore, fragment 0, fragment 1, fragment 2 and fragment 3 can be directly culled. Then, based on the depth information of fragment 7 in the second buffer 2, it is confirmed that the pixels of fragment 5, fragment 6 and fragment 7 above fragment 4 are visible pixels, and then pixel rendering is implemented based on the common pixel 4, fragment 5, fragment 6 and fragment 7 in the shading processing stage.

[0129] In some possible implementations, in step S300, the following sub-operations of steps S310 to S330 shown in FIG20 may be performed on at least one fragment among the multiple fragments:

[0130] S310: Execute fragment rendering processing.

[0131] In the embodiments of the present application, since the fragment visibility information is very accurate, processing of invisible fragment pixels can be effectively avoided during the fragment rendering process. The technical principles of the fragment rendering process can be referred to the relevant description of the aforementioned embodiments and will not be repeated here.

[0132] S320: Execute rasterization processing.

[0133] S330: Execute coloring processing.

[0134] In the embodiment of the present application, the operation of shading based on the pixel shader is the operation with the largest processing load in the entire image rendering process. In the second rendering process, shading is performed only on visible pixels based on pixel visibility information, which can greatly reduce the execution processing load of the pixel shader.

[0135] For example, as shown in FIG21 , in the second rendering processing stage, actual image rendering processing is performed on the multiple fragments recorded in FIG16 based on the visibility information: First, fragment 0 is rendered. Based on the visibility information, it is determined that fragment 0 has no visible pixels. Therefore, fragment 0 is not processed in both the fragment drawing processing stage and the shading processing stage. Secondly, fragment 1 is rendered. Based on the visibility information, it is determined that some pixels of fragment 1 are visible pixels. Only the visible pixel portion of fragment 1 is drawn in the fragment drawing processing stage and the shading processing stage. Finally, fragment 2 is rendered. Based on the visibility information, it is determined that all pixels of fragment 2 are visible pixels. All pixels of fragment 2 are drawn in the fragment drawing processing stage and the shading processing stage. Compared with FIG16 , it can be seen that when rendering the same multiple fragments, the implementation method of FIG21 can greatly reduce the processing amount of image rendering.

[0136] In summary, in the embodiments of Figures 4 and 5 above, in the fragment drawing processing stage, it is necessary to rely on the drawing order of the fragments to obtain depth information, which makes it impossible to effectively obtain the visibility information of the fragments. In addition, in the shading processing stage of the embodiments of Figures 4 and 5, only the elimination of invisible pixels related to the depth information is focused on, and the elimination of invisible pixels related to the first rendering parameters is not focused on. Therefore, in the embodiments of Figures 4 and 5, a lot of visibility information is missing, and it is impossible to maximize the extraction of invisible pixels, nor is it possible to maximize the elimination of invisible pixels in different processing stages. In the embodiments of Figures 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 and 21 of the present application, the visibility information related to the depth information after the fragment drawing processing is obtained based on the first rendering processing, and the visibility information based on the elimination instruction in the shading processing stage is also obtained. Thus, the first rendering process not only obtains visibility information related to depth information, but also pays attention to visibility information related to culling instructions in the shading stage, thereby realizing the identification and confirmation of invisible pixel portions in multiple fragments of the rendering task. Executing the second rendering process based on the visibility information obtained in the first rendering process can achieve maximum pixel culling in the fragment drawing process stage and the shading process stage of the second rendering process. The first rendering process only includes processing procedures related to depth testing and the execution of culling instructions, and this part of the processing procedure only occupies a very small part of the image rendering program. Moreover, although the depth test processing is performed in the first rendering process, the depth test processing can also be reduced accordingly in the second rendering process. Therefore, compared with the processing reduction of the second rendering process, the increased processing volume in the first rendering process can be ignored. Compared with the solutions of Figures 4 and 5, the solution based on the third image processor 10C can reduce the processing volume of image rendering more.

[0137] An embodiment of the present application further provides an image rendering device, comprising an image processing circuit and a memory. The image processing circuit is coupled to the memory. The memory stores a rendering task for a first image, wherein the rendering task includes a culling instruction. The image processing circuit executes an image rendering method according to the above embodiment and any other manner (e.g., the image rendering method described in Figures 12, 14, 15, 17, and 20) based on the rendering task.

[0138] Exemplarily, the image rendering device may be an image processing chip, the graphics processing circuit may be a graphics processing portion of the image processing chip, and the memory may be a data storage or cache portion within the image processing chip.

[0139] Exemplarily, the image rendering device may include multiple chips, the graphics processing circuit is one or more image processing chips among the multiple chips, and the memory is a storage chip among the multiple chips.

[0140] An embodiment of the present application also provides a computer-readable storage medium, which includes instructions; when the instructions are executed on an image processor (for example, the third image processor 10C described in Figures 8, 9, 10, and 11 above), the image processor executes the image rendering method described in the above embodiments and other methods (for example, the image rendering method described in Figures 12, 14, 15, 17, and 20).

[0141] An embodiment of the present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the image rendering method described in the above embodiments and other methods (for example, the image rendering method described in Figures 12, 14, 15, 17, and 20).

[0142] The processor involved in the embodiments of the present application may be a chip. For example, it may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0143] The memory involved in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0144] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0145] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0146] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0148] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located on a single device or distributed across multiple devices. Some or all of the modules may be selected to achieve the purpose of this embodiment based on actual needs.

[0149] In addition, the functional modules in the various embodiments of the present application may be integrated into one device, or each module may exist physically separately, or two or more modules may be integrated into one device.

[0150] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using a software program, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0151] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image processor, characterized in that: The image processor comprises: An acquisition module, used to acquire multiple fragments; A first rendering module, configured to perform a first rendering process on the plurality of fragments, the first rendering process comprising: determining visibility information of the plurality of fragments according to depth information of the plurality of fragments, and updating the visibility information according to a culling instruction in a rendering task, the visibility information being used to indicate at least one of the following: at least one visible fragment among the plurality of fragments or visible pixels of the at least one visible fragment; The second rendering module is used to perform a second rendering process on the at least one visible fragment according to the visibility information to obtain a first image.

2. The image processor according to claim 1, characterized in that: The first rendering process further includes: A first depth test is performed on pixels of the plurality of fragments to obtain the depth information.

3. The image processor according to claim 2, characterized in that: Before performing the first depth test, the first rendering process further includes: A rasterization process is performed on the plurality of fragments.

4. The image processor according to any one of claims 1 to 3, characterized in that: The updating of the visibility information according to the culling instruction in the rendering task includes: Eliminate invisible pixels of one or more fragments among the plurality of fragments according to the culling instruction to obtain remaining pixels of at least one fragment; Performing a second depth test on the remaining pixels to obtain updated depth information; The visibility information is updated according to the updated depth information.

5. The image processor according to any one of claims 1 to 4, characterized in that: The performing a second rendering process on the at least one visible fragment according to the visibility information comprises: Acquire the at least one visible fragment from the plurality of fragments according to the visibility information; A second rendering process is performed on the at least one visible fragment.

6. The image processor according to any one of claims 1 to 5, characterized in that: The second rendering process includes: Acquire visible pixels of the at least one visible fragment according to the visibility information; The shading process is performed on visible pixels of the at least one visible fragment.

7. The image processor according to claim 6, characterized in that: Before the shading process, the second rendering process further includes: fragment drawing process and rasterization process.

8. The image processor according to any one of claims 1 to 7, characterized in that: In the second rendering process, a processing order of the at least one visible fragment is different from an acquisition order of the plurality of fragments.

9. The image processor according to any one of claims 1 to 8, characterized in that: The image processor also includes a program generation module; The program generation module is used to clip the rendering task to obtain a clipped program related to visibility; The first rendering module is further configured to execute the first rendering process on the plurality of fragments by running the clipped program.

10. The image processor according to any one of claims 1 to 8, characterized in that: The acquisition module is further used to acquire a clipped program related to visibility from the rendering task; The first rendering module is further configured to execute the first rendering process on the plurality of fragments by running the clipped program.

11. An image rendering method, characterized in that: The method comprises: Get multiple fragments; Performing a first rendering process on the plurality of fragments, the first rendering process comprising: determining visibility information of the plurality of fragments according to depth information of the plurality of fragments, and updating the visibility information according to a culling instruction in the rendering task, the visibility information being used to indicate at least one of the following: at least one visible fragment among the plurality of fragments or visible pixels of the at least one visible fragment; A second rendering process is performed on the at least one visible fragment according to the visibility information to obtain a first image.

12. The image rendering method according to claim 11, characterized in that: The first rendering process further includes: A first depth test is performed on pixels of the plurality of fragments to obtain the depth information.

13. The image rendering method according to claim 12, characterized in that: Before performing the first depth test, the first rendering process further includes: A rasterization process is performed on the plurality of fragments.

14. The image rendering method according to any one of claims 11 to 13, characterized in that: The updating of the visibility information according to the culling instruction in the rendering task includes: Eliminate invisible pixels of one or more fragments among the plurality of fragments according to the culling instruction to obtain remaining pixels of at least one fragment; Performing a second depth test on the remaining pixels to obtain updated depth information; The visibility information is updated according to the updated depth information.

15. The image rendering method according to any one of claims 11 to 14, characterized in that: The performing a second rendering process on the at least one visible fragment according to the visibility information comprises: Acquire the at least one visible fragment from the plurality of fragments according to the visibility information; A second rendering process is performed on the at least one visible fragment.

16. The image rendering method according to any one of claims 11 to 15, characterized in that: The second rendering process includes: Acquire visible pixels of the at least one visible fragment according to the visibility information; The shading process is performed on visible pixels of the at least one visible fragment.

17. The image rendering method according to claim 16, characterized in that: Before the shading process, the second rendering process further includes: fragment drawing process and rasterization process.

18. The image rendering method according to any one of claims 11 to 17, characterized in that: In the second rendering process, a processing order of the at least one visible fragment is different from an acquisition order of the plurality of fragments.

19. The image rendering method according to any one of claims 11 to 18, characterized in that: The method further comprises: Clipping the rendering task to obtain a clipped program related to visibility; The first rendering process is performed on the plurality of fragments by running the post-cropping program.

20. The image rendering method according to any one of claims 11 to 18, characterized in that: The method further comprises: Obtaining a visibility-related clipping program from the rendering task; The first rendering process is performed on the plurality of fragments by running the post-cropping program.

21. An image rendering device, characterized in that: The image rendering device includes an image processing circuit and a memory; the image processing circuit is coupled to the memory; the memory stores a rendering task related to a first image; the rendering task includes a culling instruction; the image processing circuit is used to execute the image rendering method as described in any one of claims 11-20.

22. An electronic device, characterized in that: It comprises a circuit board and an image processor as described in any one of claims 1 to 10; the image processor is arranged on the circuit board.

23. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes instructions; when the instructions are executed on an image processor, the image processor is caused to execute the image rendering method according to any one of claims 11 to 20.

Images