Image processor, image rendering method and electronic equipment

Abstract

The embodiment of the invention provides an image processor, an image rendering method and electronic equipment, and is applied to the technical field of image processing. The image processor comprises an acquisition module used for acquiring a plurality of fragments. The first rendering module is used for executing first rendering processing on the multiple fragments, and the first rendering processing comprises the steps that visibility information of the multiple fragments is determined according to the depth information of the multiple fragments, and the visibility information is updated according to a removing instruction in a rendering task, the visibility information is used for indicating at least one of the following items: at least one visible fragment in the plurality of fragments or visible pixels of the at least one visible fragment. And the second rendering module is used for executing second rendering processing on the at least one visible fragment according to the visibility information to obtain the first image. According to the embodiment of the invention, the recognition and elimination of the invisible fragments and / or the invisible pixels are realized through the operation, the processing amount and the power consumption of image rendering are reduced, and the image processing precision is improved.

Description

Technical Field

[0001] This application relates to the technical field of image rendering processing, and in particular, to an image processor, an image rendering method, and an electronic device. Background Art

[0002] An electronic device may include an image processor. By executing an image task through the image processor, an imaging image to be displayed is rendered. Image rendering is an image processing process of presenting a three-dimensional image on a two-dimensional plane. In this image processing process, different imaging parts of the three-dimensional image are expressed in the form of multiple fragments, and the final imaging image is obtained on the two-dimensional imaging surface by performing fragment drawing processing and shading processing on the multiple fragments in a pipelined manner. However, the multiple fragments have three-dimensional characteristics. Therefore, in the finally obtained imaging image, there will inevitably be invisible fragments that are blocked and cannot be visually displayed. These invisible fragments are useless fragments for the imaging image, and performing fragment drawing processing and shading processing on them will greatly increase the processing amount of the chip system for image rendering, thereby increasing the latency and power consumption of image rendering.

[0003] In order to reduce the processing amount of processing non-visible fragments, in an existing image rendering technology, it is proposed that during the process of performing fragment drawing processing and shading processing on multiple fragments in a pipelined manner, a depth test is performed on the drawn fragments after each fragment drawing processing to obtain depth information, and the depth information is used to indicate the occlusion relationship between the multiple fragments. During the process of performing fragment drawing processing on each subsequent fragment, the occluded fragments are culled based on the depth information to reduce the processing amount brought by the culled fragments. However, the implementation of this method depends on the drawing order between multiple fragments in the rendering task. If the occluded fragment is drawn before the fragment used for occlusion, this method cannot avoid the processing amount of the occluded fragment. At the same time, the occlusion relationship does not necessarily mean that the occluded fragment is an invisible fragment. Therefore, the prior art cannot effectively identify and cull invisible fragments, and may reduce the image processing accuracy. Summary of the Invention

[0004] Embodiments of this application provide an image processor, an image rendering method, and an electronic device, which effectively realize the identification and culling of invisible fragments and / or invisible pixels, reduce the processing amount and power consumption of image rendering, and improve the image processing accuracy.

[0005] To achieve the above object, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, an image processor is provided, which includes: an acquisition module for acquiring a plurality of fragments; a first rendering module for performing a first rendering process on the plurality of fragments, the first rendering process including determining visibility information of the plurality of fragments according to depth information of the plurality of fragments, and updating the visibility information according to a culling instruction in a rendering task, the visibility information being used to indicate at least one of the following: at least one visible fragment or visible pixels of at least one visible fragment among the plurality of fragments; and a second rendering module for performing a second rendering process on at least one visible fragment according to the visibility information to obtain a first image.

[0007] First, in the traditional image rendering process, fragment drawing processing, depth testing, and rendering processing are sequentially performed on a plurality of fragments in a pipelined manner. Therefore, traditional depth testing is related to the processing order of the plurality of fragments in the process of rendering a plurality of fragments in a pipelined manner. In this processing method, the occlusion relationship obtained by depth testing also depends on the drawing order among the plurality of fragments. In the embodiments of the present application, the rendering process is divided into two stages: a first rendering process and a second rendering process. The first rendering process is a pre-rendering process, and the second rendering process is the rendering process actually used to generate the first image. In the first rendering process, visibility information of the plurality of fragments is determined based on depth information, that is, before the actual rendering process of the second rendering process, depth testing is performed on the plurality of fragments according to the result of global drawing. The visibility information obtained by this depth testing depends on the depth information obtained from the result after combining the plurality of fragments. Since the result after combining the plurality of fragments represents the result after the fragments have been drawn, this depth information is accurate information obtained based on the occlusion relationship under the global drawing order, rather than the depth information without considering the drawing order obtained in the traditional image rendering process based on pipelined processing.

[0008] Second, the depth test is based on the visible information obtained from the occlusion relationship. However, the occlusion relationship does not fully represent the visibility of the fragments. In actual image rendering, there are fragments that are occluded but still visible (such as fragments occluded by semi-transparent fragments). In traditional image rendering, the visibility information obtained based on the depth test does not fully represent the true visibility of the fragments. In the shading processing stage of image rendering, culling instructions are executed based on the rendering processing during fragment shading. According to information such as color parameters in the shading stage, invisible pixels are culled based on the culling instructions. Therefore, in the embodiments of the present application, culling instructions can also be executed in the first rendering processing stage. According to the running result of the culling instructions, fragment pixels that are occluded but can still be shaded and displayed are determined, and the visibility information obtained in the depth test stage of the first rendering processing is updated based on the running result. The updated visibility information includes the exact visible fragments and visible pixels involved in the rendering task. At least one visible fragment and / or visible pixels on at least one visible fragment among multiple fragments can be determined according to the visibility information.

[0009] Third, after obtaining the exact visibility information in the first rendering processing stage, a second rendering process is performed based on the second rendering module. In the second rendering process, only at least one visible fragment needs to be rendered based on the visibility information to obtain a first image. In the above manner, the first rendering process is equivalent to a rendering preprocessing, mainly involving the processing related to depth information and culling instructions, and the processing amount generated by running the related programs on depth information and culling instructions is relatively small. However, in the second rendering process, based on the visibility information, the processing of invisible fragments and invisible pixels can be avoided, thereby greatly reducing the processing amount of image rendering, and there will be no misprocessing of visible pixels, and the image processing accuracy is improved.

[0010] In a possible implementation manner, the first rendering process further includes: performing a first depth test on the pixels of multiple fragments to obtain depth information. In the embodiments of the present application, a first depth test can be performed on the pixels of multiple fragments to obtain the depth information of each pixel. The visibility information of multiple fragments is determined based on the depth information of multiple fragments.

[0011] In a possible implementation, before performing the first depth test, the first rendering process further includes: performing rasterization processing on a plurality of fragments. In the embodiments of the present application, first, rasterization processing is performed on a plurality of fragments. Through rasterization processing, pixel points on the imaging surface that are irrelevant to fragment imaging are removed, and the remaining pixel points are obtained. The pixel points remaining after rasterization processing are the pixel points used to display fragment pixels, and each remaining pixel point may be related to one or more fragment pixels. Then, the first depth test is performed on these remaining pixel points. Based on the first depth test, the depth information of each remaining pixel point can be determined. This depth information can indicate the fragment pixel located at the topmost position on the corresponding pixel point.

[0012] In a possible implementation, updating the visibility information according to the culling instruction in the rendering task includes: culling invisible pixels of one or more fragments among the plurality of fragments according to the culling instruction to obtain the remaining pixels of at least one fragment. Performing a second depth test on the remaining pixels to obtain updated depth information. Updating the visibility information according to the updated depth information. In the embodiments of the present application, a culling instruction is set in the pixel rendering program of the rendering task, and the culling instruction is used to cull fragment pixels that do not need to be shaded during the shading processing stage of image rendering. The culling instruction may come from an application program. These fragment pixels that do not need to be shaded are invisible pixels. Therefore, according to the culling instruction, invisible pixels of one or more fragments among the plurality of fragments are further culled to obtain the remaining pixels of at least one fragment, and these remaining pixels are the visible pixels involved in the shading processing stage. Since some visible pixels that need to be shaded are not necessarily the fragment pixels located at the topmost position on the corresponding pixel point, these non-topmost pixels to be shaded are the visible pixels ignored during the first depth test processing. Performing a second depth test on the remaining pixels can identify these non-topmost pixels to be shaded. Updated depth information is obtained based on the second depth test. The visibility information is updated according to the updated depth information, so that the updated visibility information includes the visible fragments and / or visible pixels involved in the processing flow of the rendering task.

[0013] In a possible implementation, performing a second rendering process on at least one visible fragment according to the visibility information includes: obtaining at least one visible fragment from the plurality of fragments according to the visibility information. Performing a second rendering process on at least one visible fragment. In the embodiments of the present application, when the second rendering module performs the second rendering process, based on the updated visibility information, it can effectively determine which fragments among the plurality of fragments are visible fragments. Obtaining at least one visible fragment from the plurality of fragments according to the visibility information can avoid processing invisible fragments, so as to minimize the rendering processing amount.

[0014] In a possible implementation, the second rendering process includes: obtaining visible pixels of at least one visible fragment according to visibility information, and performing shading processing on the visible pixels of at least one visible fragment. In the embodiments of the present application, when the second rendering module performs the second rendering process, based on the updated visibility information, it can effectively determine the visible pixels in the visible fragments. Obtaining visible pixels in at least one visible fragment according to visibility information can avoid processing invisible fragments, so as to minimize the rendering processing volume.

[0015] In a possible implementation, before the shading processing, the second rendering process further includes: fragment drawing processing and rasterization processing. In the embodiments of the present application, in the fragment drawing processing stage of the second rendering process, at least one visible fragment can be obtained from multiple fragments based on the visibility information, fragment drawing is performed on at least one visible fragment, and rasterization processing is performed after the fragment drawing processing. The rasterization processing can cull pixel points on the imaging surface that are irrelevant to the fragments. In the shading processing stage after the rasterization processing, visible pixels can be obtained from at least one visible fragment based on the visibility information, and shading processing operations are performed on the visible pixels. Through the above operations, the processing volume of invisible fragments and invisible pixels can be reduced.

[0016] Exemplarily, since the previous first rendering process includes visibility operations such as depth testing, in the second rendering process stage, it is not necessary to perform depth test-related processing again.

[0017] In a possible implementation, in the second rendering process, the processing order of at least one visible fragment is different from the acquisition order of multiple fragments. In the embodiments of the present application, since the visibility information has been obtained in the first rendering process, and this visibility information includes information related to visible fragments and / or visible pixels in the entire processing stage of the rendering task. Therefore, when rendering at least one visible fragment in the second rendering process stage, when determining the visible content to be rendered, it is not necessary to perform the rendering process completely according to the acquisition order of the fragments. The processing order of at least one visible fragment can be adjusted according to actual design requirements and application requirements, etc.

[0018] In a possible implementation, the image processor further includes a program generation module. The program generation module is configured to crop the rendering task to obtain a cropped program related to visibility. The first rendering module is further configured to perform a first rendering process on multiple fragments by running the cropped program. In the embodiments of the present application, since the rendering task is used to indicate rendering multiple fragments to obtain a first image. Therefore, during the rendering process, it is inevitable to process visible fragments and visible pixels. By cropping the rendering task, a cropped program related to visibility in the rendering task is obtained. The amount of operation of the cropped program is small, but it can accurately obtain visibility information in all processing stages of the rendering task. The first rendering module runs the cropped program to perform the first rendering process, which can obtain visibility information with a small amount of processing.

[0019] In a possible implementation, the acquisition module is further configured to obtain a cropped program related to visibility from the rendering task. The first rendering module is further configured to perform a first rendering process on multiple fragments by running the cropped program. In the embodiments of the present application, in some cases, the cropped program for the rendering task has been stored offline, and there is no need for the image processor to generate the cropped program by itself. At this time, the acquisition module can directly obtain the cropped program from the rendering task. Then, the first rendering module runs the cropped program to perform the first rendering process, which can obtain visibility information with a small amount of processing.

[0020] In a second aspect, the embodiments of the present application further provide an image rendering method, which includes: obtaining multiple fragments; performing a first rendering process on the multiple fragments, where the first rendering process includes: determining visibility information of the multiple fragments according to depth information of the multiple fragments, and updating the visibility information according to a culling instruction in the rendering task, and the visibility information is used to indicate at least one of the following: at least one visible fragment or visible pixels of at least one visible fragment among the multiple fragments; performing a second rendering process on at least one visible fragment according to the visibility information to obtain a first image.

[0021] In a possible implementation, the first rendering process further includes: performing a first depth test on pixels of the multiple fragments to obtain depth information.

[0022] In a possible implementation, before performing the first depth test, the first rendering process further includes: performing rasterization processing on the multiple fragments.

[0023] In a possible implementation, updating the visibility information according to the culling instruction in the rendering task includes: culling the invisible pixels of one or more fragments among the multiple fragments according to the culling instruction to obtain the remaining pixels of at least one fragment. Performing a second depth test on the remaining pixels to obtain updated depth information. Updating the visibility information according to the updated depth information.

[0024] In a possible implementation, performing a second rendering process on at least one visible fragment according to the visibility information includes: obtaining at least one visible fragment among the multiple fragments according to the visibility information. Performing a second rendering process on at least one visible fragment.

[0025] In a possible implementation, the second rendering process includes: obtaining the visible pixels of at least one visible fragment according to the visibility information. Performing a shading process on the visible pixels of at least one visible fragment.

[0026] In a possible implementation, before the shading process, the second rendering process further includes: a fragment drawing process and a rasterization process.

[0027] In a possible implementation, in the second rendering process, the processing order of at least one visible fragment is different from the obtaining order of the multiple fragments.

[0028] In a possible implementation, the method further includes: cropping the rendering task to obtain a cropped program related to visibility. By running the cropped program, performing a first rendering process on the multiple fragments.

[0029] In a possible implementation, the method further includes: obtaining a cropped program related to visibility from the rendering task. By running the cropped program, performing a first rendering process on the multiple fragments.

[0030] In a third aspect, an embodiment of the present application further provides an image rendering device, which includes an image processing circuit and a memory. The image processing circuit is coupled to the memory. The memory stores a rendering task of a first image, and the rendering task includes a culling instruction. The image processing circuit executes the image rendering method in the second aspect and any other manner as described above based on the rendering task.

[0031] In a fourth aspect, an embodiment of the present application further provides an electronic device, which includes a circuit board and an image processor as described in the first aspect above. The image processor is disposed on the circuit board.

[0032] Fifth aspect, embodiments of the present application further provide a computer-readable storage medium, which includes instructions. When the instructions run on the image processor described in the first aspect above, the image processor is caused to execute the image rendering method in the second aspect and any other manner above.

[0033] Sixth aspect, embodiments of the present application further provide a computer program product. When the computer program product runs on a computer, the computer is caused to execute the image rendering method in the second aspect and any other manner above.

[0034] Regarding the technical principles and beneficial effects of the embodiments in the second, third, fourth, fifth, and sixth aspects above, reference may be made to the relevant descriptions in the first aspect above, and details are not elaborated herein. Description of the Drawings

[0035] Figure 1 It is a schematic diagram of rendering a fragment in a visual coordinate system;

[0036] Figure 2 It is a schematic diagram of an electronic device provided by an embodiment of the present application;

[0037] Figure 3 It is a schematic diagram of a chip system provided by an embodiment of the present application;

[0038] Figure 4 It is a schematic diagram of a first image processor provided by an embodiment of the present application;

[0039] Figure 5 It is a schematic diagram of a second image processor provided by an embodiment of the present application;

[0040] Figure 6 It is a schematic diagram of different drawing orders of fragments in an occlusion relationship provided by an embodiment of the present application;

[0041] Figure 7 It is a schematic diagram of another second image processor provided by an embodiment of the present application;

[0042] Figure 8 It is a schematic diagram of a third image processor provided by an embodiment of the present application Figure 1 ;

[0043] Figure 9 It is a schematic diagram of another third image processor provided by an embodiment of the present application Figure 2 ;

[0044] Figure 10 It is a schematic diagram of yet another third image processor provided by an embodiment of the present application Figure 3 ;

[0045] Figure 11 Another schematic diagram of the third image processor provided by the embodiment of the present application Figure 4 ;

[0046] Figure 12 Flowchart of an image rendering method provided by the embodiment of the present application Figure 1 ;

[0047] Figure 13 Schematic diagram of the running program of a rendering process provided by the embodiment of the present application;

[0048] Figure 14 Flowchart of another image rendering method provided by the embodiment of the present application Figure 2 ;

[0049] Figure 15 Flowchart of yet another image rendering method provided by the embodiment of the present application Figure 3 ;

[0050] Figure 16 Schematic diagram of generating visibility information in the first rendering process stage provided by the embodiment of the present application;

[0051] Figure 17 Flowchart of yet another image rendering method provided by the embodiment of the present application Figure 4 ;

[0052] Figure 18 Schematic diagram of rendering multiple fragments for the pixels of the same pixel block provided by the embodiment of the present application;

[0053] Figure 19 Schematic diagram of culling invisible pixels based on visibility information provided by the embodiment of the present application;

[0054] Figure 20 Flowchart of yet another image rendering method provided by the embodiment of the present application Figure 5 ;

[0055] Figure 21 Schematic diagram of rendering visible pixels in the second rendering process stage provided by the embodiment of the present application. Detailed implementation manners

[0056] It should be noted that the terms "first", "second", etc. involved in the embodiments of the present application are only used for the purpose of distinguishing the same type of features, and cannot be understood as indicating relative importance, quantity, order, etc.

[0057] The terms "exemplary" or "for example" and the like in the embodiments of the present application are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the related concepts in a specific way.

[0058] The terms "coupling" and "connection" involved in the embodiments of the present application should be understood in a broad sense. For example, they may refer to a direct physical connection, or an indirect connection achieved through electronic devices, such as a connection achieved through metal wires, resistors, inductors, capacitors or other electronic devices.

[0059] First, some basic concepts involved in the embodiments of the present application are explained:

[0060] Primitives are the basic input units for image rendering. Usually, a fragment can include a single point, a line segment, or a polygon. Polygonal fragments are mostly triangle fragments.

[0061] Image rendering refers to the image processor performing image rendering processing on multiple input fragments based on the rendering task. Image rendering can include fragment drawing processing, rasterization processing, and shading processing. Figure 1 As shown, the fragment drawing process refers to drawing polygonal fragments at the target imaging position through matrix calculation based on rotation, translation and scaling, that is, drawing polygonal fragments based on a certain position, angle and size in the visual coordinate system. Rasterization refers to traversing the pixel points within the viewfinder range of the visual coordinate system. If the pixel point is covered by the polygonal fragment, it is retained, and if the pixel point is not covered by the polygonal fragment, it is discarded. Shading refers to the lighting rendering of the pixel points retained by rasterization to generate the corresponding color. In the shading stage, a discard instruction will be run to eliminate the invisible pixel parts based on the color information of multiple fragments.

[0062] Depth test (z-test), such as Figure 1 As shown, it refers to the coordinate reference system displayed in computer graphics with the y and x axes on the top, bottom, left, and right of the screen, and the z axis perpendicular to the screen. Multiple fragments are the display of image information in three-dimensional space on a two-dimensional imaging surface. Therefore, different fragments have different positions on the z axis. Therefore, z is also called the depth of the fragment. The processing operations such as detecting and comparing the depth of the fragment are called depth testing.

[0063] The present application embodiment provides an electronic device, such as Figure 2As shown, the electronic device 1000 includes a circuit board (not shown in the figure) and a chip system 100. The chip system 100 is disposed on the circuit board. As Figure 3 shown, the chip system 100 includes an image processor 10 and a memory 20. The image processor 10 and the memory 20 are coupled via a bus BUS. Among them, the memory 20 stores a rendering task for a first image and a plurality of fragments of the first image. The rendering task is used to instruct the image processor 10 to obtain a plurality of fragments from the memory 20 and perform image rendering based on the plurality of fragments to obtain a first image. The rendering process includes fragment drawing processing, rasterization processing, shading processing, etc. The rendering task may include culling instructions. The memory 20 may include various types of volatile or non-volatile memories. The image processor 10 may include, but is not limited to, a graphics processing unit (GPU).

[0064] Exemplarily, as Figure 3 shown, the chip system 100 may further include a processor 30. The processor 30 is coupled to the memory 20 and the image processor 10 via a bus BUS. The processor 30 is used to write the above-mentioned rendering task into the memory 20. Optionally, the processor 30 can run an operating system program or an application program. The processor may include, but is not limited to, a central processing unit (CPU), a microcontroller, and a microprocessor. The rendering task may be generated by an application program.

[0065] Exemplarily, the electronic device 1000 may be an image processing device, a desktop computer, a laptop computer, a tablet computer, a smart phone, a television, a smart display, or other devices with image processing functions or image display functions.

[0066] In some possible implementation manners, Figure 3 the image processor 10 described in Figure 4 shown, the first image processor 10A includes a first acquisition module 11A and a full rendering module 12A. Among them, the first acquisition module 11A is used to acquire a plurality of fragments. The full rendering module 12A is used to sequentially perform fragment drawing processing, rasterization processing, shading processing, etc. on each of the plurality of fragments in a pipelined manner according to the acquisition order of the plurality of fragments to render a first image. However, because the plurality of fragments have three-dimensional characteristics, each frame of the first image formed is a display of three-dimensional things on a two-dimensional plane. In the finally obtained first image, there will inevitably be invisible fragment pixels that are blocked and cannot be visually displayed. These invisible fragment pixels are useless fragment pixels for the imaging image. In as Figure 4In the first image processor 10A described in the embodiment, when performing fragment drawing processing and shading rendering processing on multiple pixels of multiple input fragments in sequence, too many invisible pixels will be processed, greatly increasing the processing volume of image rendering, and further increasing the latency and power consumption of image rendering. At this time, the processing volume and power consumption are large, and it is difficult to perform image rendering on the mobile electronic device 1000 under the requirements of image quality.

[0067] In order to reduce Figure 4 the processor and power consumption caused by processing invisible fragment pixels in the embodiment, in some possible implementation manners, such as Figure 3 the image processor 10 shown can be a second image processor based on depth testing, as Figure 5 shown, the second image processor 10B includes a second acquisition module 11B and a test rendering module 12B. Among them, the second acquisition module 11B is used to acquire multiple fragments. The test rendering module 12B is used to perform fragment drawing processing, rasterization processing, depth testing, and shading processing on each of the multiple fragments in a pipelined manner according to the acquisition order of the multiple fragments, so as to render the first image. Specifically, first, in the fragment drawing processing, it is determined whether the currently drawn fragment is an occluded fragment according to the existing depth information, so as to cull the occluded fragment pixels, and only perform fragment drawing processing on the unoccluded fragment pixel part. The foregoing depth information is used to indicate the occlusion relationship between multiple fragments. Secondly, in the rasterization processing, the pixel points on the imaging surface that have no imaged fragment pixels are culled according to the drawn fragment pixels, and the remaining pixel points are obtained. Each remaining pixel point may be related to one or more fragment pixels. Finally, in the depth testing process: after the rasterization processing, the fragment occlusion relationship at the current fragment drawing processing progress is determined, the depth information is recorded and updated according to the currently determined fragment occlusion relationship, and the updated depth information is used when performing fragment drawing processing on subsequent fragments. In the embodiment Figure 5 shown in the present application, in the fragment drawing processing stage, the occluded fragment pixels can be culled based on the existing depth information to reduce the drawing processing of this part of the fragment pixels. At this time, when the subsequent shading processing processes the drawn pixel part of the fragment drawing processing, it will also avoid processing the occluded fragment pixel part. Therefore, through Figure 5 the embodiment described, the processing volume of useless redundant fragments can be reduced to a certain extent, thereby reducing the power consumption of the chip system 100.

[0068] However Figure 5 the shown implementation manner depends on the acquisition order and drawing order of the fragments. As Figure 6As shown, taking the rendering of fragment 1 (represented by a black triangle) and fragment 2 (represented by a white triangle) as an example, where fragment 2 is the fragment occluded by fragment 1. In Figure 6 In the (a) figure of Figure 6 , fragment 1 is rendered before fragment 2. At this time, after the fragment rendering process of fragment 1, the rendered depth information is updated. Then, when the fragment rendering process of fragment 2 is performed, according to the updated depth information, it can be determined that some fragment pixel parts of fragment 2 are occluded by fragment 1. Then, during the process of performing the fragment rendering process on fragment 2, the occluded fragment pixel parts of fragment 2 can be removed according to the depth information. In Figure 6 Under the fragment rendering order of the (a) figure of Figure 5 , the removal of the occluded fragment pixel parts in fragment 2 based on the depth information can be normally achieved. In the (b) figure of

[0069] , fragment 2 is rendered before fragment 1. At this time, the existing depth information does not record information about fragment 1. Therefore, when the fragment rendering process of fragment 2 is performed, the occlusion relationship between fragment 1 and fragment 2 cannot be determined based on the existing depth information, and only fragment 2 can be fully rendered. After fragment 2 is rendered, the depth information is updated based on fragment 2. After the fragment rendering process and depth test of fragment 2 are completed, the fragment rendering process of fragment 1 will be performed. During the process of performing the fragment rendering process on fragment 1, it can be determined that fragment 1 occludes fragment 2 according to the depth information, so fragment 1 needs to be fully rendered. According to the above analysis, Figure 5 There are two problems in the embodiment shown: Problem 1: Although this method can remove the occluded fragment pixel parts based on the depth information, this method is limited by the rendering order of the fragments and cannot effectively remove the fragments under the rendering order. Problem 2: The occlusion relationship of the fragments is not exactly equal to the visibility of the fragments. In image rendering, some occluded fragments may also be visible fragments. If the occluded but still visible fragment pixel parts are removed during the fragment rendering process, the processing accuracy of image rendering will be reduced.

[0069] To improve the Figure 5 fragment removal accuracy of the embodiment to remove more occluded fragment pixel parts, in some possible implementation manners, such as Figure 7As shown, the second image processor 10B may further include a delay buffer 13B. The delay buffer 13B caches n fragments that have undergone fragment drawing processing. When a new fragment has completed fragment drawing processing and rasterization processing, one fragment is selected from the n fragments according to the depth information for subsequent shading processing and the like. After the new fragment is cached in the delay buffer 13B, the depth information is updated based on the fragments stored in the delay buffer 13B. In this embodiment, the updated depth information can delay the occlusion relationship among the n fragments in the delay buffer 13B, so that the occlusion relationship within the span of n fragments can be determined regardless of the fragment drawing order. However, in this embodiment, there are still the following problems: Problem 1: n is related to the storage space size of the delay buffer 13B. Therefore, the size of n is limited. If the span of the drawing order of two occluded fragments is greater than n, the solution described in the embodiment shown in Figure 7 cannot effectively identify the occlusion relationship. Problem 2: In the solution of the embodiment shown in Figure 7 , it is still necessary to perform complete fragment drawing processing on the fragments to obtain the depth information representing the occlusion relationship. This also means that it is impossible to avoid performing complete fragment drawing processing on invisible fragments, and this part of the processing amount cannot be avoided. Problem 3: The occlusion relationship of fragments is not exactly equal to the visibility of fragments. In image rendering, some occluded fragments may also be visible fragments. If the pixel parts of the occluded but still visible fragments are culled during the fragment drawing processing, the processing accuracy of image rendering will be reduced.

[0070] In order to further reduce the processing amount and power consumption of image rendering and improve the image processing accuracy, in some possible embodiments, Figure 3 the image processor 10 described in may be a third image processor that obtains visibility information based on the first rendering process, that is, rendering preprocessing, and performs a second rendering process on visible fragments based on the visibility information. In this solution, unlike Figure 4 , Figure 5 or Figure 7 , only one rendering process is performed. Instead, two processes of rendering preprocessing and rendering processing can be performed, as shown in Figure 8As shown, the third image processor 10C includes: a third acquisition module 11C for acquiring a plurality of fragments. A first rendering module 12C for performing a first rendering process on the plurality of fragments, similar to rendering preprocessing. The first rendering process includes: determining visibility information of the plurality of fragments according to the depth information of the plurality of fragments, and updating the visibility information according to the culling instruction in the rendering task, where the visibility information is used to indicate at least one of the following: at least one visible fragment or visible pixels of at least one visible fragment among the plurality of fragments. A second rendering module 13C for performing a second rendering process on at least one visible fragment according to the visibility information to obtain a first image, and the second rendering process is used for generating the first image.

[0071] Exemplarily, in the first rendering process performed by the first rendering module 12C, the operation of determining the visibility information of the plurality of fragments according to the depth information of the plurality of fragments may be an operation related to pixel culling based on the depth information in the fragment drawing process. In the embodiments of the present application, in the traditional complete fragment drawing process, the occlusion relationship between the currently drawn fragment and the previously drawn fragments is determined according to the depth information, and the visibility information of the fragment is determined according to the occlusion relationship. Then, based on the visibility information, the determined invisible fragments are pixel-culled in the fragment drawing process stage. Therefore, the operation of the pixel culling part of the invisible fragments in the fragment drawing process can be performed in the first rendering process, and this part of the operation only accounts for a very small part of the complete fragment drawing process, but based on this small part of the operation, it is sufficient to obtain accurate visibility information about the fragment drawing process stage.

[0072] Exemplarily, in the first rendering process performed by the first rendering module 12C, the operation of updating the visibility information according to the culling instruction in the rendering task may be an operation related to pixel culling based on the culling instruction in the traditional shading process. In the embodiments of the present application, in the complete process of image rendering, the shading process occupies the vast majority of the processing volume. However, there is also visibility information related to colors, etc. in the shading process. Therefore, how to obtain the visibility information related to colors has always been a major problem in image rendering. In practical applications, the shading process needs to perform culling processing on the invisible pixel part based on the culling instruction in the rendering task. After the culling instruction is executed, the fragment pixels that do not need to be shaded are culled based on the color parameters of the fragment, etc., and these fragment pixels that do not need to be shaded are the invisible pixel part of the fragment. The running program volume of the culling instruction is small, but it is related to the pixel color display of the fragments to be processed in the shading process stage. Therefore, the visibility information about colors, etc. can be obtained during the execution of the culling instruction. And the execution of the culling instruction only occupies a very small part of the complete shading process. Therefore, in the present application, such as Figure 8In the illustrated embodiment, during the first rendering process, only the part related to the culling instruction can be executed during the process of performing the complete shading process, so that accurate visibility information about the shading process stage can be obtained with a very small amount of processing.

[0073] The traditional image rendering process includes sequentially performing fragment drawing processing, rasterization processing, shading processing, etc. on each of multiple fragments in a pipelined manner. In image rendering processing, the visibility of a fragment is reflected at two levels: Level 1, whether the fragment is occluded by other fragments. Level 2, whether the fragment needs to be shaded (i.e., whether the fragment is occluded by a transparent or semi-transparent fragment). However, the above Figure 5 and Figure 7 embodiments only involve obtaining visibility information based on occlusion relationships according to depth information, that is, Figure 5 and Figure 7 the embodiments only focus on the visibility of Level 1 and do not pay attention to the visibility of Level 2, and cannot effectively implement the culling of invisible fragments. In addition, in Figure 5 and Figure 7 the embodiments, depth testing is a step in the actual image rendering process, that is, when processing multiple fragments in a pipelined manner, depth testing is sequentially performed on each fragment. Then in Figure 5 the embodiments, the occlusion relationship obtained by each depth test depends on the drawing order of the fragments; in Figure 7 the embodiments, within the storage space size n of the latency buffer 13B, the occlusion relationship obtained by depth testing does not depend on the drawing order of the fragments, but for two fragments with a drawing order span greater than n, the result of depth testing still needs to depend on the drawing order of the fragments. In the embodiment of the present application, the image rendering operation is divided into two rendering stages, specifically including: the first rendering process as a pre-rendering process, and the second rendering process for actually performing image rendering to generate the first image. In the first rendering process as a pre-rendering process, only the processing related to depth information is performed on multiple fragments to obtain visibility information about the occlusion relationships of multiple fragments. That is, before setting the depth information to the actual image rendering process for performing the second rendering process, the processing related to depth information is performed on multiple fragments, so as to obtain the depth information of multiple fragments. This depth information is based on the results after depth testing is respectively performed on all fragments, and all fragments are taken into consideration. Therefore, the depth information obtained in the first rendering process stage indicates the accurate occlusion relationships of multiple fragments, and it does not depend on the drawing order of the fragments. Compared with Figure 5 and Figure 7In the embodiment, the depth information obtained by the first rendering process is more accurate. In addition, the depth information can only be used to obtain visibility information about occlusion relationships. However, the occlusion relationship does not fully represent the visibility of the fragment. In actual image rendering, there are fragments that are occluded but still visible (such as fragments occluded by translucent fragments). In traditional image rendering, the visibility information obtained based on depth testing does not fully represent the true fragment visibility, ignoring the color-related visibility information during the shading process. During the shading process of image rendering, culling instructions are executed based on the rendering process during fragment shading, and invisible pixels are culled according to information such as color parameters in the shading stage. Therefore, in the first rendering process stage, the visibility information about shading in the shading stage can also be obtained according to the culling instructions, and the visibility information obtained based on the depth information is updated based on the visibility information about shading. Through the above operations, the visibility information obtained in the first rendering process stage can accurately and effectively confirm which fragments among multiple fragments are visible fragments, and confirm which pixel parts of each visible fragment are visible pixels. Then, only the second rendering process (i.e., actual image rendering) needs to be performed on at least one visible fragment according to the visibility information to obtain the first image. The embodiments of the present application can achieve processing on the determined visible parts in the fragment drawing process stage and the shading process stage, more accurately and effectively avoiding the rendering process of invisible parts, greatly reducing the processing volume and power consumption of image rendering, and at the same time not reducing the processing accuracy of image rendering.

[0074] In some possible implementation manners, the visibility information may include pixel visibility information and / or fragment visibility information. The fragment visibility information is used to indicate at least one visible fragment among multiple fragments. The pixel visibility information is used to indicate the visible pixels of at least one visible fragment.

[0075] In some possible implementation manners, as Figure 9 shown, the first rendering process includes a rasterization process and a first depth test executed in sequence. In the embodiments of the present application, performing the rasterization process in the first rendering process can cull the pixel points on the imaging surface that are irrelevant to the fragment pixels. The first depth test is performed on the retained pixel points to obtain the depth information of multiple fragments at the retained pixel points. For the technical principles and technical effects of the rasterization process, the first depth test, and the depth information, reference can be made to the relevant descriptions of rasterization, depth test, and depth information in the foregoing embodiments, which will not be elaborated here.

[0076] In some possible implementation manners, as Figure 9 shown, the above Figure 8Updating visibility information according to the culling instruction in the rendering task described in the embodiment includes: culling invisible pixels of one or more fragments among multiple fragments according to the culling instruction to obtain remaining pixels of at least one fragment. Performing a second depth test on the remaining pixels to obtain updated depth information. Updating the visibility information according to the updated depth information. Exemplarily, the rendering task may be generated by the Figure 3 processor 30 in the illustrated embodiment, for example, by the processor 30 executing an application to generate. The processor 30 is provided with an execution function regarding the culling instruction in the generated rendering task. Therefore, the culling instruction may be obtained from the processor 30 executing the application.

[0077] In some possible implementation manners, the above-mentioned performing a second rendering process on at least one visible fragment according to the visibility information includes: obtaining at least one visible fragment among multiple fragments according to the visibility information. Performing a second rendering process on at least one visible fragment.

[0078] In some examples, the second rendering process includes: obtaining visible pixels of at least one visible fragment according to the visibility information. Performing a shading process on the visible pixels of at least one visible fragment, that is, a pixel rendering process.

[0079] Exemplarily, as Figure 9 shown, the second rendering process includes: a fragment drawing process, a rasterization process, and a shading process. In the embodiment of the present application, the second rendering module 13C obtains multiple fragments and visibility information. Obtaining at least one visible fragment among multiple fragments according to the visibility information, that is, performing a second rendering process on at least one visible fragment according to the fragment visibility information in the visibility information, such as fragment drawing, and performing subsequent rasterization and shading process operations based thereon. Specifically, the shading process performed on each visible fragment includes: obtaining visible pixels of at least one visible fragment according to the pixel visibility information in the visibility information, and performing pixel shading rendering on the visible pixels. In this implementation manner, the fragment drawing process in the second rendering process only needs to perform drawing on the visible pixel part of the visible fragment, and perform rasterization and shading processes based on the drawn visible pixel part. The processing of the second rendering process can be targeted at fragments with visibility, and this execution is at the pixel-level granularity. Therefore, the second rendering process can effectively avoid the rendering process of invisible fragment pixel parts during the entire rendering process, greatly reducing the processing amount of image rendering.

[0080] In some possible implementation manners, the first rendering module 12C may store the visibility information based on different methods.

[0081] In some examples, the first rendering module 12C stores the generated visibility information in the memory 20. The second rendering module 13C obtains the visibility information from the memory 20.

[0082] In some examples, the first rendering module 12C stores the generated visibility information in a cache circuit inside the image processor 10. Exemplarily, the cache circuit can be a buffer or a register.

[0083] Exemplarily, the cache circuit includes a first cache circuit and a second cache circuit; the first cache circuit is used to store fragment visibility information. The second cache circuit is used to store pixel visibility information.

[0084] Exemplarily, the second cache circuit can include multiple buffers, and the multiple buffers are used to store incompatible pixel visibility information. For example, when there are multiple visible fragments on the same pixel block, there may be corresponding pixel visibility information for the multiple visible fragments on the same pixel block, and there may be an incompatibility problem between these pixel visibility information. At this time, different buffers in the multiple buffers can be used to store the incompatible pixel visibility information.

[0085] Exemplarily, the multiple buffers of the second cache circuit can include a depth buffer, and the depth buffer is a buffer used to store depth information.

[0086] In some possible implementation manners, as Figure 10 shown, the third image processor 10C further includes: a program generation module 14C, configured to crop a rendering task to obtain a cropped program related to visibility. A first rendering module 12C, configured to perform a first rendering process on multiple fragments by running the cropped program. In the embodiments of the present application, the purpose of the first rendering process is to determine the visibility information designed for multiple fragments in the rendering task, so that the second rendering process can perform image rendering with a minimized processing amount based on the accurately determined visibility information. Therefore, the third image processor 10C can crop the rendering task to remove the program parts unrelated to visibility in the rendering task, so as to obtain a cropped program related to visibility. The processing amount of the cropped program is very small, and the first rendering module 12C can run the cropped program to execute the first rendering process with a very small processing amount overhead to obtain the visibility information.

[0087] In some possible implementation manners, as Figure 11As shown, the third acquisition module 11C is further configured to acquire a cropped program related to visibility from the rendering task. The first rendering module 12C is configured to perform a first rendering process on multiple fragments by directly running the cropped program. In the embodiments of the present application, the cropped program may be already set in the rendering task, or may be pre-cropped by the third image processor 10C based on the Figure 10 embodiment shown in advance. At this time, the third image processor 10C may acquire the existing cropped program based on the rendering task, and the first rendering module 12C runs the cropped program to obtain visibility information.

[0088] Based on the above Figure 8 、 Figure 9 、 Figure 10 and Figure 11 shown third image processor 10C, the following image rendering method including steps S100 - S300 as shown in Figure 12 can be executed:

[0089] S100. Acquire multiple fragments.

[0090] In some possible implementation manners, as shown in Figure 8 、 Figure 9 、 Figure 10 and Figure 11 shown, the processor 30 of the chip system 100 writes the rendering task and multiple fragments into the memory 20. The rendering task is used to indicate image rendering of multiple fragments to generate a first image. The third acquisition module 11C of the third image processor 10C acquires the rendering task from the memory 20 and acquires multiple fragments according to the rendering task.

[0091] In some possible implementation manners, step S100 further includes: when acquiring multiple fragments, also obtaining a cropped program related to visibility based on the rendering task. In some examples, as shown in Figure 10 shown, the program generation module 14C of the third image processor 10C crops the rendering task to obtain a cropped program related to visibility. In some examples, as shown in Figure 11 shown, the third acquisition module 11C of the third image processor 10C acquires a cropped program related to visibility based on the rendering task.

[0092] In some examples, the running program of the rendering task can be cropped, and the remaining running program after cropping is used as the cropped program. Exemplarily, as shown in Figure 13 the (a) figure of, it is a schematic diagram of the running program included in the rendering task acquired by an image processor 10. In Figure 13The running program recorded in Figure (a) includes programs that fully execute operations such as fragment drawing processing, depth testing, rasterization processing and shading processing. Among them, fragment drawing processing and the like can be performed based on the fragment processing program (vertexshader). Shading processing is performed based on the pixel rendering program (fragment shader). In the process of executing the fragment processing program, depth-related posture (position) information of the fragment, the first rendering parameter and the second rendering parameter are obtained. The first rendering parameter is a rendering parameter related to visibility (such as color and other information), and the second rendering parameter is a rendering parameter unrelated to visibility. A depth test is performed based on the posture information to obtain the depth information of the relevant fragment. The pixel rendering program executes a culling instruction based on the depth information and the first rendering parameter to cull the invisible pixel portion, and performs shading processing on the remaining pixel portion after culling based on the second rendering parameter to generate a first image. Figure 13 By cutting the running program shown in Figure (a), we can get the cut program required for the first rendering processing stage. Figure 13 As shown in FIG. (b), it is a schematic diagram of the running program included in the program after clipping. The running program part related to visibility in the fragment processing program is the running program part that generates posture information and the first rendering parameter. Figure 13 The fragment processing program in Figure (a) is cut, and only the running program related to the generation and processing of the posture information and the first rendering parameter is retained. The running program part related to visibility in the pixel rendering program is the execution part of the culling instruction. Figure 13 The pixel rendering program in Figure (a) is cropped, and the running program part related to the execution of the culling instruction is retained. Through the above cropping operation, we can get Figure 13 The clipped program recorded in (b), that is, the running program related to the first rendering process, simplifies the processing process.

[0093] In some examples, you can Figure 13 The running program of (a) of FIG. 1 can be used as the running program of the second rendering process, or the running program of (a) of FIG. Figure 13 The running program part related to the depth test in Figure (a) is clipped and eliminated to obtain the running program related to the second rendering processing.

[0094] S200: Perform a first rendering process on a plurality of fragments to obtain visibility information.

[0095] In some possible implementations, the first rendering process includes determining visibility information of the plurality of fragments according to depth information of the plurality of fragments, and updating the visibility information according to a culling instruction in the rendering task, wherein the visibility information is used to indicate at least one of the following: at least one visible fragment or a visible pixel of at least one visible fragment among the plurality of fragments, that is,Figure 10 The fragment visibility information and pixel visibility information recorded in the related embodiments.

[0096] Exemplarily, such as Figure 8 , Figure 9 , Figure 10 and Figure 11 in the first rendering module of the third image processor 10C in Figure 13 can perform the first rendering process based on the cropped program obtained by the third acquisition module 11C. Based on Figure 13 the cropped fragment processing program and depth test program in the (b) diagram of

[0097] In some possible implementation manners, step S200 may include the following sub-operations of step S210-step S220 as shown in Figure 14 :

[0098] S210. Determine the visibility information of multiple fragments according to the depth information of multiple fragments.

[0099] Exemplarily, step S200 may include the following sub-operations of step S211-step S214 as shown in Figure 15 :

[0100] First, perform the operations of the following step S211-step S213 on multiple fragments in a streaming manner to obtain the depth information of multiple fragments:

[0101] S211. Generate the pose information of the currently processed fragment according to the existing depth information.

[0102] Exemplarily, the cropped fragment processing program in the cropped program as shown in the (b) diagram of Figure 13 can be run, and the currently processed fragment is drawn based on the existing depth information, and the corresponding pose information is generated. Exemplarily, while running the cropped fragment processing program in the cropped program as shown in the (b) diagram of Figure 13 to generate the pose information, the first rendering parameter can also be generated.

[0103] S212. Perform rasterization processing on the currently processed fragment.

[0104] S213. Perform a first depth test on the fragment pixels of the currently processed fragment according to the pose information of the currently processed fragment to obtain the depth information of the currently processed fragment. After recording the depth information of the currently processed fragment, continue to return to S211 to process the next fragment. The pose processing of the next fragment can refer to the depth information of the previously processed fragments.

[0105] In some possible implementation manners, such as Figure 9 , Figure 10 and Figure 11 shown, after rasterization processing, the first rendering module 12C can perform a first depth test on the pixels of the currently processed fragment based on the pose information to obtain the depth information of the currently processed fragment. Exemplarily, the first rendering module 12C obtains the pose information based on the fragment processing program shown in the (b) diagram of Figure 13 , and performs a first depth test on the pixels of multiple fragments, thereby obtaining the depth information of the currently processed fragment. The existing depth information can be updated according to the depth information of the currently processed fragment.

[0106] After performing the operations of the above steps S211 - S213 on multiple fragments in a pipelined manner, the depth information of multiple fragments is obtained, and then perform the sub-operations of step S214:

[0107] S214. Determine the visibility information of multiple fragments according to the depth information of multiple fragments.

[0108] Exemplarily, such as Figure 16As shown, taking the sequential drawing of fragment 0, fragment 1, and fragment 2 as an example: First, draw fragment 0 based on the operation in step S211 to obtain the pose information of fragment 0. Since fragment 0 is the first fragment to be drawn, there is no depth information of previously processed fragments recorded in the current depth information. After fragment 0 undergoes the rasterization process in step S212, perform the operation in step S213 based on the pose information of fragment 0 to obtain the depth information of fragment 0. Based on the operation in step S213, it is determined that none of the pixels of fragment 0 are occluded. Secondly, based on the operation in step S211, draw fragment 1 to obtain the pose information of fragment 1. It is confirmed through the depth information of the previous fragment 0 that fragment 1 occludes a part of the pixels of fragment 0. After fragment 1 undergoes the rasterization process in step S212, perform the operation in step S213 based on the pose information of fragment 1 to obtain the depth information of fragment 1, and update the existing depth information. The updated depth information is used to indicate that none of the pixels of fragment 1 are occluded, and some pixels of fragment 0 are not occluded. Then, draw fragment 2 based on the operation in step S211 to obtain the pose information of fragment 2. Based on the depth information of the previous fragments 0 and 1, it can be determined that fragment 2 occludes a part of the pixels of fragment 1 and also occludes the remaining visible pixels of fragment 0. After fragment 2 undergoes the rasterization process in step S212, based on the operation in step S213, obtain the depth information of fragment 2, and update the existing depth information. The updated depth information is used to indicate that none of the pixels of fragment 2 are occluded, some pixels of fragment 1 are not occluded, and all pixels of fragment 0 are occluded. After performing the operations in steps S211 - S213 on multiple fragments, in the operation in step S214, based on the finally obtained depth information of multiple fragments, the occlusion relationship of each fragment pixel can be determined, and thus the invisible pixels and visible pixels in multiple fragments can be determined based on the occlusion relationship. In this way, the pixel visibility information of multiple fragments can be obtained.

[0109] S220. Execute the culling instruction in the rendering task for multiple fragments to update the visibility information.

[0110] Exemplarily, as Figure 8 , Figure 9 , Figure 10 and Figure 11 shown, the third image processor 10C can execute the culling instruction in the rendering task based on the first rendering module 12C to update the visibility information. In some examples, the rendering task can be generated by the processor 30 in the embodiment shown in Figure 3 . For example, an application program runs in the processor 30, and different application programs can call different task interfaces in the processor 30 to generate corresponding rendering tasks.

[0111] Exemplarily, step S220 may include the following as Figure 17Sub-operations of step S221 - step S223 shown:

[0112] S221. Discard invisible pixels of one or more fragments among multiple fragments according to the discard instruction to obtain the remaining pixels of at least one fragment.

[0113] Exemplarily, after the clipped fragment processing program, depth test, and rasterization processing in the running program shown in (b) of Figure 13 are executed in step S210, the clipped pixel rendering program in the running program shown in (b) of Figure 13 can be executed in step S221. Specifically: execute the discard instruction in the clipped pixel rendering program. In some examples, the discard instruction can be executed based on the depth information and the first rendering parameter obtained in step S210. During the execution of the discard instruction, pixels that do not require shading processing are discarded based on whether the fragment needs shading processing, etc. Since the visibility information in step S210 only considers the influence factor of occlusion relationship on visibility, and the occluded fragment pixels may also need to be shaded (for example, the pixel is partially occluded by the transparent or semi-transparent pixels of other fragments), the visibility information obtained in step S210 can ignore the visibility of the occluded pixels. At this time, in step S220, the invisible pixels can be re-determined based on the discard instruction from the application program, such as the application program running on the processor 30, and the invisible pixels of one or more fragments among multiple fragments are discarded to obtain the remaining pixels of at least one fragment.

[0114] S222. Perform a second depth test on the remaining pixels to obtain updated depth information.

[0115] Exemplarily, in the third image processor 10C shown in Figure 9 , Figure 10 and Figure 11 , after the first rendering module 12C executes the discard instruction in the shading processing stage corresponding to the pixel rendering program, a second depth test is performed on the remaining pixels after discard. Updated depth information can be obtained through the second depth test. The updated depth information includes the relevant information of the pixel part where the fragment is occluded but needs to be shaded and displayed.

[0116] S223. Update the visibility information according to the updated depth information.

[0117] Exemplarily, in Figure 9 , Figure 10 and Figure 11In the third image processor 10C shown, the first rendering module 12C may update the pixel visibility information obtained in the above step S210 according to the updated depth information. The updated pixel visibility information includes pixel-level visibility information in the full-image rendering process.

[0118] Exemplarily, as Figure 18 shown in the (a) diagram of, on the same 2×2 pixel block (including pixel 0, pixel 1, pixel 2, and pixel 3), first draw fragment 0, and then draw fragment 1 on the basis of fragment 0, and fragment 1 is opaque. In the first depth test of step S210, it is recorded in the visibility information that pixel 2 of fragment 0 is a visible pixel, pixel 0, pixel 1, and pixel 3 of fragment 0 are invisible pixels, and pixel 0, pixel 1, and pixel 3 of fragment 1 are visible pixels. When executing the culling instruction in the first rendering process, since fragment 1 is opaque, it is determined that the occluded pixels of fragment 0 need to be shaded at pixel 2 on the final pixel block. In the second depth test of step S220, it is confirmed that pixel 2 of fragment 0 is recorded as a visible pixel, and pixel 0, pixel 1, and pixel 3 of fragment 1 are visible pixels. As Figure 18 shown in the (b) diagram of, fragment 0 is opaque and fragment 1 is semi-transparent. When fragment 1 occludes fragment 0, the shading effects of both fragment 0 and fragment 1 need to be displayed on this pixel block. In step S210 of the first rendering process, in the visibility information obtained based on the occlusion relationship of the first depth test, it is recorded that the pixels of fragment 0 are invisible pixels and the pixels of fragment 1 are visible pixels. In the second depth test of step S220 in the first rendering process, the visibility information can be updated based on the culling instruction, and the pixels of both fragment 0 and fragment 1 are recorded as visible pixels.

[0119] Exemplarily, since the pixel visibility information includes the record information of whether each pixel of each fragment of the first image is visible. Therefore, as long as a fragment has one visible pixel, it is a visible fragment. The fragment visibility information can be obtained based on the pixel visibility information. At the same time, the visible pixel part of each visible fragment can also be recorded in the fragment visibility information.

[0120] S300. Perform a second rendering process on at least one visible fragment according to the visibility information to obtain the first image.

[0121] In some possible implementation manners, the second rendering process includes obtaining the visible pixels of at least one visible fragment according to the visibility information. Performing a pixel rendering process on the visible pixels of at least one visible fragment.

[0122] In some possible implementation manners, in the second rendering process, the processing order of at least one visible fragment is different from the acquisition order of multiple fragments. In this application, asFigure 4 In the illustrated embodiment, the order of fragment drawing depends on the order of fragment acquisition. In the embodiments of the present application, such as Figure 5 In the illustrated embodiment, the drawing order of the n fragments cached in the latency buffer 13B may be different from the acquisition order, but the drawing order of the fragments other than n also needs to depend on the acquisition order. In the embodiments of the present application, such as Figure 8 In the illustrated embodiment, since very accurate visibility information has been obtained in the first rendering process stage. Therefore, in the second rendering process stage, it is not necessary to perform fragment rendering processing based on the acquisition order of multiple fragments. In actual applications, the processing order of visible fragments can be adjusted according to design requirements.

[0123] In some possible implementation manners, depth testing may not be performed in the second rendering process. In the embodiments of the present application, since very accurate visibility information has been obtained in the first rendering process stage. Therefore, in the second rendering process stage, during the rendering process, pixels that do not need to be processed can be culled directly based on the visibility information, rather than necessarily relying on depth testing to perform pixel culling in the second rendering process stage.

[0124] In some possible implementation manners, in image rendering, there is a principle that only one fragment is drawn for one pixel block. In the above Figure 18 In the illustrated embodiment, there may be pixel visibility information of multiple fragments in one pixel block. When storing the pixel visibility information of multiple visible fragments corresponding to the same pixel block in the same cache circuit, there may be an incompatibility problem. The incompatible multiple pixel visibility information can be stored in different buffers in the second cache circuit of the third image processor 10C.

[0125] Exemplarily, as Figure 19 In the illustration, taking the rendering process of a total of eight fragments, fragment 0 to fragment 7, in one pixel block as an example, as Figure 19As shown in Figure (a), fragment 0, fragment 3, fragment 5, fragment 6, and fragment 7 are transparent (or semi-transparent) fragments. Fragment 1, fragment 2, and fragment 4 are opaque fragments. When the first rendering module 12C performs the first rendering process, since fragment 4 is an opaque fragment and blocks fragment 0, fragment 1, fragment 2, and fragment 3, the pixel visibility information of fragment 4 is stored in the first buffer buffer1 in the second buffer circuit. And fragment 5, fragment 6, and fragment 7 are fragments incompatible with fragment 4, and the pixel visibility information of fragment 5, fragment 6, and fragment 7 can be stored in the second buffer buffer2 in the second buffer circuit. In some examples, the buffer for storing depth information in the third image processor 10C can be reused as the second buffer buffer2. For example, since fragment 4 is the topmost fragment among multiple opaque fragments, and fragment 7 is the topmost fragment among multiple transparent (or semi-transparent) fragments, and fragment 7 covers fragment 4. Therefore, only the depth information of fragment 7 can be stored in the second buffer buffer2 as a parameter representing pixel visibility information.

[0126] As Figure 19 shown in Figure (b), when the second rendering module 13C performs the second rendering process, the corresponding pixel visibility information is read from the first buffer buffer1 and the second buffer buffer2 respectively. According to the pixel visibility information of fragment 4 obtained from the first buffer buffer1, it can be determined that the pixels of fragment 0, fragment 1, fragment 2, and fragment 3 are invisible pixels. Therefore, fragment 0, fragment 1, fragment 2, and fragment 3 can be directly culled. Then, based on the depth information of fragment 7 in the second buffer buffer2, it is confirmed that the pixels of fragment 5, fragment 6, and fragment 7 above fragment 4 are visible pixels, and then pixel rendering is jointly implemented based on fragment 4, fragment 5, fragment 6, and fragment 7 in the shading processing stage.

[0127] In some possible implementation manners, in step S300, at least one fragment among multiple fragments may perform the following sub-operations of step S310-step S330 as Figure 20 shown:

[0128] S310. Perform fragment drawing processing.

[0129] In the embodiments of the present application, since the fragment visibility information is very accurate visibility information. Therefore, in the fragment drawing processing stage, processing of invisible fragment pixels can be effectively avoided. For the technical principle of fragment drawing processing, reference can be made to the relevant descriptions in the foregoing embodiments, which will not be elaborated here.

[0130] S320. Perform rasterization processing.

[0131] S330. Perform shading processing.

[0132] In the embodiments of the present application, the operation of performing shading processing based on the pixel rendering program is the operation with the largest processing volume in the entire image rendering process. In the second rendering process, only performing shading processing on visible pixels based on the pixel visibility information can greatly reduce the execution processing volume of the pixel rendering program.

[0133] Exemplarily, as Figure 21 shown, in the second rendering stage, perform actual image rendering processing on the Figure 16 multiple fragments recorded: First, perform rendering processing on fragment 0. According to the visibility information, it is determined that fragment 0 does not have visible pixels. Therefore, no processing is performed on fragment 0 in both the fragment drawing processing stage and the shading processing stage. Secondly, perform rendering processing on fragment 1. According to the visibility information, it is determined that some pixels of fragment 1 are visible pixels. Only the visible pixel part of fragment 1 is drawn in both the fragment drawing processing stage and the shading processing stage. Finally, perform rendering processing on fragment 2. According to the visibility information, it is determined that all pixels of fragment 2 are visible pixels. All pixels of fragment 2 are drawn in both the fragment drawing processing stage and the shading processing stage. Compared with Figure 16 , it can be seen that in the case of rendering the same multiple fragments, Figure 21 the implementation method can greatly reduce the processing volume of image rendering.

[0134] In summary, in the above Figure 4 and Figure 5 embodiments, in the fragment drawing processing stage, it is necessary to rely on the drawing order of the fragments to obtain depth information, making it impossible to effectively obtain the fragment visibility information. In addition, in the Figure 4 and Figure 5 embodiments in the shading processing stage, only the elimination of invisible pixels related to depth information is concerned, and the elimination of invisible pixels related to the first rendering parameter is not concerned. Therefore, in the Figure 4 and Figure 5 embodiments, there is a lack of confirmation of many visibility information, and it is impossible to maximize the extraction of invisible pixels, nor can it maximize the elimination of invisible pixels in different processing stages. However, in the present application, such as Figure 8 , Figure 9 , Figure 10 , Figure 11 , Figure 12 , Figure 13 , Figure 14 , Figure 15 , Figure 16 , Figure 17 , Figure 18 , Figure 19 , Figure 20 andFigure 21 In the embodiment, visibility information related to depth information after fragment drawing processing is obtained based on the first rendering process, and visibility information based on culling instructions is also obtained in the shading process stage. Thus, through the first rendering process, not only the visibility information related to depth information is obtained, but also the visibility information related to culling instructions in the shading stage is concerned, so as to identify and confirm the invisible pixel parts in multiple fragments of the rendering task. Executing the second rendering process based on the visibility information obtained from the first rendering process can maximize pixel culling in stages such as the fragment drawing process stage and the shading process stage in the second rendering process stage. And the first rendering process only includes processing programs related to the execution of depth testing and culling instructions, and this part of the processing program only occupies a very small part of the image rendering program. Also, although depth testing is performed in the first rendering process, the depth testing process can be correspondingly reduced in the second rendering process. Therefore, compared with the reduction in processing volume of the second rendering process, the increase in processing volume in the first rendering process can be ignored. Compared with Figure 4 and Figure 5 's solutions, the solution based on the third image processor 10C can reduce the processing volume of more image rendering.

[0135] The embodiment of the present application also provides an image rendering device, which includes an image processing circuit and a memory. The image processing circuit is coupled to the memory. The memory stores a rendering task of the first image, and the rendering task includes culling instructions. The image processing circuit executes the image rendering method in the above embodiment and any other manner based on the rendering task (for example Figure 12 、 Figure 14 、 Figure 15 、 Figure 17 and Figure 20 the image rendering methods described).

[0136] Exemplarily, the image rendering device can be an image processing chip, the graphics processing circuit is the graphics processing part of the image processing chip, and the memory is the data storage or cache part in the image processing chip.

[0137] Exemplarily, the image rendering device can include multiple chips, the graphics processing circuit is one or more image processing chips among the multiple chips, and the memory is the storage chip among the multiple chips.

[0138] The embodiment of the present application also provides a computer-readable storage medium, which includes instructions; when the instructions are in an image processor (such as the above Figure 8 、 Figure 9 、 Figure 10 and Figure 11When running on the third image processor 10C) described, it causes the image processor to execute the image rendering method described in the above embodiments and other ways (for example Figure 12 , Figure 14 , Figure 15 , Figure 17 and Figure 20 the image rendering method described).

[0139] An embodiment of this application also provides a computer program product. When the computer program product runs on a computer, it causes the computer to execute the image rendering method described in the above embodiments and other ways (for example Figure 12 , Figure 14 , Figure 15 , Figure 17 and Figure 20 the image rendering method described).

[0140] The processor involved in the embodiments of this application can be a chip. For example, it can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processing circuit (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0141] The memory involved in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include but not be limited to these and any other suitable types of memory.

[0142] It should be understood that in various embodiments of the present application, the magnitudes of the sequence numbers of the above processes do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0143] Those of ordinary skill in the art can realize that the modules and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0144] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and modules described above can refer to the corresponding processes in the foregoing method embodiments, and will not be described herein again.

[0145] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or modules can be in electrical, mechanical, or other forms.

[0146] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical modules, that is, they can be located in one device or distributed to multiple devices. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] In addition, in each embodiment of the present application, the functional modules can be integrated in one device, or each module can exist physically alone, or two or more modules can be integrated in one device.

[0148] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or data center that contains one or more integrated media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a Solid State Disk (SSD)), etc.

[0149] As described above, it is only the specific implementation manner of the present application. However, the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims described above.

Claims

1. An image processor, characterized in that, the image processor includes: an acquisition module, configured to acquire a plurality of fragments; a first rendering module, configured to perform a first rendering process on the plurality of fragments, the first rendering process including: determining visibility information of the plurality of fragments according to depth information of the plurality of fragments, and updating the visibility information according to a culling instruction in a rendering task, the visibility information being used to indicate at least one of the following: at least one visible fragment among the plurality of fragments or visible pixels of the at least one visible fragment; a second rendering module, configured to perform a second rendering process on the at least one visible fragment according to the visibility information to obtain a first image.

2. The image processor according to claim 1, characterized in that, the first rendering process further includes: performing a first depth test on pixels of the plurality of fragments to obtain the depth information.

3. The image processor according to claim 2, characterized in that, before performing the first depth test, the first rendering process further includes: performing rasterization processing on the plurality of fragments.

4. The image processor according to any one of claims 1-3, characterized in that, the updating the visibility information according to the culling instruction in the rendering task includes: culling invisible pixels of one or more fragments among the plurality of fragments according to the culling instruction to obtain remaining pixels of at least one fragment; performing a second depth test on the remaining pixels to obtain updated depth information; updating the visibility information according to the updated depth information.

5. The image processor according to any one of claims 1-4, characterized in that, the performing the second rendering process on the at least one visible fragment according to the visibility information includes: acquiring the at least one visible fragment among the plurality of fragments according to the visibility information; performing a second rendering process on the at least one visible fragment.

6. The image processor according to any one of claims 1-5, characterized in that, the second rendering process includes: acquiring visible pixels of the at least one visible fragment according to the visibility information; performing the shading process on the visible pixels of the at least one visible fragment.

7. The image processor according to claim 6, characterized in that, before the shading process, the second rendering process further includes: fragment drawing processing and rasterization processing.

8. The image processor according to any one of claims 1-7, characterized in that, in the second rendering process, a processing order of the at least one visible fragment is different from an acquisition order of the plurality of fragments.

9. The image processor according to any one of claims 1-8, characterized in that, the image processor further includes a program generation module; the program generation module is configured to perform clipping on the rendering task to obtain a clipped program related to visibility; the first rendering module is further configured to perform the first rendering process on the plurality of fragments by running the clipped program.

10. The image processor according to any one of claims 1-8, characterized in that, The obtaining module is further configured to obtain a cropped program related to visibility from the rendering task; The first rendering module is further configured to perform the first rendering process on the plurality of fragments by running the cropped program.

11. An image rendering method, characterized in that, the method includes: obtaining a plurality of fragments; performing a first rendering process on the plurality of fragments, where the first rendering process includes: determining visibility information of the plurality of fragments according to depth information of the plurality of fragments, and updating the visibility information according to a culling instruction in the rendering task, where the visibility information is used to indicate at least one of the following: at least one visible fragment among the plurality of fragments or visible pixels of the at least one visible fragment; performing a second rendering process on the at least one visible fragment according to the visibility information to obtain a first image.

12. The image rendering method according to claim 11, characterized in that, the first rendering process further includes: performing a first depth test on pixels of the plurality of fragments to obtain the depth information.

13. The image rendering method according to claim 12, characterized in that, before performing the first depth test, the first rendering process further includes: performing rasterization processing on the plurality of fragments.

14. The image rendering method according to any one of claims 11-13, characterized in that, the updating the visibility information according to the culling instruction in the rendering task includes: culling invisible pixels of one or more fragments among the plurality of fragments according to the culling instruction to obtain remaining pixels of at least one fragment; performing a second depth test on the remaining pixels to obtain updated depth information; updating the visibility information according to the updated depth information.

15. The image rendering method according to any one of claims 11-14, characterized in that, the performing a second rendering process on the at least one visible fragment according to the visibility information includes: obtaining the at least one visible fragment among the plurality of fragments according to the visibility information; performing a second rendering process on the at least one visible fragment.

16. The image rendering method according to any one of claims 11-15, characterized in that, the second rendering process includes: obtaining visible pixels of the at least one visible fragment according to the visibility information; performing the shading process on the visible pixels of the at least one visible fragment.

17. The image rendering method according to claim 16, characterized in that, before the shading process, the second rendering process further includes: fragment drawing process and rasterization process.

18. The image rendering method according to any one of claims 11-17, characterized in that, in the second rendering process, the processing order of the at least one visible fragment is different from the obtaining order of the plurality of fragments.

19. The image rendering method according to any one of claims 11-18, characterized in that, the method further includes: cropping the rendering task to obtain a cropped program related to visibility; By running the cropped program, perform the first rendering process on the plurality of fragments.

20. The image rendering method according to any one of claims 11-18, characterized in that the method further includes: obtaining a cropped program related to visibility from the rendering task; By running the cropped program, perform the first rendering process on the plurality of fragments.

21. An image rendering apparatus, characterized in that the image rendering apparatus includes an image processing circuit and a memory; the image processing circuit is coupled to the memory; the memory stores a rendering task for a first image; the rendering task includes culling instructions; the image processing circuit is configured to execute the image rendering method according to any one of claims 11-20.

22. An electronic device, characterized in that it includes a circuit board and an image processor according to any one of claims 1-10; the image processor is disposed on the circuit board.

23. A computer-readable storage medium, characterized in that the computer-readable storage medium includes instructions; when the instructions are run on an image processor, the image processor is caused to execute the image rendering method according to any one of claims 11-20.

Images