Image rendering method based on variable rate coloring and electronic equipment

Abstract

The invention provides an image rendering method based on variable rate coloring and electronic equipment. The method comprises the steps that a sheltered area and a non-sheltered area on a to-be-displayed image of a first application are determined, the sheltered area is a sheltered area in the to-be-displayed image, and the non-sheltered area is a non-sheltered area in the to-be-displayed image; the to-be-displayed image is colored and rendered based on the variable rate, a display image of the first application is generated, the occlusion area is rendered at least based on the first coloring rate, the non-occlusion area is rendered at least based on the second coloring rate, and the first coloring rate is smaller than the second coloring rate. According to the method provided by the embodiment of the invention, the occlusion area of the application is identified and variable-rate coloring is carried out, so that the rendering calculation amount can be reduced while the user experience is not reduced, the rendering performance is improved, and the display power consumption is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image rendering method and electronic device based on variable rate shading. Background Art

[0002] In the field of image display, shading is one of the most computationally expensive parts of the rendering pipeline (Render pass). As screen resolution and refresh rates increase, the demand for shading calculations is growing significantly, and even the most powerful GPUs cannot meet the growing demand.

[0003] To meet this challenge, some graphics processing unit (GPU) manufacturers have introduced a more flexible shading mechanism in their chipsets, called variable rate shading (VRS).

[0004] VRS can perform shading at a coarser frequency than pixels for a specified image area, thereby allocating rendering performance or computing power at different rates of rendering images. In other words, for a specified image area, VRS can shade a group of pixels as a unit, and then copy the result to all samples in the group, thereby reducing the computational cost of shading the specified area and improving rendering performance.

[0005] Although VRS has many advantages: flexible shading mechanism, decoupling of pixels and shading frequency, and improved rendering performance without reducing the perceptible output quality, it still has many shortcomings. In the application scenarios of VRS, in order to avoid any reduction in perceptible output quality and improve performance at the same time, it is necessary to identify which areas can be shaded at a low rate, and at the same time, these areas need to be maximized to achieve the best VRS application effect. In the existing VRS solution, the method of area identification is single and limited, which leads to limited performance improvement.

[0006] Therefore, a new VRS method is needed to improve rendering performance. Summary of the invention

[0007] In order to improve the rendering performance, the present application provides an image rendering method and an electronic device based on variable rate shading. The present application also provides a computer-readable storage medium.

[0008] The present application embodiment adopts the following technical solutions:

[0009] In a first aspect, the present application provides an image rendering method based on variable rate shading, which is applied to an electronic device. The method includes: determining an occluded area and a non-occluded area on an image to be displayed of a first application, wherein the occluded area is an occluded area in the image to be displayed, and the non-occluded area is an unoccluded area in the image to be displayed; based on variable rate shading, rendering the image to be displayed to generate a display image of the first application, wherein the occluded area is rendered based on at least a first shading rate, and the non-occluded area is rendered based on at least a second shading rate, and the first shading rate is less than the second shading rate.

[0010] According to the method of the embodiment of the present application, the occluded area of ​​the application is identified and variable rate shading is performed, which can reduce the amount of rendering calculations without reducing the user experience, thereby improving the rendering performance and reducing the display power consumption. According to the method of the embodiment of the present application, compared with the image motion area, the occluded area has a larger effective range and less user perception, so the variable rate shading benefits are stable and the scene coverage is wider.

[0011] In an implementation of the first aspect, the occlusion area includes any one or more combinations of a hand occlusion area, a user interface occlusion area, other application occlusion areas, and a device self-occlusion area, wherein: the hand occlusion area includes a hand coverage area or a part of a hand coverage area covered by the user's hand on the display screen when the user performs a touch operation; the user interface occlusion area includes an area in the image area of ​​the user interface of the first application that blocks the application display content of the first application; the other application occlusion area includes a window area of ​​the foreground active window of the second application, or a part of the window area of ​​the foreground active window of the second application; the device self-occlusion area includes an area of ​​the display screen of the electronic device corresponding to the front camera.

[0012] According to the method of the embodiment of the present application, the occlusion area is determined from multiple angles, the effective range of the occlusion area is improved, the stability of the variable rate shading gain is improved, and the scene coverage is expanded.

[0013] In an implementation of the first aspect, an occluded area and a non-occluded area on an image to be displayed of a first application are determined, including: responding to a touch event of a user, parsing the touch event to obtain touch area information; determining a hand coverage area based on the touch area information; marking the hand coverage area as a hand occlusion area; or, marking an area in the hand coverage area that overlaps with a display interface of the first application as a hand occlusion area.

[0014] In an implementation of the first aspect, determining the occluded area and the non-occluded area on the image to be displayed of the first application includes: determining the image area of ​​the user interface based on the display image of the previous frame; when the image area of ​​the user interface will cause occlusion of the application content of the first application, marking the image area in the image area of ​​the user interface that causes occlusion as the user interface occlusion area. In an implementation of the first aspect, determining the occluded area and the non-occluded area on the image to be displayed of the first application includes: identifying the user interface rendering channel of the previous frame; sampling the user interface rendering channel to obtain sampled pixels; when the sampled pixels have color values ​​and the transparency is greater than a preset threshold, marking the image area corresponding to the sampled pixels as the user interface occlusion area.

[0015] In an implementation of the first aspect, determining an occluded area and a non-occluded area on an image to be displayed of a first application includes: obtaining window information of a foreground active window of a second application; drawing a window area of ​​the foreground active window according to the window information of the foreground active window of the second application; marking the window area of ​​the foreground active window as an occluded area for other applications; or marking an area in the window area of ​​the foreground active window that overlaps with a display interface of the first application as an occluded area for other applications.

[0016] In an implementation of the first aspect, an occluded area and a non-occluded area on an image to be displayed of a first application are determined, including: comparing the shape of a normal display area of ​​a display screen with a rectangular display image, and drawing an abnormal display area of ​​the display screen according to the comparison result; marking the abnormal display area as a self-occluded area of ​​a device; or, marking an area in the abnormal display area that overlaps with a display interface of the first application as a self-occluded area of ​​a device.

[0017] In an implementation of the first aspect, the occluded area includes at least a first area. Based on variable rate shading, rendering the image to be displayed to generate a display image of the first application includes: selecting a first pixel area from the first area according to a first shading rate; the first area includes a first pixel area and a second pixel area; shading pixels of the first pixel area; and filling the second pixel area with a shading result based on the shading result of the first pixel area.

[0018] In an implementation of the first aspect, the occluded area also includes at least a second area. Based on variable rate shading, rendering the image to be displayed to generate a display image of the first application also includes: shading the second area using a third shading rate, the third shading rate being different from the first shading rate, and the third shading rate being less than the second shading rate.

[0019] In an implementation manner of the first aspect, the second area and the first area correspond to different shielding area types respectively.

[0020] In an implementation of the first aspect, the method includes: determining the occluded area and the non-occluded area on the display image of the first application of the N-1th frame; obtaining the rendering instruction stream and associated data of the image to be displayed of the first application of the Nth frame; performing display image rendering according to the rendering instruction stream and associated data of the image to be displayed of the first application of the Nth frame, and generating the display image of the first application of the Nth frame, wherein variable rate shading is applied according to the occluded area and the non-occluded area on the image to be displayed of the first application of the N-1th frame; determining the occluded area and the non-occluded area on the display image of the first application of the Nth frame.

[0021] In a second aspect, the present application provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method steps of the first aspect.

[0022] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program is stored, and when the computer-readable storage medium is run on a computer, the computer executes the method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 FIG. 1 is a flowchart of VRS execution according to an embodiment of the present application;

[0024] Figure 2 FIG. 1 is a flowchart of VRS execution according to an embodiment of the present application;

[0025] Figure 3 FIG. 1 is a flowchart of VRS execution according to an embodiment of the present application;

[0026] Figure 4 The figure is a schematic diagram of the display effect of a mobile phone screen according to an embodiment of the present application;

[0027] Figure 5 Shown is a guide mask diagram according to an embodiment of the present application;

[0028] Figure 6 It is a flowchart of drawing a guidance mask map according to an embodiment of the present application;

[0029] Figure 7 The figure shows the schematic diagram of the finger touch position of the user in the game state;

[0030] Figure 8 Shown is a guide mask diagram according to an embodiment of the present application;

[0031] Fig. 9 It is a flowchart of drawing a guidance mask map according to an embodiment of the present application;

[0032] Fig.10 Shown is a guide mask diagram according to an embodiment of the present application;

[0033] Fig.11 It is a flowchart of drawing a guidance mask map according to an embodiment of the present application;

[0034] Fig.12 Shown is a guide mask diagram according to an embodiment of the present application;

[0035] Fig.13 It is a flowchart of drawing a guidance mask map according to an embodiment of the present application;

[0036] Fig.14 Shown is a guide mask diagram according to an embodiment of the present application;

[0037] Fig.15 The figure is a schematic diagram of a mobile phone display screen according to an embodiment of the present application;

[0038] Fig.16 Shown is a guide mask diagram according to an embodiment of the present application;

[0039] Fig.17 The figure shows the values ​​of 8*8 data bits in a pixel shading list with a shading rate of 4*4;

[0040] Fig.18 The figure shows the shading result of 8*8 pixels with a shading rate of 4*4;

[0041] Fig.19 Shown is based on Fig.18 The coloring result is colored to supplement the result schematic diagram;

[0042] Fig. 20 is a flowchart of VRS implementation according to an embodiment of the present application;

[0043] Fig.21 is a schematic diagram of a system architecture according to an embodiment of the present application;

[0044] Fig. 22 This is a schematic diagram of the operation flow of the data acquisition module 2231 according to an embodiment of the present application;

[0045] Fig.23 2 is a schematic diagram of the operation flow of the variable rate shading module 2238 according to an embodiment of the present application;

[0046] Fig.24 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0048] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0049] In the application scenario of VRS, the key to improving rendering performance without reducing the perceptible output image quality is to identify areas that can be shaded at a low rate.

[0050] Figure 1 FIG. 4 is a flowchart of a VRS execution according to an embodiment of the present application.

[0051] In one embodiment, the GPU performs the following Figure 1 The following process is shown to implement VRS.

[0052] S100, obtaining regional motion data of an image frame.

[0053] In a dynamic image display scene, there is a moving object in the dynamic image picture, and the image area of ​​the moving object in the picture is the moving area of ​​the image frame.

[0054] For example, in a video of an object falling, when the shooting position and viewing angle remain unchanged (the background in the video remains unchanged), in the video, the falling object falls from the top of the screen to the bottom. Between different image frames of the video, the position of the falling object in the image frame keeps changing, and the moving area in the dynamic image is the image area corresponding to the falling object. For example, in the first image frame, the image area of ​​the falling object is the first area; in the second image frame (the first image frame is the previous frame of the second image frame), the image area of ​​the falling object is the second area; compared with the first image frame, the moving area of ​​the second image frame is the combination of the first area and the second area.

[0055] The regional motion data is data used to describe the motion region in the dynamic image. For example, in one embodiment, the regional motion data of the image frame may be the position of the motion region, the amplitude of the position change of the moving object in the motion region, the speed of the position change of the moving object in the motion region, etc.

[0056] S110, calculating a motion value of each motion region of the image frame according to the region motion data.

[0057] In some examples, the motion value is used to quantitatively describe the motion state of a moving object in a motion region. Generally, the faster the moving object moves, the greater the motion value.

[0058] For example, in a video of a falling object, the faster the object falls, the larger the corresponding motion value.

[0059] S120: Determine a coloring rate of the motion region based on the motion value of the motion region.

[0060] In dynamic image display scenarios, even if the display effect (e.g., display resolution) of the image frames is consistent, the image of the moving object is blurrier than that of the non-moving object in the image in the visual effect of the human eye. Therefore, actively reducing the image display effect of the moving object will not affect the visual effect perceived by the user's eyes.

[0061] In some embodiments, for moving areas in image frames whose movement speed exceeds the dynamic visual resolution capability of the human eye (the movement value exceeds a preset threshold), a relatively low shading rate can be used (actively reducing the image display fineness); for non-moving areas of the image frames and whose movement speed does not exceed the dynamic visual resolution capability of the human eye, a relatively high shading rate can be used.

[0062] The faster an object moves, the blurrier it appears to human vision. Therefore, a relatively low shading rate can be used for moving areas with high moving speeds (higher motion values), and a relatively high shading rate can be used for moving areas with low moving speeds (lower motion values).

[0063] S130 , rendering the graphics primitives at the shading rate determined in S120 to generate color data of the motion area.

[0064] according to Figure 1 The process shown can effectively implement the variable rate shading function, which can improve rendering performance and reduce power consumption to a certain extent. However, there are limitations to making shading rate decisions based solely on motion values. When the image is in a low speed or stationary state, the benefits of the above VRS solution are reduced.

[0065] Figure 2 FIG. 4 is a flowchart of a VRS execution according to an embodiment of the present application.

[0066] In one embodiment, the GPU performs the following Figure 2 The following process is shown to implement VRS.

[0067] S200, obtaining first content data rendered by a current image frame and second content data rendered by an image frame previous to the current image frame.

[0068] S210: Compare pixel values ​​of a current image frame and a previous image frame according to the first content data and the second content data.

[0069] S220, filtering the comparison result of S210 to determine the image portion (high-frequency details) corresponding to the pixels whose pixel value difference is lower than the preset threshold, and the image portion (non-high-frequency details) corresponding to the pixels whose pixel value difference is higher than or equal to the preset threshold in the current image frame.

[0070] Generally, due to the persistence of vision of the human eye, the faster the image changes, the less the human eye can recognize the image details. For example, a video shows a process where image A gradually changes to image B. The faster image A changes to image B, the less the human eye can recognize the details of the change from image A to image B in the picture.

[0071] Therefore, when the speed of picture changes exceeds the dynamic recognition ability of the human eye, actively reducing the image refinement in the picture change area will not affect the visual effect perceived by the user's eyes.

[0072] In the application scenario of displaying dynamic images, for a certain image area, the smaller the image difference between two adjacent image frames, the slower the picture changes in the image area, that is, the human eye can more clearly identify the image details of the image area.

[0073] Therefore, a preset threshold is set for the dynamic recognition ability of the human eye. In S220, the image portion (high-frequency details) corresponding to the pixels whose pixel value difference is lower than the preset threshold in the current image frame is obtained. The high-frequency details are the parts of the image details that can be recognized by the human eye. Correspondingly, the remaining non-high-frequency details are the parts of the image details that cannot be clearly recognized by the human eye. Reducing the image display effect of non-high-frequency details will not affect the visual effect perceived by the user's eyes.

[0074] S230, applying variable rate shading to the current image frame, and using low rate shading for non-high frequency details.

[0075] according to Figure 3 The process shown can effectively implement the variable rate shading function, which can improve rendering performance and reduce power consumption to a certain extent. However, there are limitations to determining the shading rate based solely on high-frequency details. When the picture is full of high-frequency details, the above VRS solution may not be beneficial.

[0076] In order to improve rendering performance without reducing the perceptible output image quality, an embodiment of the present application proposes a VRS method. In the method of the embodiment of the present application, before displaying an image for the current application (APP) (first application), it is necessary to render the image to be displayed to generate a display image for display.

[0077] Before rendering the image to be displayed, an occluded area and a non-occluded area on the image to be displayed of the first application are determined, wherein the occluded area is an occluded area in the image to be displayed, and the non-occluded area is an unoccluded area in the image to be displayed.

[0078] In the process of rendering the image to be displayed, low-rate shading is used for occluded areas, thereby improving rendering performance without reducing the perceptible output quality.

[0079] Figure 3 FIG. 4 is a flowchart of a VRS execution according to an embodiment of the present application.

[0080] In one embodiment, the GPU performs the following Figure 3 The following process is shown to implement VRS.

[0081] S310: Determine an occluded area and a non-occluded area on an image to be displayed of a first application.

[0082] In some examples, the first application is an application that currently needs to render a display image. That is, the first application is an application running in the foreground and displaying an image. The first application outputs an image to be displayed, and the image rendering system renders the image to be displayed to generate a display image (current image frame).

[0083] The first application can be displayed in full screen (the application currently running in the foreground is only the first application), and the first application can also be displayed in split screen with other applications (the applications currently running in the foreground are multiple applications including the first application).

[0084] The occluded area on the image to be displayed of the first application is the occluded area on the image to be displayed of the first application (current display interface); the non-occluded area on the image to be displayed of the first application is the unoccluded area on the image to be displayed of the first application (current display interface).

[0085] In one embodiment, in S310 , a guidance mask map is drawn, and the occlusion area and the non-occlusion area are described by the guidance mask map.

[0086] In some examples, the guidance mask image includes an interface graphic whose shape is consistent with the image to be rendered of the first application, whose size is consistent with the image to be rendered, or whose size is proportionally enlarged or reduced.

[0087] The interface graphics are divided into an occluded area and a non-occluded area by different color blocks. The occluded area can be a connected area or a collection of multiple connected areas; the non-occluded area can be a connected area or a collection of multiple connected areas.

[0088] Figure 4 Shown is a schematic diagram of a mobile phone screen display effect according to an embodiment of the present application.

[0089] like Figure 4 As shown, 400 is a mobile phone screen. A 3D game scene of a game application is displayed in full screen on screen 400; 401 is a game tool component; 402 is a character status component; 403, 404 and 405 are skill button components; 406 is a game assistant component. Screen 400 is a curved screen with a bangs, 409 is the bangs part, and the curved part of the screen is above the dotted line 407 and below the dotted line 408.

[0090] Figure 5 FIG. 4 is a guide mask diagram according to an embodiment of the present application.

[0091] against Figure 4 The display interface of the screen 400 is shown as a guide mask diagram. The drawn guide mask diagram is shown as 501. Rectangle 501 is an interface graphic with the same size and shape as the full-screen display interface of the screen 400. The graphic area of ​​the black block in the rectangle 501 corresponds to the blocked area in the display interface of the screen 400; the graphic area of ​​the white block in the rectangle 501 corresponds to the non-blocked area in the display interface of the screen 400.

[0092] In one embodiment, before S310, a graphics application programming interface (Application Programming Interface, API) call is intercepted, a rendering instruction stream and associated data are obtained, and an occluded area and a non-occluded area on the image to be displayed of the first application are determined according to the rendering instruction stream and the associated data.

[0093] In some examples, the rendering instruction stream includes rendering instructions that need to be executed during the image rendering process. The rendering instruction stream may include parameters such as interface size, light direction, light intensity, light color, fiber loss parameters, model pointing to be rendered, and material pointing to be rendered. The associated data includes data that needs to be loaded during the image rendering process. The associated data may include model data to be rendered, material data to be rendered, and texture data.

[0094] In some embodiments, the occlusion area may be any one or more combinations of the following situations.

[0095] 1: Hand occluded area.

[0096] When the user performs touch operations on the application, the user's hand (including fingers, palms, and wrists) will be between the display screen and the user's eyes. The user's hand blocks the user's line of sight, so that part of the display content is blocked by the user's hand. When rendering, the area blocked by the user's hand is shaded at a low rate without reducing the perceptible output quality.

[0097] Therefore, in one embodiment, the hand occlusion area includes the hand coverage area or a part of the hand coverage area covered by the user's hand on the display screen when the user performs a touch operation.

[0098] Taking a game application scenario as an example, the first application is a game application, and the first application is displayed in full screen.

[0099] When in game mode, the user holds the phone with both hands (the phone is in landscape orientation), and the fingers of both hands of the user touch the lower left and lower right corners of the phone screen respectively.

[0100] Figure 6 Shown is a flow chart of drawing a guidance mask map according to an embodiment of the present application.

[0101] S400: Obtain the current touch event.

[0102] In some examples, in S400, when rendering the current frame, at the end of the previous frame or the beginning of the current frame, the current touch event (Motion Event) is obtained through the system framework layer.

[0103] In an implementation of S400, the current touch event is a touch event that is currently occurring; or, when no touch event occurs at the current moment, the latest touch event is a touch event that is before the current moment and is closest to the current moment.

[0104] In another implementation of S400, the latest touch event is a touch event that occurs within a time period from a preset time length (eg, last second) before the current moment to the current moment.

[0105] S410: Analyze the touch event to obtain touch area information.

[0106] In some embodiments, the touch area information includes the number of simultaneous touches by the user, the center coordinates of the touch points, the major and minor axes of the touch ellipse area, and the deflection angle of the ellipse.

[0107] Figure 7 The figure shows a schematic diagram of the user's finger touch positions in the game state.

[0108] Screen 500 is a mobile phone screen.

[0109] In the holding posture of holding the mobile phone with both hands (the mobile phone is horizontal), the touch area of ​​the user's fingers can be obtained as follows: Figure 7 As shown in 501 and 502.

[0110] After S410 , S420 is executed.

[0111] S420, based on the touch area information and the user's holding habits ( Figure 5 The holding posture shown in the figure is used to predict the area covered by the user's hand on the display screen and generate the hand coverage area.

[0112] For example, based on Figure 7 In some embodiments, S420 includes:

[0113] The touch center point (center point 501) on the left half of the screen is displaced to the lower left corner of the screen by 1.5 times the distance of the long axis of the touch area, and the touch center point (center point 502) on the right half of the screen is displaced to the lower right corner of the screen by 1.5 times the distance of the long axis of the touch area.

[0114] The major axis and minor axis are doubled respectively, and the deflection angle of the ellipse remains unchanged to obtain the predicted hand coverage area.

[0115] like Figure 7 As shown, according to the positions of 501 and 502 and the user's holding habits, it can be predicted that areas 510 and 520 are the areas covered by the user's hands.

[0116] S430, drawing a guidance mask map, wherein an occlusion area (hand occlusion area) of the guidance mask map is drawn according to the predicted hand coverage area.

[0117] In some embodiments, in S430, the occlusion area (hand occlusion area) of the guidance mask image is drawn based on the ellipse information of the predicted hand coverage area (the predicted long and short axis lengths of the hand occlusion area, and the predicted center point coordinates of the hand occlusion area).

[0118] In an implementation of S430, the predicted hand coverage area is used as the hand occlusion area.

[0119] In another implementation of S430, after obtaining the hand coverage area, it is also determined whether the hand coverage area overlaps with the display interface of the first application. If there is an overlap, the area in the hand coverage area that overlaps with the display interface of the first application is marked as a hand occlusion area (drawing a hand occlusion area mask).

[0120] For example, when the first application is displayed in full screen, the entire area covered by the hand will block the display interface of the first application, and the entire area covered by the hand is marked as the hand blocking area. When the first application and other applications are displayed in split screen, the part of the hand covering area that overlaps with the display area of ​​the first application will block the display interface of the first application, and the part of the hand covering area that overlaps with the display area of ​​the first application is marked as the hand blocking area.

[0121] Figure 8 FIG. 4 is a guide mask diagram according to an embodiment of the present application.

[0122] After S430, you can finally draw Figure 8 The guidance mask diagram 800 is shown. Figure 8 In the illustrated guidance mask image 800 , the areas of black blocks 610 and 620 are occluded areas (hand occluded areas).

[0123] The touch hotspot is the area on the screen where the user frequently performs touch operations. For example, the screen area where the user's finger touches for a period of time greater than or equal to 70% within a unit time is the touch hotspot. In some embodiments, based on the touch hotspot and the user's hand gesture, the area on the display screen covered by the user's hand when the user performs a touch operation in the touch hotspot is predicted to generate a hand coverage area.

[0124] 2: User interface (UI) occlusion area.

[0125] The user interface is software designed to facilitate communication between users and hardware. It aims to enable users to operate hardware conveniently and efficiently to achieve two-way interaction and complete the tasks they want to accomplish with the help of the hardware. The user interface has a broad definition, including human-computer interaction and graphical user interface. User interfaces exist in all fields involving information exchange between humans and machines.

[0126] When the application is running, the application UI will be provided for the convenience of user operation. The application UI is overlaid on the application content itself, and the application content blocked by the UI is invisible. When rendering, the area blocked by the UI is shaded at a low rate, which does not reduce the perceptible output quality.

[0127] For example, when a video application plays a video, if the UI (video playback control button) is hidden, the display screen displays the entire video screen. If the UI is displayed while the video is playing, one UI display scheme is that the UI is displayed floating above the video screen, covering a portion of the video screen.

[0128] For another example, when a game application is running, the rendering of the game application screen is usually divided into the main game scene drawing and the game UI drawing. The main game scene includes the game terrain, vegetation, buildings, characters, special effects, etc.; the game UI includes operation buttons, character status bar, game system bar, etc. The game UI is displayed floating above the main game scene, covering part of the main game scene.

[0129] Therefore, in one embodiment, the user interface (UI) blocking area includes an area in the image area of ​​the user interface that blocks the application display content.

[0130] In some embodiments, in S310, an image area of ​​the user interface is identified from the displayed image of the previous frame; it is determined whether the image area of ​​the user interface will cause occlusion of the application content; if it will cause occlusion, the image area corresponding to the user interface that causes the occlusion is confirmed as the UI occlusion area.

[0131] In the image rendering process, the data to be rendered is usually divided into multiple parts for separate rendering according to the rendering method and the type of rendering data. The rendering process of each part is called a rendering pass. Finally, the rendering results of all rendering passes are combined into the final rendered image result. For example, the image to be displayed is divided into multiple layers, a rendering process (rendering pass) is created for each layer, and finally the rendering results of multiple rendering passes are superimposed and displayed.

[0132] For example, in an application scenario of a game application, the display content of the game interface includes 3D background images, 3D item images, 2D game character images, moves and special effects, character dialogues, item names, and game operation interfaces. The display order is to display the 3D background image; superimpose the 3D item images and 2D game character images on the 3D background image; mark the item names on the 3D item images; attach character dialogues and moves and special effects to the 2D game character images; and the top of all layers is the game operation interface. In the image rendering process, a 3D background rendering channel is created, and the 3D background rendering channel is used to render and generate a background layer (the background layer includes a 3D background image); a 3D item rendering channel is created, and the 3D background rendering channel is used to render and generate an item layer (the item layer includes a 3D item image marked with an item name); a 2D rendering channel is created, and the 2D background rendering channel is used to render and generate a 2D layer (the 2D layer includes a 2D game character image with attached character dialogues and moves and special effects); a UI rendering channel is created, and the UI rendering channel is used to render and generate a game operation interface. Recently, the background layer, item layer, 2D layer and game operation interface are superimposed and displayed in order.

[0133] In one implementation of S310, a rendering channel including UI rendering is identified in the rendering channel of the previous image frame. The rendering channel including UI rendering may be a rendering channel including only UI rendering (in the rendering process, an independent rendering channel is used for UI rendering), or may be a rendering channel including UI rendering and other content rendering (in the rendering process, UI rendering and other rendering contents are merged into one rendering channel for rendering).

[0134] The UI rendering part of the rendering channel including UI rendering is split into an independent rendering channel, so as to obtain a UI rendering channel used only for drawing UI. According to the UI rendering channel, the UI interface displayed opaquely is determined, and the UI occlusion area is generated.

[0135] For example, according to the UI rendering channel, the pixels of the UI interface are sampled. If the color of the pixel is not the initial color (the pixel is not the initial color, that is, the pixel has color), and the transparency (alpha value) of the pixel is greater than the preset blending threshold (for example, 0.2), it means that the pixel is not transparent, and the pixel is marked as a UI occlusion pixel. The pixels of the entire UI rendering channel are traversed, and finally the UI occlusion area is determined according to the set of UI occlusion pixels.

[0136] Fig. 9 Shown is a flow chart of drawing a guidance mask map according to an embodiment of the present application.

[0137] S700: Obtain a rendering channel including UI rendering.

[0138] S710: Determine whether the rendering channel including UI rendering is a rendering channel for performing UI rendering alone.

[0139] If the rendering channel includes UI rendering and is not a rendering channel for performing UI rendering alone, execute S720.

[0140] S720: Split and draw a UI rendering part in a rendering channel including UI rendering into an independent UI rendering channel.

[0141] S730, pixel sampling for independent UI rendering channels.

[0142] If the rendering channel includes UI rendering and is not a rendering channel for performing UI rendering alone, execute S730.

[0143] S740, determining whether the sampling result has a color value and a transparency value greater than a preset mixing threshold.

[0144] If the judgment result is no, execute S751.

[0145] If the judgment result is yes, execute S750.

[0146] S750: Mark the sampled pixel as a UI occlusion pixel.

[0147] S751, determine whether to traverse all pixels of the UI rendering channel.

[0148] If not, return to S730.

[0149] If yes, execute S760.

[0150] S760 , drawing a guidance mask image, wherein the occlusion area (UI occlusion area) in the guidance mask image is drawn according to the marking result (UI occlusion pixel) of S750 .

[0151] Take a game application scenario as an example. Figure 4 As shown, the game tool component 401, the character status component 402, the skill button components 403, 404 and 405 in the mobile phone screen 400 will block the 3D game scene screen below.

[0152] Fig.10 FIG. 4 is a guide mask diagram according to an embodiment of the present application.

[0153] according to Figure 4 The positions of the game tool component 401, the character status component 402, the skill button components 403, 404 and 405 in the mobile phone screen 400 can be drawn as follows: Fig.10 The guidance mask image 1000 is shown. In the guidance mask image 1000, black block areas 1001, 1002, 1003, 1004 and 1005 are occlusion areas (UI occlusion areas).

[0154] 3: Other applications block the area.

[0155] When the first application is running in the foreground, there may be other applications running, and the other applications may be applications running in the background or applications running in the foreground and displayed in a split screen with the first application. For example, the first application is running in the foreground, and the other applications other than the first application include at least a second application, and the second application is not the same application as the first application. The second application may be running in the background, or the second application may be running in the foreground and displayed in a split screen with the first application.

[0156] Applications other than the first application may cause a pop-up window to be displayed above the display interface of the first application running in the foreground, or a floating window to be maintained above the display interface of the first application running in the foreground.

[0157] When other applications display pop-up windows or maintain floating window display above the display interface of the first application, the content of the first application blocked by the pop-up window or floating window is not visible. When rendering, low-rate shading is performed on the area blocked by the pop-up window or floating window, and the perceptible output image quality is not reduced.

[0158] For example, when a video application is playing a video, if a chat application running in the background receives a new chat message, the chat message is displayed as a pop-up window above the video screen, blocking a portion of the video screen.

[0159] For another example, when a game application is running, in order to improve the gaming experience, the user can run a game assistant in the background, and the game assistant provides a quick operation bar for the game. The quick operation bar is displayed above the main game scene in the form of a floating window, covering a part of the main game scene.

[0160] Therefore, in one embodiment, the area blocked by other applications includes the window area of ​​the foreground active window of other applications (including at least the second application) other than the first application, or a part of the window area of ​​the foreground active window of other applications (including at least the second application) other than the first application.

[0161] In one embodiment, in S310, when rendering the current frame, at the end of the previous frame or the beginning of the current frame, the window information of the foreground active window of other applications (other applications include at least the second application, and the second application is not the same application as the first application running in the foreground) other than the first application is obtained through the system framework layer. The foreground active window includes a fixed window, a floating window, and a pop-up window displayed on the screen.

[0162] Parse the window information of the foreground active window to obtain the size, shape, center coordinates and other information of the foreground active window. Draw the window area of ​​the foreground active window of other applications and mark the window area of ​​the foreground active window of other applications as the occlusion area of ​​other applications.

[0163] Alternatively, in another embodiment, after obtaining the window area of ​​the foreground active window of another application, it is determined whether the window area overlaps with the display interface of the first application. If there is overlap, the overlapping area is marked as the other application occlusion area.

[0164] In some embodiments, when the first application running in the foreground is displayed in full screen (there is only one foreground application running), when there is a foreground active window of other applications other than the first application (for example, the other applications other than the first application include the second application, and the second application runs in the background), the entire area of ​​the foreground active window overlaps with the display interface of the first application. At this time, the window area of ​​the foreground active window of the other application other than the first application is marked as the other application obstruction area.

[0165] When multiple foreground applications are running (for example, the first application is displayed in split screen with one or more other applications), if there is an active foreground window of another application other than the first application, the overlapping area of ​​the foreground active window and the display interface of the first application is determined according to the position / size of the foreground active window. The overlapping area of ​​the foreground active window of another application other than the first application and the display interface of the first application is marked as the other application occlusion area.

[0166] Taking a game application scenario as an example, the first application is a game application, and the first application is displayed in full screen.

[0167] Fig.11 Shown is a flow chart of drawing a guidance mask map according to an embodiment of the present application.

[0168] S1000, determining through the system framework layer whether there are other application windows currently on the game screen (whether there are foreground active windows of other applications).

[0169] If not present, the process ends.

[0170] If it exists, execute S1010.

[0171] S1010, parsing the application window information obtained through the framework layer, and obtaining the window information of the foreground active window of other applications.

[0172] The window information of the foreground active window of other applications includes the number of windows and the window coordinates, window shape, and window size information of each window.

[0173] S1020, drawing a guidance mask image, wherein the occlusion area (other application occlusion area) in the guidance mask image is drawn according to other application window information.

[0174] like Figure 4 As shown, the game assistant component 406 in the mobile phone screen 400 is an application interface of the game assistant application other than the game application. The game assistant component 406 will block the 3D game scene screen below.

[0175] Fig.12 FIG. 4 is a guide mask diagram according to an embodiment of the present application.

[0176] according to Figure 4 The position of the game assistant component 406 in the mobile phone screen 400 shown in FIG. 4 can be drawn as follows: Fig.12 The guidance mask image 1200 is shown. In the guidance mask image 1200, the black block area 1201 is the occlusion area (other application occlusion area).

[0177] 4: Device self-occlusion area.

[0178] A mobile phone is usually equipped with a front camera. Generally, the front camera is installed on the mobile phone display.

[0179] For example, some display screens have a bangs area; for another example, some display screens have a hole area. For example, the bangs area or the hole area may correspond to one or more front cameras. Due to the presence of the front camera, the area corresponding to the front camera cannot display images.

[0180] When rendering, the parts of the application's rectangular display screen that cannot be displayed (corresponding to the display content of the rectangular top corners / bangs area / hole area) are shaded at a low rate, which does not reduce the perceptible output image quality.

[0181] Therefore, in one embodiment, the device self-occlusion area includes an area of ​​the display screen of the electronic device corresponding to the front camera.

[0182] In some embodiments, the device self-occlusion area may include the area of ​​the under-screen camera, and the area corresponding to the under-screen camera can display images. It should be understood that the area corresponding to the under-screen camera is generally close to the edge of the screen, so the methods described in the embodiments of the present application can also be applied.

[0183] Generally, to ensure the universality of displayed content, the display screen of an application is rectangular. However, the shape of some display screens is not strictly rectangular.

[0184] For example, some displays are rounded rectangles. The top corners of the rectangular display screen cannot be displayed by the display screen. When rendering, the undisplayable parts of the rectangular display screen of the application (corresponding to the display content of the top corners / fringe area / hole area of ​​the rectangle) are shaded at a low rate without reducing the perceptible output quality.

[0185] For another example, some displays (curved screens) have curved display areas. Due to viewing angle limitations, users do not carefully distinguish the display content in the curved display area. When rendering, the display content in the curved display area of ​​the application's rectangular display screen is shaded at a low rate, which does not reduce the perceptible output image quality.

[0186] Therefore, in one embodiment, the self-occlusion area of ​​the device also includes the missing portion of the display screen compared to the rectangular display screen (the top corner of the rectangle missing from the rounded corner portion) and the curved display portion.

[0187] In summary, in one embodiment, the device self-occlusion area includes the portion of the rectangular display screen that is blocked by the device itself. That is, the device self-occlusion area includes the portion of the normal display area of ​​the display screen that is missing compared to the rectangular display screen (non-normal display area). The non-normal display area can be a combination of one or more of the area corresponding to the front camera, the rectangular top corner missing from the rounded corner area of ​​the display screen, and the curved display area of ​​the display screen.

[0188] In some embodiments, the device self-occlusion area is drawn according to the occlusion of the rectangular display screen by the current device (compared with the rectangular display screen according to the shape of the normal display area of ​​the display screen).

[0189] For example, based on the occlusion of the rectangular display screen when the application is displayed in full screen by the current device, a device self-occlusion image is pre-generated. The device self-occlusion image includes an interface graphic whose size and shape are consistent with the rectangular display screen. In the interface graphic, based on blocks of different colors, the occlusion part (non-normal display area) of the rectangular display screen by the current device is marked.

[0190] Fig.13 Shown is a flow chart of drawing a guidance mask map according to an embodiment of the present application.

[0191] S1300, determining whether there is a pre-stored device component self-occlusion image.

[0192] If not present, the process ends.

[0193] If it exists, execute S1310.

[0194] In some embodiments, the execution of S1300 is triggered during the initialization phase of VRS.

[0195] In other embodiments, the execution of S1300 is triggered during the device startup phase.

[0196] S1310, determining whether the device component self-occlusion image has been loaded.

[0197] If loaded, execute S1320.

[0198] S1320, passing the memory pointer of the loaded device component self-occlusion image to the thread for drawing the guidance mask image.

[0199] If not loaded, execute S1330.

[0200] S1330, reading the device component self-occlusion image stored in the memory.

[0201] S1320 is executed after S1330.

[0202] S1340 , based on the memory pointer of the loaded device component self-occlusion image, obtain the device component self-occlusion image, and draw a guidance mask image according to the device component self-occlusion image.

[0203] In one embodiment, in S1340, the device self-occlusion image is used as the guidance mask image. In the guidance mask image, the occlusion area (device self-occlusion area) is the abnormal display area of ​​the display screen.

[0204] Alternatively, in another embodiment, in S1340, it is determined whether the non-normal display area in the device self-occlusion image overlaps with the display interface of the first application. If there is overlap, a guidance mask image is generated according to the device self-occlusion image. In the guidance mask image, the area in the non-normal display area that overlaps with the display interface of the first application is marked as an occlusion area (device self-occlusion area).

[0205] For example, when the first application is displayed in full screen, all areas of the non-normal display area will block the display interface of the first application, and all non-normal display areas are marked as device self-blocking areas. When the first application is displayed in split screen with other applications, the part of the non-normal display area that overlaps with the display area of ​​the first application will block the display interface of the first application, and the part of the non-normal display area that overlaps with the display area of ​​the first application is marked as device self-blocking area.

[0206] Take a game application scenario as an example. Figure 4 As shown, screen 400 is a curved screen with bangs, 409 is the bangs part, and the part above dotted line 407 and below dotted line 408 is the curved part of the screen.

[0207] Fig.14 FIG. 4 is a guide mask diagram according to an embodiment of the present application.

[0208] according to Figure 4 The position of the fringe portion 409 and the curved portion in the mobile phone screen 400 can be drawn as follows Fig.14 The guidance mask diagram 1400 is shown. In the guidance mask diagram 1400, the black block areas 1401, 1402 and 1403 are occlusion areas (device device self-occlusion areas).

[0209] Fig.15 Shown is a schematic diagram of a mobile phone display screen according to an embodiment of the present application.

[0210] like Fig.15 As shown, screen 1500 is a hole-punch screen, and 1501 is the hole-punch part.

[0211] Fig.16FIG. 4 is a guide mask diagram according to an embodiment of the present application.

[0212] according to Fig.15 The position of the hole portion 1501 in the mobile phone screen 1500 shown in FIG. 1 can be drawn as follows: Fig.16 The guidance mask diagram 1600 is shown. In the guidance mask diagram 1600, the black block area 1601 is the occlusion area (the device self-occlusion area).

[0213] In S310 , when the occlusion region is identified by combining multiple occlusion region types, after the corresponding occlusion region is determined for each occlusion region type, all occlusion regions are merged into the occlusion region in the guidance mask image.

[0214] For example, Figure 8 The occluded areas (hand occluded areas) 610 and 620 are shown; Fig.10 The occlusion areas (UI occlusion areas) 1001, 1002, 1003, 1004 and 1005 are shown; Fig.12 The occlusion area (other application occlusion area) 1201 shown; and Fig.14 The occlusion area (device self-occlusion area) 1401 shown is merged into Figure 5 The occluded area in the guidance mask shown. ; fused into Figure 4 Guidance mask shown.

[0215] In S310, other methods may also be used to identify the blocked area.

[0216] Taking the game application scenario as an example, during the drawing process of the main game scene, there is an occlusion relationship between different models in the main game scene.

[0217] In one embodiment, in S310, the main scene rendering channel texture is obtained, sliced ​​according to 16x16, all pixels in each slice are read, their RGB values ​​are converted into brightness values, and the average brightness is calculated as a reference threshold. The frequency reference value is obtained from the frequency domain information calculation module and the motion rate reference value is obtained from the motion rate calculation module, and the two are multiplied to obtain the final estimated reference value. The estimated reference value is compared with the reference threshold, and the slices below the reference threshold are marked as occluded areas (slices above the reference threshold are marked as high-rate shading, and slices below the reference threshold are marked as low-rate shading), and a guidance mask map is generated.

[0218] According to the method of the embodiment of the present application, the occlusion area is determined from multiple angles, the effective range of the occlusion area is increased, the stability of the variable rate shading gain is improved, and the scene coverage is expanded.

[0219] S320, based on variable rate shading, render the image to be displayed to generate a display image of the first application, wherein the occluded area is rendered based on at least a first shading rate, and the non-occluded area is rendered based on at least a second shading rate, and the first shading rate is less than the second shading rate.

[0220] In S320 , variable rate shading is applied to the rendering instruction stream and associated data acquired in S300 according to the guidance mask image drawn in S310 to generate a display image of the first application.

[0221] S320 includes:

[0222] Using the guidance mask image as a mask of the image to be displayed, matching the occluded area in the guidance mask image with the image to be displayed in terms of shape and position;

[0223] Based on the guidance mask image, the image to be displayed is rendered, wherein the occluded area is rendered based on at least a first shading rate, and the non-occluded area is rendered based on at least a second shading rate, and the first shading rate is lower than the second shading rate. That is, for the image area corresponding to the occluded area in the guidance mask image in the image to be displayed, low-rate shading is adopted.

[0224] In an implementation of S320, a uniform shading rate is used for the occluded area.

[0225] In one embodiment, the shading rate is described by a matrix. The shading rate a*b means that one pixel is reserved for rendering and shading among a*b pixels in column a and row b.

[0226] For example, in one embodiment, a shading rate of 4*4 is used for the occluded area, that is, for every 4*4 pixels corresponding to the occluded area, 15 pixels are removed and one pixel is retained for rendering and shading. After the rendering and shading is completed, the previously removed pixels are supplemented by sampling the previously retained pixels.

[0227] For example, the guidance mask image and the image to be rendered are set to be equal in ratio of 1:64. According to the shape and position matching relationship between the guidance mask image and the image to be rendered, one pixel in the guidance mask image corresponds to 8x8 pixels of the image to be rendered (one mask pixel guides the shading of 8*8 pixels). For example, if the image to be rendered contains 8m*8n pixels, then the guidance mask image contains m*n mask pixels.

[0228] A pixel shading list is generated according to the guidance mask image. The pixel shading list contains data bits that are consistent with the number of pixels of the image to be rendered. For example, if the image to be rendered contains 8m*8n pixels, then the pixel shading list contains 8m*8n data bits.

[0229] The data bits in the pixel shading list correspond one-to-one to the pixels of the image to be rendered. According to the correspondence between the mask pixels in the guidance mask image and the pixels in the image to be rendered, the correspondence between the mask pixels in the guidance mask image and the data bits in the pixel shading list is set. A mask pixel in the guidance mask image guides the value of 8*8 data bits in the pixel shading list.

[0230] The value of each data bit in the pixel shading list can be 0 or 1. When a mask pixel belongs to the non-occluded area of ​​the guidance mask image, the shading rate of the 8x8 pixels of the image to be rendered corresponding to the mask pixel is configured to be 4*4. That is, when a mask pixel belongs to the non-occluded area of ​​the guidance mask image, the value of the 8*8 data bits in the pixel shading list corresponding to the mask pixel is configured to be 15 of every 16 data bits are set to 0 and one data bit is set to 1.

[0231] During the coloring process, coloring is performed according to the values ​​in the pixel coloring list. Pixels corresponding to the values ​​of the data bits in the pixel coloring list being 0 are eliminated, and pixels corresponding to the values ​​of the data bits in the pixel coloring list being 1 are colored.

[0232] Fig.17 The figure shows the values ​​of 8*8 data bits in a pixel shading list with a shading rate of 4*4.

[0233] like Fig.17 As shown, the values ​​of 15 data bits out of every 16 data bits are set to 0, and the value of one data bit is set to 1.

[0234] Fig.18 The figure shows a schematic diagram of the shading result of 8*8 pixels with a shading rate of 4*4.

[0235] like Fig.18 As shown, only pixels 1801, 1802, 1803, and 1804 are colored, and the remaining pixels are not colored.

[0236] After rendering and shading are completed, the previously discarded pixels are supplemented by sampling the previously retained pixels. That is, the shading results of the colored pixels are used to supplement the previously uncolored pixels.

[0237] Fig.19 Shown is based on Fig.18 The coloring results are colored to supplement the schematic diagram of the results.

[0238] like Fig.19 As shown, the coloring results of pixels 1801, 1802, 1803, and 1804 are added to the surrounding uncolored pixels.

[0239] Taking a game application scenario as an example, the depth value of the main scene corresponding to the low shading rate area is set to 0.0f. When the game draws the main scene, the early-z feature is used to remove the shading overhead of 15 / 16 (93.75%) pixels in the occluded area. Finally, the empty pixels are restored by sampling the adjacent points for interpolation.

[0240] In some embodiments, for image rendering, the graphics system is configured with a depth buffer. The depth buffer is used to store the depth or distance (depth) value of each pixel in the window from an observation plane (near clipping plane). The depth buffer can be implemented by hardware, for example, a fixed storage space is designated as the depth buffer in the hardware memory; the depth buffer can be implemented by software, for example, when rendering, a storage address space is dynamically designated as the depth buffer.

[0241] Since the depth buffer can store the depth values ​​of all pixels of the image to be rendered, the depth value can be replaced with the data bit of the pixel shading list, and the pixel shading list can be directly saved in the depth buffer without reconfiguring the data structure.

[0242] Therefore, in one embodiment, the pixel shading list is implemented based on a depth buffer. For example, depth buffer data is generated according to a guidance mask image, and the depth buffer data is stored in the depth buffer.

[0243] A mask pixel in the guidance mask image guides the depth value of 8*8 pixels in the depth buffer data. When a mask pixel belongs to the non-occluded area of ​​the guidance mask image, the depth value of 8x8 pixels in the depth buffer data corresponding to the mask pixel is configured, and 15 depth values ​​are set to 0 and one depth value is set to 1 in every 16 depth values. (0 represents the closest depth and 1 represents the farthest depth)

[0244] During the shading process, shading is performed according to the depth buffer data. Pixels corresponding to the depth value of 0 in the depth buffer data are eliminated, and pixels corresponding to the depth value of 1 in the depth buffer data are shaded.

[0245] In another implementation of S320, a uniform shading rate is not used for the occluded area, but the occluded area is divided into multiple sub-areas, and different shading rates are used for different sub-areas.

[0246] For example, in one embodiment, the occlusion area is divided into multiple sub-areas according to different types of occlusion areas. For example, the hand occlusion area, the user interface occlusion area, the other application occlusion area, and the device self-occlusion area correspond to four sub-areas respectively; for another example, the hand occlusion area and the device self-occlusion area correspond to one sub-area, and the user interface occlusion area and the other application occlusion area correspond to one sub-area.

[0247] In some embodiments, in the guidance mask image, different sub-regions in the occluded region are marked by pixel values ​​of mask pixels, and different pixel values ​​correspond to different sub-regions. A corresponding shading rate is adopted according to the pixel values ​​of mask pixels in the guidance mask image.

[0248] In some embodiments, different mask pixel values ​​are set to correspond to different shading rates. As shown in Table 1:

[0249] Table 1

[0250]

[0251]

[0252] Set one pixel in the guidance mask image to correspond to 8x8 pixels of the display screen (one mask pixel guides the shading of 8*8 pixels). When drawing the guidance mask image, set the corresponding pixel value for each mask pixel in the guidance mask image according to the shading rate allocation. For example, for mask pixels in the non-occluded area, set the mask pixel value to 0 (shading rate is 1*1, normal shading, no low-rate shading); for mask pixels in the hand occluded area, set the mask pixel value to 9 (shading rate is 4*2); for mask pixels in the UI occluded area and other application occluded areas, set the mask pixel value to 10 (shading rate is 4*4); for mask pixels in the self-occluded area of ​​the device, set the mask pixel value to 16 (shading rate is 8*8).

[0253] The depth buffer data is generated according to the guidance mask map. One mask pixel guides the depth value of 8*8 pixels in the depth buffer data. When the mask pixel value is 0, the depth value of the corresponding 8*8 pixels in the depth buffer is 1; when the mask pixel is 9, the depth value of 7 pixels in every 4*2 pixels in the corresponding 8x8 pixels in the depth buffer is set to 0, and the depth value of one pixel is set to 1 (the shading rate is 4*2).

[0254] According to the method of the embodiment of the present application, the occlusion area of ​​the application is identified and VRS is performed, which can reduce the amount of rendering calculations without reducing the user experience, thereby improving the rendering performance and reducing the display power consumption. According to the method of the embodiment of the present application, since the effective range of the occlusion area is large and the user perception is small, the VRS benefits are stable and the scene coverage is wide.

[0255] In one embodiment, frame-based variable rate shading is employed.

[0256] Fig. 20 FIG. 4 is a flowchart of a VRS implementation according to an embodiment of the present application.

[0257] S2000, drawing a guidance mask map of the N-1th frame.

[0258] S2010, obtaining the rendering instruction stream and associated data of the Nth frame.

[0259] S2020, according to the guidance mask map drawn by S2000, apply variable rate shading to the rendering instruction stream and associated data obtained by S2010.

[0260] S2030: Identify the occluded area of ​​the Nth frame.

[0261] S2040, draw a guidance mask map of the Nth frame.

[0262] In one embodiment, VRS is implemented based on software, so that VRS can be used on different hardware platforms.

[0263] Fig.21 Schematic diagram of a system architecture according to an embodiment of the present application.

[0264] like Fig.21 As shown, the application layer 2210 is used to run applications (eg, game application 2211, video playback application 2212, etc.).

[0265] The graphics API layer 2220 is used to load a graphics driver, which is a driver program capable of drawing graphics, such as opengl and vulkan.

[0266] The operating system (OS) kernel layer 2240 is used to load the system kernel and drivers.

[0267] The chip layer 2250 is the hardware layer of the electronic device, which includes a CPU and a GPU.

[0268] The VRS layer 2230 is used to implement the VRS method flow of the application embodiment. The VRS layer 2230 is included in the instruction reassembly layer in the terminal OS system, and includes a data acquisition module 2231, an occlusion region recognition module 2232, and a variable rate shading module 2238.

[0269] The data acquisition module 2231 is used to intercept the graphics API call from the graphics API layer 2220, intercept the graphics instruction data stream, collect and calculate rendering related data, such as the main scene rendering channel and UI rendering channel of the current frame, and cache the relevant data. The data acquisition module 2231 uses the reserved interface of the graphics driver to intercept all driver instructions before the engine layer calls the driver layer instructions, and caches them into an instruction stream. Some of the driver instructions contain the main scene rendering channel information and UI rendering channel information during the drawing process.

[0270] Take a game application scenario as an example. Fig. 22 2 is a schematic diagram of the operation flow of the data acquisition module 2231 according to an embodiment of the present application.

[0271] S2300, the game engine calls the graphics driver interface.

[0272] S2310, the data acquisition module 2231 determines whether the game rendering needs to be optimized.

[0273] If no, the process ends.

[0274] If yes, execute S2320.

[0275] S2320, the data acquisition module 2231 intercepts the graphics instruction stream.

[0276] S2321, the data acquisition module 2231 analyzes the graphics instruction stream to obtain information such as the game main scene rendering channel and the UI rendering channel.

[0277] S2322, the data acquisition module 2231 caches the data acquired in S2321.

[0278] The occlusion area recognition module 2232 performs recognition at the end of the frame, parses the touch events and other application window information received in the current frame, calculates the hand occlusion and other application window occlusion areas; parses the UI rendering channel at the same time, calculates the UI occlusion area; loads the device component occlusion area; and finally comprehensively draws the guidance mask map (which areas use low-rate shading).

[0279] The occlusion area recognition module 2232 includes a hand occlusion area prediction module 2233 , other window recognition module 2234 , a UI occlusion area recognition module 2235 , a device occlusion recognition module 2236 and an occlusion area drawing module 2237 .

[0280] The variable rate shading module 2238 is used to decide which areas to use which rate to shade when drawing the main scene in the next frame based on the guidance mask image, reorganize the intercepted instruction stream, and transmit it to the system kernel and driver (operating system (OS) kernel layer 2240).

[0281] Take a game application scenario as an example. Fig.23 2 is a schematic diagram of the operation flow of the variable rate shading module 2238 according to an embodiment of the present application.

[0282] S2400 , obtaining the graphics instruction stream cache data cached by the data acquisition module 2231 .

[0283] S2410, obtaining the depth cache data of the game main scene rendering channel according to the graphics instruction stream cache data.

[0284] S2420 , according to the guidance mask image generated by the occlusion area identification module 2232 , write a corresponding depth value (0 or 1) for the occlusion area in the depth buffer data of the main game scene.

[0285] For example, a mask pixel controls the shading rate of the 8x8 area pixels in the depth cache; when the mask pixel value is 10, the shading rate of the corresponding 8x8 pixel area in the depth cache should be 4x4, which means that only 1 pixel is retained for shading out of 16 pixels, the depth value of the shading pixel value is set to 1, and the depth values ​​of the remaining 15 pixels that need to be removed from shading are set to 0 (0 represents the closest depth, and 1 represents the farthest depth).

[0286] S2430, rendering the main scene of the game, for pixels with a depth value of 0, the pixel will not be shaded through the early-z mechanism, thereby saving the overhead of FS execution.

[0287] S2440, after the main scene is rendered, the colors of the uncolored pixels in the occluded area are filled in through a sampling interpolation algorithm according to the guidance mask generated by the occluded area drawing module.

[0288] According to the method of the embodiment of the present application, VRS is implemented in the instruction reorganization layer in the terminal OS system, and there is no need to adapt the application of the application layer. The implementation of VRS does not depend on the application, thereby ensuring that the VRS method of the embodiment of the present application can be ported to any platform and applied to any application, thereby improving the portability and versatility of VRS.

[0289] In the description of the embodiments of the present application, for the convenience of description, the device is described by dividing it into various modules according to its functions. The division of each module is merely a division of logical functions. When implementing the embodiments of the present application, the functions of each module can be implemented in the same or one or more software and / or hardware.

[0290] The device proposed in the embodiment of the present application can be fully or partially integrated into a physical entity during actual implementation, or it can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the detection module can be a separately established processing element, or it can be integrated in a chip of an electronic device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. In the implementation process, each step of the above method or each of the above modules can be completed by an integrated logic circuit of hardware in a processor element or instructions in the form of software.

[0291] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0292] An embodiment of the present application further provides an electronic device.

[0293] Fig.24 Schematic diagram of the structure of an electronic device according to an embodiment of the present application.

[0294] like Fig.24 As shown, the electronic device 2500 includes a memory 2502 for storing computer program instructions and a processor 2501 for executing the program instructions, wherein when the computer program instructions are executed by the processor 2501, the electronic device 2500 is triggered to execute the method steps described in the embodiments of the present application.

[0295] In one embodiment of the present application, the above-mentioned one or more computer programs are stored in the above-mentioned memory 2502, and the above-mentioned one or more computer programs include instructions. When the above-mentioned instructions are executed by the above-mentioned electronic device 2500, the above-mentioned electronic device 2500 executes the method steps described in the embodiment of the present application.

[0296] It is understandable that the structural description of the electronic device 2500 in the embodiment of the present application does not constitute a specific limitation on the electronic device 2500. In other embodiments of the present application, the electronic device 2500 may include other components besides the processor 2501 and the memory 2502.

[0297] The processor 2501 may be a device on a chip (SOC), and the processor 2501 may include a central processing unit (CPU), and may further include other types of processors.

[0298] The processor involved in processor 2501 may include, for example, a CPU, a DSP, a microcontroller or a digital signal processor, and may also include a GPU, an embedded neural network processor (Neural-network Process Units, NPU) and an image signal processor (Image Signal Processing, ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASICs, or one or more integrated circuits for controlling the execution of the program of the technical solution of the present application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

[0299] The processor 2501 may include one or more processing units. For example, the processor may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 2500 may also include one or more processors 2501. The controller may generate an operation control signal based on the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0300] In some embodiments, the processor 2501 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc. Among them, the USB interface is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface can be used to connect a charger to charge an electronic device, and can also be used to transmit data between an electronic device and a peripheral device.

[0301] The electronic device 2500 may also include an external memory interface, which is used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 2501 through the external memory interface to implement a data storage function. For example, files such as music and videos are stored in the external memory card.

[0302] The memory 2502 may include a code storage area and a data storage area. The code storage area may store an operating system. The data storage area may store data created during the use of the electronic device 2500, etc. In addition, the memory 2502 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, a flash memory component, a universal flash storage (UFS), etc.

[0303] The memory 2502 may be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or may be any computer-readable medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer.

[0304] The processor 2501 and the memory 2502 may be integrated into one processing device, or more commonly, may be independent components.

[0305] The devices, apparatuses, and modules described in the embodiments of the present application may be implemented by computer chips or entities, or by products having certain functions.

[0306] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0307] In the several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0308] In one embodiment of the present application, a computer-readable storage medium is further provided, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the embodiment of the present application.

[0309] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is run on a computer, the computer executes the method provided by the embodiment of the present application.

[0310] The embodiments described in this application are described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to the embodiments of the application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0311] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0312] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0313] It should also be noted that in the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0314] In the embodiments of the present application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, commodity or device including the element.

[0315] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0316] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0317] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments of the present application can be implemented by a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0318] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0319] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image rendering method based on variable rate shading, applied to electronic equipment, characterized in that: The method comprises: Determine an occluded area and a non-occluded area on the image to be displayed of the first application, wherein the occluded area is an occluded area in the image to be displayed, and the non-occluded area is an unoccluded area in the image to be displayed; Based on variable rate shading, the image to be displayed is rendered to generate a display image of the first application, wherein the occluded area is rendered based on at least a first shading rate, and the non-occluded area is rendered based on at least a second shading rate, and the first shading rate is less than the second shading rate.

2. The method according to claim 1, characterized in that The occlusion area includes any one or more combinations of a hand occlusion area, a user interface occlusion area, other application occlusion areas, and a device self-occlusion area, wherein: The hand-blocking area includes a hand-covered area on the display screen determined in response to a touch operation of a user or a portion of the hand-covered area; The user interface blocking area includes, in the image area of ​​the user interface of the first application, an area that blocks the application display content of the first application; The other application blocking area includes the window area of ​​the foreground active window of the second application, or a part of the window area of ​​the foreground active window of the second application; The device self-occlusion area includes an area of ​​the display screen of the electronic device corresponding to the front camera.

3. The method according to claim 2, characterized in that The determining of the blocked area and the non-blocked area on the image to be displayed of the first application includes: In response to a touch event of a user, analyzing the touch event to obtain touch area information; Determining the hand coverage area according to the touch area information; Mark the hand coverage area as the hand obstruction area; or, mark the area in the hand coverage area that overlaps with the display interface of the first application as the hand obstruction area.

4. The method according to claim 2, characterized in that: The determining of the blocked area and the non-blocked area on the image to be displayed of the first application includes: Determine an image area of ​​the user interface according to a display image of a previous frame; When the image area of ​​the user interface causes occlusion of the application content of the first application, the image area of ​​the user interface that causes occlusion is marked as the user interface occlusion area.

5. The method according to claim 4, characterized in that The determining of the blocked area and the non-blocked area on the image to be displayed of the first application includes: Identify the previous frame's UI rendering pass; Sampling the user interface rendering channel to obtain sampled pixels; When the sampled pixel has a color value and the transparency is greater than a preset threshold, the image area corresponding to the sampled pixel is marked as the user interface occlusion area.

6. The method according to claim 2, characterized in that The determining of the blocked area and the non-blocked area on the image to be displayed of the first application includes: Obtaining window information of the foreground active window of the second application; Drawing a window area of ​​the foreground active window according to the window information of the foreground active window of the second application; Mark the window area of ​​the foreground active window as the other application obstruction area; or, mark the area of ​​the window area of ​​the foreground active window that overlaps with the display interface of the first application as the other application obstruction area.

7. The method according to claim 2, characterized in that The determining of the blocked area and the non-blocked area on the image to be displayed of the first application includes: Comparing the shape of a normal display area of ​​the display screen with a rectangular display picture, and drawing the abnormal display area of ​​the display screen according to the comparison result; The non-normal display area is marked as a device self-occlusion area; or, an area in the non-normal display area that overlaps with the display interface of the first application is marked as the device self-occlusion area.

8. The method according to any one of claims 1 to 7, characterized in that The shielding area includes at least a first area; The step of rendering the image to be displayed based on variable rate shading to generate a display image of the first application includes: According to the first shading rate, a first pixel region is selected from the first region; the first region includes a first pixel region and a second pixel region; Coloring the pixels of the first pixel area; and filling the second pixel area with the coloring result based on the coloring result of the first pixel area.

9. The method according to claim 8, characterized in that The shielding area also includes at least a second area; The step of rendering the image to be displayed based on variable rate shading to generate a display image of the first application further includes: The second region is colored using a third coloring rate, where the third coloring rate is different from the first coloring rate and is smaller than the second coloring rate.

10. The method according to claim 9, characterized in that The second area and the first area correspond to different shielding area types respectively.

11. The method according to any one of claims 1 to 10, characterized in that The method comprises: Determine an occluded area and a non-occluded area on a display image of the first application in the N-1th frame; Obtaining a rendering instruction stream and associated data of an image to be displayed of the first application in the Nth frame; Performing display image rendering according to the rendering instruction stream and associated data of the image to be displayed of the first application of the Nth frame to generate a display image of the first application of the Nth frame, wherein the variable rate shading is applied according to the occluded area and the non-occluded area on the image to be displayed of the first application of the N-1th frame; An occluded area and a non-occluded area on a display image of the first application in the Nth frame are determined.

12. An electronic device, characterized in that: The electronic device comprises a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method steps as described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 11.

Images