Image rendering method based on variable rates shading, and electronic device

Abstract

An image rendering method based on variable rates shading, and an electronic device. The method comprises: determining a shielding area and a non-shielding area on an image to be displayed of a first application (S310), wherein the shielding area is a shielded area in the image to be displayed, and the non-shielding area is an unshielded area in the image to be displayed; and on the basis of variable rates shading, rendering the image to be displayed, so as to generate a display image of the first application, wherein the shielding area is rendered at least on the basis of a first shading rate, the non-shielding area is rendered at least on the basis of a second shading rate, and the first shading rate is less than the second shading rate (S320). According to the method in the embodiments of the present application, a shielding area of an application is identified and variable rates shading is performed, such that the rendering calculation amount can be reduced without compromising the user experience, thereby improving the rendering performance and reducing the display power consumption.

Description

Image rendering method and electronic device based on variable rate shading

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 25, 2023, with application number 2023112566777 and application name “An image rendering method and electronic device based on variable rate shading”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to an image rendering method and electronic device based on variable rate shading. Background Art

[0003] In the field of image display, shading is one of the most computationally expensive parts of the rendering pipeline (render pass). As screen resolution and refresh rate increase, the demand for shading calculations is growing significantly, and even the most powerful GPUs cannot meet the growing demand.

[0004] To address this challenge, some graphics processing unit (GPU) manufacturers have introduced a more flexible shading mechanism in their chipsets, called variable rate shading (VRS).

[0005] VRS can perform shading at a coarser frequency than pixels for a specified image area, thereby allocating rendering performance or computing power at different rendering image rates. In other words, for a specified image area, VRS can shade a group of pixels as a unit and then copy the result to all samples in the group, thereby reducing the computational cost of shading the specified area and improving rendering performance.

[0006] While VRS has many advantages, including a flexible shading mechanism, decoupling of pixels and shading frequency, and improved rendering performance without compromising perceptible output quality, it still has many shortcomings. In VRS applications, to ensure that there is no degradation in perceptible output quality while improving performance, it is necessary to identify areas that can be shaded at a low rate and maximize these areas to achieve optimal VRS results. Existing VRS solutions, however, use a limited and sparse method for identifying regions, resulting in limited performance gains.

[0007] Therefore, a new VRS method is needed to improve rendering performance.

[0008] Summary of the Invention

[0009] In order to solve the problem of how to improve rendering performance, the present application provides an image rendering method and electronic device based on variable rate shading, and the present application also provides a computer-readable storage medium.

[0010] The embodiments of this application adopt the following technical solutions:

[0011] In a first aspect, the present application provides an image rendering method based on variable rate shading, which is applied to an electronic device. The method includes: determining an occluded area and a non-occluded area on an image to be displayed of a first application, wherein the occluded area is an occluded area in the image to be displayed, and the non-occluded area is an unoccluded area in the image to be displayed; rendering the image to be displayed based on variable rate shading to generate a display image of the first application, wherein the occluded area is rendered based on at least a first shading rate, and the non-occluded area is rendered based on at least a second shading rate, and the first shading rate is less than the second shading rate.

[0012] According to the method of the embodiment of the present application, the occlusion area of ​​the application is identified and variable rate shading is performed, which can reduce the rendering calculation amount without reducing the user experience, thereby improving rendering performance and reducing display power consumption. According to the method of the embodiment of the present application, compared with the image motion area, the occlusion area has a larger effective range and is less perceived by the user. Therefore, the variable rate shading benefits are stable and the scene coverage is wider.

[0013] In an implementation of the first aspect, the occlusion area includes any one or more combinations of a hand occlusion area, a user interface occlusion area, an other application occlusion area, and a device self-occlusion area, wherein: the hand occlusion area includes the hand coverage area or a part of the hand coverage area covered by the user's hand on the display screen when the user performs a touch operation; the user interface occlusion area includes the area in the image area of ​​the user interface of the first application that blocks the application display content of the first application; the other application occlusion area includes the window area of ​​the foreground active window of the second application, or a part of the window area of ​​the foreground active window of the second application; the device self-occlusion area includes the area of ​​the display screen of the electronic device corresponding to the front camera.

[0014] According to the method of the embodiment of the present application, the occlusion area is determined from multiple angles, the effective range of the occlusion area is increased, the stability of the variable rate shading gain is improved, and the scene coverage is expanded.

[0015] In one implementation of the first aspect, determining the occluded area and the non-occluded area on the image to be displayed of the first application includes: responding to a user's touch event, parsing the touch event to obtain touch area information; determining the hand coverage area based on the touch area information; marking the hand coverage area as a hand occlusion area; or, marking the area in the hand coverage area that overlaps with the display interface of the first application as a hand occlusion area.

[0016] In one implementation of the first aspect, determining the occluded area and the non-occluded area on the image to be displayed of the first application includes: determining the image area of ​​the user interface based on the display image of the previous frame; when the image area of ​​the user interface will cause occlusion of the application content of the first application, marking the image area of ​​the user interface that causes occlusion as the user interface occluded area. In one implementation of the first aspect, determining the occluded area and the non-occluded area on the image to be displayed of the first application includes: identifying the user interface rendering channel of the previous frame; sampling the user interface rendering channel to obtain sampled pixels; when the sampled pixels have color values ​​and the transparency is greater than a preset threshold, marking the image area corresponding to the sampled pixels as the user interface occluded area.

[0017] In an implementation of the first aspect, determining the blocked area and the non-blocked area on the image to be displayed of the first application includes: obtaining the window information of the foreground active window of the second application; drawing the window area of ​​the foreground active window according to the window information of the foreground active window of the second application; marking the window area of ​​the foreground active window as an blocked area for other applications; or marking the area in the window area of ​​the foreground active window that overlaps with the display interface of the first application as an blocked area for other applications.

[0018] In one implementation of the first aspect, determining the occluded area and the non-occluded area on the image to be displayed of the first application includes: comparing the shape of the normal display area of ​​the display screen with the rectangular display screen, and drawing the abnormal display area of ​​the display screen according to the comparison result; marking the abnormal display area as a self-occluded area of ​​the device; or marking the area in the abnormal display area that overlaps with the display interface of the first application as a self-occluded area of ​​the device.

[0019] In one implementation of the first aspect, the occluded area includes at least a first area. Rendering an image to be displayed based on variable rate shading to generate a display image for the first application includes: selecting a first pixel area from the first area according to a first shading rate; the first area includes a first pixel area and a second pixel area; shading pixels in the first pixel area; and filling the second pixel area with a shading result based on the shading result of the first pixel area.

[0020] In one implementation of the first aspect, the occluded area also includes at least a second area. Rendering the image to be displayed based on variable rate shading to generate a display image for the first application further includes: shading the second area using a third shading rate, where the third shading rate is different from the first shading rate and is less than the second shading rate.

[0021] In an implementation manner of the first aspect, the second area and the first area correspond to different occlusion area types respectively.

[0022] In an implementation of the first aspect, the method includes: determining the occluded area and the non-occluded area on the display image of the first application of the N-1th frame; obtaining the rendering instruction stream and associated data of the image to be displayed of the first application of the Nth frame; performing display image rendering according to the rendering instruction stream and associated data of the image to be displayed of the first application of the Nth frame, and generating the display image of the first application of the Nth frame, wherein variable rate shading is applied according to the occluded area and the non-occluded area on the image to be displayed of the first application of the N-1th frame; determining the occluded area and the non-occluded area on the display image of the first application of the Nth frame.

[0023] In a second aspect, the present application provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method steps of the first aspect.

[0024] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] FIG1 is a flowchart illustrating a VRS execution process according to an embodiment of the present application;

[0026] FIG2 is a flowchart illustrating a VRS execution process according to an embodiment of the present application;

[0027] FIG3 is a flowchart illustrating a VRS execution process according to an embodiment of the present application;

[0028] FIG4 is a schematic diagram showing a display effect of a mobile phone screen according to an embodiment of the present application;

[0029] FIG5 shows a guide mask diagram according to an embodiment of the present application;

[0030] FIG6 is a flowchart showing a method of drawing a guidance mask according to an embodiment of the present application;

[0031] FIG7 is a schematic diagram showing the touch positions of the user's fingers in the game state;

[0032] FIG8 shows a guide mask diagram according to an embodiment of the present application;

[0033] FIG9 is a flowchart showing a method of drawing a guidance mask according to an embodiment of the present application;

[0034] FIG10 shows a guide mask diagram according to an embodiment of the present application;

[0035] FIG11 is a flow chart showing a method of drawing a guidance mask according to an embodiment of the present application;

[0036] FIG12 shows a guide mask diagram according to an embodiment of the present application;

[0037] FIG13 is a flow chart showing a method of drawing a guidance mask according to an embodiment of the present application;

[0038] FIG14 shows a guide mask diagram according to an embodiment of the present application;

[0039] FIG15 is a schematic diagram of a mobile phone display screen according to an embodiment of the present application;

[0040] FIG16 shows a guide mask diagram according to an embodiment of the present application;

[0041] FIG17 is a diagram showing the values ​​of 8*8 data bits in a pixel shading list with a shading rate of 4*4;

[0042] FIG18 is a schematic diagram showing the shading result of 8*8 pixels with a shading rate of 4*4;

[0043] FIG19 is a schematic diagram showing a coloring supplement result based on the coloring result of FIG18 ;

[0044] FIG20 is a flowchart of VRS implementation according to an embodiment of the present application;

[0045] FIG21 is a schematic diagram of a system architecture according to an embodiment of the present application;

[0046] FIG22 is a schematic diagram of the operation flow of the data acquisition module 2231 according to an embodiment of the present application;

[0047] FIG23 is a schematic diagram of the operation flow of the variable rate shading module 2238 according to an embodiment of the present application;

[0048] FIG24 is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0051] In VRS application scenarios, the key to improving rendering performance without reducing the perceptible output quality is to identify areas that can be shaded at a low rate.

[0052] FIG1 is a flowchart showing a VRS execution process according to an embodiment of the present application.

[0053] In one embodiment, the GPU executes the following process as shown in FIG1 to implement VRS.

[0054] S100, obtaining regional motion data of an image frame.

[0055] In a dynamic image display scene, there is a moving object in the dynamic image picture, and the image area of ​​the moving object in the picture is the moving area of ​​the image frame.

[0056] For example, in a video of a falling object, when the shooting position and viewing angle remain unchanged (the background in the video remains unchanged), the falling object falls from the top of the screen to the bottom of the screen. Between different image frames of the video, the position of the falling object in the image frame constantly changes, and the moving area in the dynamic image is the image area corresponding to the falling object. For example, in the first image frame, the image area of ​​the falling object is the first area; in the second image frame (the first image frame is the previous frame of the second image frame), the image area of ​​the falling object is the second area; compared to the first image frame, the moving area of ​​the second image frame is the combination of the first area and the second area.

[0057] Regional motion data is data used to describe the motion region in a dynamic image. For example, in one embodiment, the regional motion data of an image frame may include the position of the motion region, the magnitude of the position change of the moving object in the motion region, the speed of the position change of the moving object in the motion region, etc.

[0058] S110 , calculating a motion value of each motion region of the image frame according to the region motion data.

[0059] In some examples, the motion value is used to quantitatively describe the motion state of a moving object in a motion region. Generally, the faster the moving object moves, the greater the motion value.

[0060] For example, in a video of a falling object, the faster the object falls, the larger the corresponding motion value.

[0061] S120 , determining a shading rate of the motion region based on the motion value of the motion region.

[0062] In dynamic image display scenarios, even if the display quality (e.g., display resolution) of the image frames is consistent, the image of moving objects is blurrier to the human eye than that of non-moving objects in the image. Therefore, actively reducing the image quality of moving objects does not affect the visual effect perceived by the user's eyes.

[0063] In some embodiments, for moving areas in an image frame whose motion speed exceeds the dynamic visual resolution capability of the human eye (the motion value exceeds a preset threshold), a relatively low shading rate can be used (actively reducing the image display fineness); for non-moving areas of the image frame and whose motion speed does not exceed the dynamic visual resolution capability of the human eye, a relatively high shading rate can be used.

[0064] Since the faster an object moves, the blurrier it appears to human vision, a relatively low shading rate can be used for moving areas with high speed (higher motion value), and a relatively high shading rate can be used for moving areas with low speed (lower motion value).

[0065] S130 , rendering the graphics primitives at the shading rate determined in S120 to generate color data for the motion area.

[0066] The process shown in Figure 1 effectively implements variable rate shading, improving rendering performance and reducing power consumption to a certain extent. However, there are limitations to making shading rate decisions based solely on motion values. When the scene is slow or stationary, the benefits of the VRS solution decrease.

[0067] FIG2 is a flowchart showing a VRS execution process according to an embodiment of the present application.

[0068] In one embodiment, the GPU executes the following process as shown in FIG. 2 to implement VRS.

[0069] S200 , obtaining first content data rendered by a current image frame and second content data rendered by an image frame previous to the current image frame.

[0070] S210 : Compare pixel values ​​of a current image frame and a previous image frame according to the first content data and the second content data.

[0071] S220, filtering the comparison result of S210 to determine the image portion (high-frequency details) corresponding to pixels whose pixel value differences are lower than a preset threshold, and the image portion (non-high-frequency details) corresponding to pixels whose pixel value differences are greater than or equal to the preset threshold in the current image frame.

[0072] Generally, due to the persistence of vision phenomenon, the faster the image changes, the less the human eye can discern image details. For example, consider a video showing a gradual transition from image A to image B. The faster the transition, the less the human eye can discern the details of the change.

[0073] Therefore, when the speed of picture changes exceeds the dynamic recognition ability of the human eye, actively reducing the image fineness in the picture change area will not affect the visual effect perceived by the user's eyes.

[0074] In an application scenario of displaying dynamic images, for a certain image area, the smaller the image difference between two adjacent image frames, the slower the picture changes in the image area, that is, the human eye can more clearly identify the image details of the image area.

[0075] Therefore, a preset threshold is set based on the human eye's dynamic recognition ability. In S220, the image portion (high-frequency details) corresponding to pixels in the current image frame whose pixel value difference is lower than the preset threshold is obtained. The high-frequency details are the image details that the human eye can recognize. Correspondingly, the remaining non-high-frequency details are the image details that the human eye cannot clearly recognize. Reducing the image display effect of non-high-frequency details does not affect the visual effect perceived by the user's eyes.

[0076] S230 , applying variable rate shading to the current image frame, using low rate shading for non-high frequency details.

[0077] According to the process shown in Figure 3, the variable rate shading function can be effectively implemented, which can improve rendering performance and reduce power consumption to a certain extent. However, there are limitations to determining the shading rate based solely on high-frequency details. When the picture is full of high-frequency details, the above VRS solution may not be beneficial.

[0078] In order to improve rendering performance without degrading the perceptible output image quality, an embodiment of the present application proposes a VRS method. In the method of the embodiment of the present application, before displaying an image for the current application (APP) (first application), it is necessary to render the image to be displayed to generate a display image for display.

[0079] Before rendering the image to be displayed, an occluded area and a non-occluded area on the image to be displayed of the first application are determined, wherein the occluded area is an occluded area in the image to be displayed, and the non-occluded area is an unoccluded area in the image to be displayed.

[0080] When rendering the image to be displayed, low-rate shading is used for occluded areas, thereby improving rendering performance without reducing the perceptible output quality.

[0081] FIG3 is a flowchart showing a VRS execution process according to an embodiment of the present application.

[0082] In one embodiment, the GPU executes the following process as shown in FIG3 to implement VRS.

[0083] S310: Determine an occluded area and a non-occluded area on an image to be displayed of a first application.

[0084] In some examples, the first application is an application that currently needs to render a display image. That is, the first application is an application running in the foreground and currently displaying an image. The first application outputs an image to be displayed, and the image rendering system renders the image to be displayed to generate a display image (current image frame).

[0085] The first application can be displayed in full screen (the application currently running in the foreground is only the first application), and the first application can also be displayed in split screen with other applications (the applications currently running in the foreground are multiple applications including the first application).

[0086] The blocked area on the image to be displayed of the first application is the blocked area on the image to be displayed of the first application (current display interface); the non-blocked area on the image to be displayed of the first application is the unblocked area on the image to be displayed of the first application (current display interface).

[0087] In one embodiment, in S310 , a guidance mask map is drawn, and the occlusion area and the non-occlusion area are described by the guidance mask map.

[0088] In some examples, the guidance mask image includes an interface graphic whose shape is consistent with the image to be rendered of the first application. The size of the interface graphic is consistent with the size of the image to be rendered, or the interface graphic is proportionally enlarged or reduced in size.

[0089] The interface graphics are divided into occlusion areas and non-occlusion areas by different color blocks. The occlusion area can be a connected area or a collection of multiple connected areas; the non-occlusion area can be a connected area or a collection of multiple connected areas.

[0090] FIG4 is a schematic diagram showing a display effect of a mobile phone screen according to an embodiment of the present application.

[0091] As shown in Figure 4, 400 is the mobile phone screen. A 3D game scene from a game application is displayed full screen on screen 400; 401 is the game tool component; 402 is the character status component; 403, 404, and 405 are the skill button components; and 406 is the game assistant component. Screen 400 is a curved screen with a notch. 409 is the notch portion, and the curved portion of the screen is above the dotted line 407 and below the dotted line 408.

[0092] FIG. 5 shows a guide mask diagram according to an embodiment of the present application.

[0093] A guidance mask is drawn for the display interface of screen 400 shown in FIG4 . The drawn guidance mask is shown as 501 . Rectangle 501 is an interface graphic that is the same size and shape as the full-screen display interface of screen 400 . The black block graphic area within rectangle 501 corresponds to the occluded area of ​​the display interface of screen 400 ; the white block graphic area within rectangle 501 corresponds to the non-occluded area of ​​the display interface of screen 400 .

[0094] In one embodiment, before S310, a graphics application programming interface (API) call is intercepted to obtain a rendering instruction stream and associated data, and an occluded area and a non-occluded area on the image to be displayed of the first application are determined based on the rendering instruction stream and the associated data.

[0095] In some examples, the rendering instruction stream contains rendering instructions required for image rendering. This can include parameters such as interface size, light direction, light intensity, light color, fiber loss parameters, the orientation of the model to be rendered, and the orientation of the material to be rendered. Associated data contains data required for image rendering. This can include data for the model to be rendered, material data to be rendered, and texture data.

[0096] In some embodiments, the occlusion area may be any one or more combinations of the following situations.

[0097] 1: Hand occluded area.

[0098] When a user performs touch operations on an app, their hand (including fingers, palm, and wrist) is positioned between the display and their eyes. This blocks their view, partially obscuring the display. During rendering, areas obscured by the hand are shaded at a lower rate without degrading the perceived output quality.

[0099] Therefore, in one embodiment, the hand occlusion area includes the hand coverage area or a portion of the hand coverage area covered by the user's hand on the display screen when the user performs a touch operation.

[0100] Taking a game application scenario as an example, the first application is a game application, and the first application is displayed in full screen.

[0101] When in game mode, the user holds the phone with both hands (the phone is in landscape orientation), and the fingers of both hands touch the lower left and lower right corners of the phone screen respectively.

[0102] FIG6 shows a flow chart of drawing a guidance mask according to an embodiment of the present application.

[0103] S400: Obtain the current touch event.

[0104] In some examples, in S400 , when rendering the current frame, at the end of the previous frame or the beginning of the current frame, a current motion event is obtained through a system framework layer.

[0105] In an implementation of S400 , the current touch event is a touch event currently occurring; or, when no touch event has occurred at the current moment, the latest touch event is a touch event that occurred before the current moment and is closest to the current moment.

[0106] In another implementation of S400 , the latest touch event is a touch event that occurs within a time period from a preset time length (eg, the last second) before the current moment to the current moment.

[0107] S410: Analyze the touch event to obtain touch area information.

[0108] In some embodiments, the touch area information includes the number of simultaneous touches by the user, the center coordinates of the touch points, the major and minor axes of the touch ellipse area, and the deflection angle of the ellipse.

[0109] FIG7 is a schematic diagram showing the touch positions of the user's fingers in the game state.

[0110] Screen 500 is a mobile phone screen.

[0111] In the holding posture of holding the mobile phone with both hands (the mobile phone is in landscape orientation), the touch areas of the user's fingers can be obtained as shown in 501 and 502 of FIG. 7 .

[0112] After S410 , S420 is executed.

[0113] S420 , predicting the area covered by the user's hand on the display screen based on the touch area information and the user's holding habits (holding posture shown in FIG5 ), and generating a hand coverage area.

[0114] For example, based on the embodiment shown in FIG7 , in some embodiments, S420 includes:

[0115] The touch center point on the left half of the screen (the center point of 501) is displaced to the lower left corner of the screen by 1.5 times the distance of the long axis of the touch area, and the touch center point on the right half of the screen (the center point of 502) is displaced to the lower right corner of the screen by 1.5 times the distance of the long axis of the touch area.

[0116] The major axis and minor axis are doubled respectively, and the deflection angle of the ellipse remains unchanged to obtain the predicted hand coverage area.

[0117] As shown in FIG. 7 , based on the positions of 501 and 502 and the user's gripping habits, it can be predicted that areas 510 and 520 are areas covered by the user's hands.

[0118] S430 , drawing a guidance mask image, wherein the occlusion area (hand occlusion area) of the guidance mask image is drawn according to the predicted hand coverage area.

[0119] In some embodiments, in S430, the occlusion area (hand occlusion area) of the guidance mask image is drawn based on the ellipse information of the predicted hand coverage area (the predicted long and short axis lengths of the hand occlusion area, and the predicted center point coordinates of the hand occlusion area).

[0120] In one implementation of S430, the predicted hand coverage area is used as the hand occlusion area.

[0121] In another implementation of S430, after obtaining the hand coverage area, it is also determined whether the hand coverage area overlaps with the display interface of the first application. If there is an overlap, the area in the hand coverage area that overlaps with the display interface of the first application is marked as a hand occlusion area (drawing a hand occlusion area mask).

[0122] For example, when the first application is displayed full-screen, the entire hand-covered area will block the display interface of the first application. The entire hand-covered area is marked as the hand-blocking area. When the first application is displayed in split-screen with another application, the portion of the hand-covered area that overlaps with the display area of ​​the first application will block the display interface of the first application. The portion of the hand-covered area that overlaps with the display area of ​​the first application is marked as the hand-blocking area.

[0123] FIG8 shows a guide mask diagram according to an embodiment of the present application.

[0124] After S430, a guidance mask image 800 as shown in Figure 8 can be drawn. In the guidance mask image 800 shown in Figure 8, the areas of black blocks 610 and 620 are occlusion areas (hand occlusion areas).

[0125] A touch hotspot is an area on the screen where the user frequently performs touch operations. For example, the screen area where the user's finger touches the screen for 70% or more of the time within a unit of time is considered a touch hotspot. In some embodiments, based on the touch hotspot and the user's hand gesture, the area on the display screen covered by the user's hand when performing a touch operation in the touch hotspot is predicted to generate a hand coverage area.

[0126] 2: User interface (UI) occlusion area.

[0127] A user interface is software designed to facilitate communication between users and hardware. Its purpose is to enable users to operate hardware conveniently and efficiently to achieve two-way interaction and complete tasks they wish to accomplish with the help of the hardware. The definition of a user interface is broad, encompassing both human-computer interaction and graphical user interfaces. User interfaces exist in any field involving information exchange between humans and machines.

[0128] When an app is running, its UI is provided to facilitate user control. This UI overlays the app's content, making any content obscured by the UI invisible. During rendering, areas obscured by the UI are shaded at a lower rate, without compromising perceptible output quality.

[0129] For example, when a video application plays a video, if the UI (video playback control button) is hidden, the display screen shows the entire video screen. If the UI is displayed while the video is playing, one UI display solution is to display the UI floating above the video screen, blocking a portion of the video screen.

[0130] For example, when a game app is running, the rendering of the game application screen is generally divided into the rendering of the main game scene and the rendering of the game UI. The main game scene includes the game terrain, vegetation, buildings, characters, special effects, etc.; the game UI includes operation buttons, character status bar, game system bar, etc. The game UI is displayed floating above the main game scene, obscuring a portion of the main game scene.

[0131] Therefore, in one embodiment, the user interface (UI) blocking area includes an area in the image area of ​​the user interface that blocks the application display content.

[0132] In some embodiments, in S310, an image area of ​​the user interface is identified from the displayed image of the previous frame; it is determined whether the image area of ​​the user interface will cause occlusion of the application content; if it will cause occlusion, the image area corresponding to the user interface that causes occlusion is confirmed as the UI occlusion area.

[0133] During image rendering, the data to be rendered is typically divided into multiple parts, depending on the rendering method and type of data being rendered. Each part of the rendering process is called a rendering pass. Ultimately, the rendering results of all rendering passes are combined to form the final rendered image. For example, the image to be displayed can be divided into multiple layers, and a rendering process (rendering pass) is created for each layer. Finally, the rendering results of multiple rendering passes are overlaid for display.

[0134] For example, in a game application scenario, the game interface displays content including a 3D background image, 3D item images, 2D game character images, special effects for moves and actions, character dialogues, item names, and a game operation interface. The display order is as follows: display the 3D background image; overlay 3D item images and 2D game character images on the 3D background image; label the item names on the 3D item images; attach character dialogues and special effects for moves and actions to the 2D game character images; and the game operation interface is placed on top of all layers. During the image rendering process, a 3D background rendering channel is created, and the 3D background rendering channel is used to render and generate a background layer (the background layer contains a 3D background image); a 3D item rendering channel is created, and the 3D background rendering channel is used to render and generate an item layer (the item layer contains 3D item images labeled with item names); a 2D rendering channel is created, and the 2D background rendering channel is used to render and generate a 2D layer (the 2D layer contains 2D game character images with attached character dialogues and special effects for moves and actions); and a UI rendering channel is created, and the UI rendering channel is used to render and generate the game operation interface. Recently, the background layer, item layer, 2D layer and game operation interface are superimposed and displayed in order.

[0135] In one implementation of S310, a rendering channel including UI rendering is identified in a rendering channel of a previous image frame. The rendering channel including UI rendering may be a rendering channel including only UI rendering (in the rendering process, a separate rendering channel is used for UI rendering), or a rendering channel including UI rendering and rendering of other content (in the rendering process, UI rendering and other rendering content are merged into one rendering channel for rendering).

[0136] Split the UI rendering portion of the rendering pass that includes UI rendering into an independent rendering pass, thereby obtaining a UI rendering pass used only for drawing the UI. Based on the UI rendering pass, determine the UI interface to be displayed opaquely and generate the UI occlusion area.

[0137] For example, based on the UI rendering pass, pixels of the UI interface are sampled. If the pixel's color is not the initial color (a pixel with a color is not the initial color), and the pixel's transparency (alpha value) is greater than a preset blend threshold (e.g., 0.2), the pixel is considered opaque and marked as a UI occlusion pixel. The pixels of the entire UI rendering pass are traversed, and the UI occlusion area is ultimately determined based on the set of UI occlusion pixels.

[0138] FIG9 shows a flow chart of drawing a guidance mask according to an embodiment of the present application.

[0139] S700: Obtain a rendering channel including UI rendering.

[0140] S710: Determine whether the rendering channel including UI rendering is a rendering channel for performing UI rendering alone.

[0141] If the rendering channel includes UI rendering and is not a separate rendering channel for UI rendering, execute S720.

[0142] S720: Split and draw the UI rendering portion of the rendering channel including the UI rendering into an independent UI rendering channel.

[0143] S730, pixel sampling for independent UI rendering channels.

[0144] If the rendering channel includes UI rendering and is not a separate rendering channel for UI rendering, execute S730.

[0145] S740: Determine whether the sampling result has a color value and a transparency value greater than a preset mixing threshold.

[0146] If the judgment result is no, execute S751.

[0147] If the judgment result is yes, execute S750.

[0148] S750: Mark the sampled pixels as UI occlusion pixels.

[0149] S751, determine whether to traverse all pixels of the UI rendering channel.

[0150] If not, return to S730.

[0151] If yes, execute S760.

[0152] S760 , drawing a guidance mask image, wherein the occlusion area (UI occlusion area) in the guidance mask image is drawn according to the marking result (UI occlusion pixel) of S750 .

[0153] Taking a game application scenario as an example, as shown in FIG4 , the game tool component 401 , the character status component 402 , the skill button components 403 , 404 and 405 on the mobile phone screen 400 will block the 3D game scene screen below.

[0154] FIG. 10 shows a guide mask diagram according to an embodiment of the present application.

[0155] According to the positions of the game tool component 401, the character status component 402, and the skill button components 403, 404, and 405 on the mobile phone screen 400 shown in Figure 4, a guidance mask 1000 as shown in Figure 10 can be drawn. In the guidance mask 1000, the black block areas 1001, 1002, 1003, 1004, and 1005 are occlusion areas (UI occlusion areas).

[0156] 3: Area blocked by other applications.

[0157] While the first application is running in the foreground, other applications may also be running. These other applications may be background applications or foreground applications displayed in a split-screen with the first application. For example, the first application is running in the foreground, and the other applications other than the first application include at least a second application, which is different from the first application. The second application may be running in the background, or the second application may be running in the foreground and displayed in a split-screen with the first application.

[0158] Applications other than the first application may cause a pop-up window to be displayed above the display interface of the first application running in the foreground, or maintain a floating window display above the display interface of the first application running in the foreground.

[0159] When another application displays a pop-up window or maintains a floating window display above the display interface of the first application, the content of the first application obscured by the pop-up window or floating window is not visible. When rendering, the area obscured by the pop-up window or floating window is rendered at a low rate without reducing the perceptible output quality.

[0160] For example, when a video application is playing a video, if a chat application running in the background receives a new chat message, the chat message is displayed as a pop-up window above the video screen, blocking a portion of the video screen.

[0161] For example, when a game app is running, to improve the gaming experience, the user can run a game assistant in the background. The game assistant provides a quick operation bar for the game. The quick operation bar is displayed above the main game scene in the form of a floating window, covering a portion of the main game scene.

[0162] Therefore, in one embodiment, the area blocked by other applications includes the window area of ​​the foreground active window of other applications (including at least the second application) other than the first application, or a part of the window area of ​​the foreground active window of other applications (including at least the second application) other than the first application.

[0163] In one embodiment, in S310, when rendering the current frame, at the end of the previous frame or the beginning of the current frame, window information of active foreground windows of applications other than the first application (the other applications include at least a second application, which is different from the first application running in the foreground) is obtained through the system framework layer. Active foreground windows include fixed windows, floating windows, and pop-up windows displayed on the screen.

[0164] Parse the window information of the active foreground window to obtain its size, shape, center coordinates, and other information. Draw the window area of ​​the active foreground window of other applications and mark the window area of ​​the active foreground window of other applications as the occlusion area of ​​other applications.

[0165] Alternatively, in another embodiment, after obtaining the window area of ​​the foreground active window of another application, it is determined whether the window area overlaps with the display interface of the first application. If there is overlap, the overlapping area is marked as the other application occlusion area.

[0166] In some embodiments, when a first application running in the foreground is displayed full screen (there is only one foreground application), if an application other than the first application (for example, the other application other than the first application includes a second application, and the second application is running in the background) has an active foreground window, the entire area of ​​the active foreground window overlaps with the display interface of the first application. In this case, the window area of ​​the active foreground window of the application other than the first application is marked as an "other application obstruction area."

[0167] When multiple applications are running in the foreground (for example, a first application is displayed in a split-screen with one or more other applications), and an application other than the first application has an active foreground window, the overlap area between the foreground active window and the display interface of the first application is determined based on the position / size of the foreground active window. The overlap area between the foreground active window of the other application other than the first application and the display interface of the first application is marked as an "other application obstruction area."

[0168] Taking a game application scenario as an example, the first application is a game application, and the first application is displayed in full screen.

[0169] FIG. 11 is a flowchart showing a method of drawing a guidance mask according to an embodiment of the present application.

[0170] S1000: Determine through the system framework layer whether there is currently any other application window on the game screen (whether there is an active foreground window of another application).

[0171] If it does not exist, the process ends.

[0172] If it exists, execute S1010.

[0173] S1010: Parse the application window information obtained through the framework layer to obtain the window information of the foreground active window of other applications.

[0174] The window information of the foreground active windows of other applications includes the number of windows and the window coordinates, window shape, and window size information of each window.

[0175] S1020: Draw a guidance mask image, wherein the occlusion area (other application occlusion area) in the guidance mask image is drawn according to other application window information.

[0176] As shown in Figure 4, the game assistant component 406 on the mobile phone screen 400 is an application interface of the game assistant application other than the game application. The game assistant component 406 will block the 3D game scene screen below.

[0177] FIG. 12 shows a guide mask diagram according to an embodiment of the present application.

[0178] According to the position of the game assistant component 406 in the mobile phone screen 400 shown in Figure 4, a guidance mask 1200 as shown in Figure 12 can be drawn. In the guidance mask 1200, the black block area 1201 is the occlusion area (other application occlusion area).

[0179] 4: Device self-occlusion area.

[0180] Mobile phones are usually equipped with a front camera. Generally, the front camera is installed on the mobile phone display.

[0181] For example, some displays have a notch area; another example is a hole-punch area. For example, the notch area or the hole-punch area may correspond to one or more front-facing cameras. Due to the presence of the front-facing camera, the area corresponding to the front-facing camera cannot display images.

[0182] When rendering, the parts of the application's rectangular display screen that cannot be displayed (corresponding to the display content of the rectangular top corners / bangs area / hole area) are shaded at a low rate, which does not reduce the perceptible output image quality.

[0183] Therefore, in one embodiment, the device self-occlusion area includes an area of ​​the display screen of the electronic device corresponding to the front camera.

[0184] In some embodiments, the device's self-occlusion area may include the area of ​​the under-screen camera, and the area corresponding to the under-screen camera can be used for image display. It should be understood that the area corresponding to the under-screen camera is generally close to the edge of the screen, and therefore the methods described in the various embodiments of this application can also be applied.

[0185] Generally, to ensure the versatility of displayed content, the display screen of an application is rectangular. However, the shape of some display screens is not strictly rectangular.

[0186] For example, some displays are rectangular with rounded corners. The top corners of the rectangular display cannot be displayed by the display. During rendering, the undisplayable parts of the application's rectangular display (corresponding to the display content in the rectangular top corners / notch area / hole cutout area) are shaded at a low rate without reducing the perceived output quality.

[0187] For example, some displays (curved screens) have curved display areas. Due to viewing angle limitations, users do not carefully distinguish the content displayed in the curved display area. During rendering, the content displayed in the curved display area within the application's rectangular display is shaded at a low rate without reducing the perceived output quality.

[0188] Therefore, in one embodiment, the self-occlusion area of ​​the device also includes the missing portion of the display screen compared to the rectangular display screen (the missing top corner of the rectangle caused by the rounded corner) and the curved display portion.

[0189] In summary, in one embodiment, the device self-occlusion area includes the portion of the rectangular display screen that is blocked by the device itself. That is, the device self-occlusion area includes the portion of the display screen's normal display area that is missing from the rectangular display screen (the non-normal display area). The non-normal display area can be one or a combination of the following: the area corresponding to the front camera, the missing corners of the rectangular display screen's rounded corners, or the curved display area of ​​the display screen.

[0190] In some embodiments, the device self-occlusion area is drawn according to the current device's occlusion of the rectangular display screen (compared with the rectangular display screen by the shape of the normal display area of ​​the display screen).

[0191] For example, a device self-occlusion image is pre-generated based on the current device's occlusion of a rectangular display screen when the application is displayed in full screen. This device self-occlusion image includes an interface graphic whose size and shape match the rectangular display screen. Within the interface graphic, blocks of different colors are used to mark the portion of the rectangular display screen that is occluded by the current device (the non-normal display area).

[0192] FIG. 13 is a flowchart showing a method of drawing a guidance mask according to an embodiment of the present application.

[0193] S1300: Determine whether there is a pre-stored device self-occlusion image.

[0194] If it does not exist, the process ends.

[0195] If so, execute S1310.

[0196] In some embodiments, the execution of S1300 is triggered during the initialization phase of VRS.

[0197] In other embodiments, the execution of S1300 is triggered during the device startup phase.

[0198] S1310: Determine whether the device self-occlusion image has been loaded.

[0199] If loaded, execute S1320.

[0200] S1320: Pass the memory pointer of the loaded device self-occlusion image to the thread that draws the guidance mask image.

[0201] If not loaded, execute S1330.

[0202] S1330: Read the device self-occlusion image stored in the memory.

[0203] S1320 is executed after S1330.

[0204] S1340 : Based on the memory pointer of the loaded device self-occlusion image, obtain the device self-occlusion image, and draw a guidance mask image according to the device self-occlusion image.

[0205] In one embodiment, in S1340, the device self-occlusion image is used as a guidance mask image. In the guidance mask image, the occlusion area (device self-occlusion area) is the abnormal display area of ​​the display screen.

[0206] Alternatively, in another embodiment, at S1340, it is determined whether the non-normal display area in the device self-occlusion image overlaps with the display interface of the first application. If there is overlap, a guidance mask image is generated based on the device self-occlusion image. In the guidance mask image, the area in the non-normal display area that overlaps with the display interface of the first application is marked as an occlusion area (device self-occlusion area).

[0207] For example, when the first application is displayed full-screen, the entire non-normal display area will block the display interface of the first application. The entire non-normal display area is marked as a device self-occlusion area. When the first application is displayed in split-screen with other applications, the portion of the non-normal display area that overlaps with the display area of ​​the first application will block the display interface of the first application. The portion of the non-normal display area that overlaps with the display area of ​​the first application is marked as a device self-occlusion area.

[0208] Taking a game application scenario as an example, as shown in FIG4 , the screen 400 is a curved screen with a bangs, 409 is the bangs part, and the part above the dotted line 407 and below the dotted line 408 is the curved part of the screen.

[0209] FIG. 14 shows a guide mask diagram according to an embodiment of the present application.

[0210] According to the position of the bangs 409 and the curved surface in the mobile phone screen 400 shown in Figure 4, a guidance mask 1400 can be drawn as shown in Figure 14. In the guidance mask 1400, the black block areas 1401, 1402 and 1403 are occlusion areas (device self-occlusion areas).

[0211] FIG15 is a schematic diagram showing a display screen of a mobile phone according to an embodiment of the present application.

[0212] As shown in FIG15 , screen 1500 is a hole-punch screen, and 1501 is the hole-punch part.

[0213] FIG. 16 shows a guide mask diagram according to an embodiment of the present application.

[0214] According to the position of the hole portion 1501 in the mobile phone screen 1500 shown in Figure 15, a guidance mask diagram 1600 as shown in Figure 16 can be drawn. In the guidance mask diagram 1600, the black block area 1601 is the occlusion area (device self-occlusion area).

[0215] In S310 , when the occlusion region is identified by combining multiple occlusion region types, after determining the corresponding occlusion region for each occlusion region type, all occlusion regions are merged into the occlusion region in the guidance mask image.

[0216] For example, the occlusion areas (hand occlusion areas) 610 and 620 shown in FIG8 ; the occlusion areas (UI occlusion areas) 1001, 1002, 1003, 1004, and 1005 shown in FIG10 ; the occlusion area (other application occlusion area) 1201 shown in FIG12 ; and the occlusion area (device self-occlusion area) 1401 shown in FIG14 ; are fused into the occlusion areas in the guidance mask image shown in FIG5 ; and fused into the guidance mask image shown in FIG4 .

[0217] In S310 , other methods may be used to identify the blocked area.

[0218] Taking the game application scenario as an example, during the drawing process of the main game scene, there is an occlusion relationship between different models in the main game scene.

[0219] In one embodiment, in S310, the main scene rendering channel texture is obtained, sliced ​​into 16x16 slices, all pixels in each slice are read, their RGB values ​​are converted into brightness values, and the average brightness is calculated as a reference threshold. The frequency reference value is obtained from the frequency domain information calculation module and the motion rate reference value is obtained from the motion rate calculation module, and the two are multiplied to obtain the final estimated reference value. The estimated reference value is compared with the reference threshold, and the slices below the reference threshold are marked as occlusion areas (slices above the reference threshold are marked as high-rate shading, and slices below the reference threshold are marked as low-rate shading), and a guidance mask is generated.

[0220] According to the method of the embodiment of the present application, the occlusion area is determined from multiple angles, the effective range of the occlusion area is increased, the stability of the variable rate shading gain is improved, and the scene coverage is expanded.

[0221] S320, rendering the image to be displayed based on variable rate shading to generate a display image of the first application, wherein the occluded area is rendered based on at least a first shading rate, and the non-occluded area is rendered based on at least a second shading rate, and the first shading rate is less than the second shading rate.

[0222] In S320 , variable rate shading is applied to the rendering instruction stream and associated data acquired in S300 according to the guidance mask image drawn in S310 to generate a display image of the first application.

[0223] The S320 includes:

[0224] Using the guidance mask image as a mask for the image to be displayed, matching the occluded area in the guidance mask image with the image to be displayed in terms of shape and position;

[0225] Based on the guidance mask, the image to be displayed is rendered, wherein occluded areas are rendered at least at a first shading rate, and unoccluded areas are rendered at least at a second shading rate, wherein the first shading rate is less than the second shading rate. That is, image areas in the image to be displayed corresponding to occluded areas in the guidance mask are shaded at a lower rate.

[0226] In one implementation of S320 , a uniform shading rate is adopted for the occluded area.

[0227] In one embodiment, the shading rate is described using a matrix. A shading rate a*b means that one pixel is reserved for rendering and shading among a*b pixels in column a and row b.

[0228] For example, in one embodiment, a shading rate of 4*4 is used for the occluded area, that is, for every 4*4 pixels corresponding to the occluded area, 15 pixels are eliminated and one pixel is retained for rendering and shading. After the rendering and shading is completed, the previously eliminated pixels are supplemented by sampling the previously retained pixels.

[0229] For example, let's set the guidance mask image and the image to be rendered to a 1:64 ratio. Based on the shape and position matching between the guidance mask image and the image to be rendered, one pixel in the guidance mask image corresponds to 8x8 pixels in the image to be rendered (one mask pixel guides the shading of 8*8 pixels). For example, if the image to be rendered contains 8m*8n pixels, then the guidance mask image contains m*n mask pixels.

[0230] Generate a pixel shading list based on the guidance mask. The pixel shading list contains data bits that are consistent with the number of pixels in the image to be rendered. For example, if the image to be rendered contains 8m*8n pixels, then the pixel shading list contains 8m*8n data bits.

[0231] The data bits in the pixel shading list correspond one-to-one with the pixels of the image to be rendered. Based on the correspondence between the mask pixels in the guidance mask image and the pixels in the image to be rendered, the correspondence between the mask pixels in the guidance mask image and the data bits in the pixel shading list is set. A mask pixel in the guidance mask image guides the value of the 8*8 data bits in the pixel shading list.

[0232] The value of each data bit in the pixel shading list can be 0 or 1. When a mask pixel belongs to the non-occluded area of ​​the guidance mask image, the shading rate of the 8x8 pixels of the to-be-rendered image corresponding to the mask pixel is configured to be 4*4. That is, when a mask pixel belongs to the non-occluded area of ​​the guidance mask image, the value of the 8*8 data bits in the pixel shading list corresponding to the mask pixel is configured to be 15 of every 16 data bits set to 0 and one data bit set to 1.

[0233] During the shading process, the pixels are shaded according to the values ​​in the pixel shading list. The pixels corresponding to the data bits with the values ​​of 0 in the pixel shading list are eliminated, and the pixels corresponding to the data bits with the values ​​of 1 in the pixel shading list are shaded.

[0234] FIG17 is a schematic diagram showing the values ​​of 8*8 data bits in a pixel shading list with a shading rate of 4*4.

[0235] As shown in FIG17 , the values ​​of 15 data bits out of every 16 data bits are set to 0, and the value of one data bit is set to 1.

[0236] FIG18 is a schematic diagram showing the shading result of 8*8 pixels with a shading rate of 4*4.

[0237] As shown in FIG18 , only pixels 1801 , 1802 , 1803 , and 1804 are colored, and the remaining pixels are not colored.

[0238] After rendering and shading are completed, the previously eliminated pixels are supplemented by sampling the previously retained pixels. That is, the shading results of the shaded pixels are used to supplement the previously unshaded pixels.

[0239] FIG19 is a schematic diagram showing the coloring supplement results based on the coloring results of FIG18 .

[0240] As shown in FIG19 , the coloring results of pixels 1801 , 1802 , 1803 , and 1804 are added to the surrounding uncolored pixels.

[0241] Taking a game application scenario as an example, the depth value of the main scene corresponding to the low shading rate area is set to 0.0f. When the game draws the main scene, the early-z feature is used to eliminate the shading overhead of 15 / 16 (93.75%) pixels in the occluded area. Finally, the empty pixels are restored by sampling the adjacent points interpolation.

[0242] In some embodiments, the graphics system is configured with a depth buffer for image rendering. The depth buffer is used to store the depth or distance value of each pixel in the window from a viewing plane (near clipping plane). The depth buffer can be implemented in hardware, for example, by designating a fixed storage space in the hardware memory as the depth buffer; the depth buffer can also be implemented in software, for example, by dynamically designating a storage address space as the depth buffer during rendering.

[0243] Since the depth buffer can store the depth values ​​of all pixels of the image to be rendered, the depth value can be replaced with the data bit of the pixel shading list, and the pixel shading list can be directly stored in the depth buffer without the need to reconfigure the data structure.

[0244] Therefore, in one embodiment, the pixel shading list is implemented based on a depth buffer. For example, depth buffer data is generated according to a guidance mask image, and the depth buffer data is stored in the depth buffer.

[0245] A mask pixel in the guidance mask image guides the depth value of 8*8 pixels in the depth buffer data. When a mask pixel belongs to the non-occluded area of ​​the guidance mask image, the depth value of the 8x8 pixels in the depth buffer data corresponding to the mask pixel is configured, and 15 of the 16 depth values ​​are set to 0 and one depth value is set to 1. (0 represents the closest depth and 1 represents the farthest depth)

[0246] During the shading process, the depth buffer data is used for shading, pixels with a depth value of 0 in the depth buffer data are discarded, and pixels with a depth value of 1 in the depth buffer data are shaded.

[0247] In another implementation of S320 , a uniform shading rate is not used for the occluded area. Instead, the occluded area is divided into multiple sub-areas, and different shading rates are used for different sub-areas.

[0248] For example, in one embodiment, the occlusion area is divided into multiple sub-areas based on the different types of occlusion areas. For example, the hand occlusion area, the user interface occlusion area, the other application occlusion area, and the device self-occlusion area each correspond to four sub-areas. In another example, the hand occlusion area and the device self-occlusion area correspond to one sub-area, and the user interface occlusion area and the other application occlusion area correspond to one sub-area.

[0249] In some embodiments, in the guidance mask image, different sub-regions in the occluded region are marked by pixel values ​​of mask pixels, and different pixel values ​​correspond to different sub-regions. A corresponding shading rate is adopted according to the pixel values ​​of the mask pixels in the guidance mask image.

[0250] In some embodiments, different mask pixel values ​​are set to correspond to different shading rates. As shown in Table 1:

[0251] Table 1

[0252] Set one pixel in the guidance mask image to correspond to 8x8 pixels on the display screen (one mask pixel guides the shading of 8*8 pixels). When drawing the guidance mask image, set the corresponding pixel value for each mask pixel in the guidance mask image according to the shading rate distribution. For example, for mask pixels in the non-occluded area, set the mask pixel value to 0 (shading rate is 1*1, normal shading, no low-rate shading); for mask pixels in the hand occlusion area, set the mask pixel value to 9 (shading rate is 4*2); for mask pixels in the UI occlusion area and other application occlusion areas, set the mask pixel value to 10 (shading rate is 4*4); for mask pixels in the self-occlusion area of ​​the device, set the mask pixel value to 16 (shading rate is 8*8).

[0253] The depth buffer data is generated based on the guidance mask image. One mask pixel guides the depth value of 8*8 pixels in the depth buffer data. When the mask pixel value is 0, the depth value of the corresponding 8*8 pixels in the depth buffer is 1; when the mask pixel is 9, the depth value of 7 pixels in each 4*2 pixels of the corresponding 8x8 pixels in the depth buffer is set to 0, and the depth value of one pixel is set to 1 (the shading rate is 4*2).

[0254] According to the method of the embodiment of the present application, identifying the occluded area of ​​the application and performing VRS can reduce the rendering calculation amount without reducing the user experience, thereby improving rendering performance and reducing display power consumption. According to the method of the embodiment of the present application, because the occluded area has a large effective range and the user's perception is small, VRS benefits are stable and cover a wide range of scenarios.

[0255] In one embodiment, frame-based variable rate shading is employed.

[0256] FIG20 is a flowchart of a VRS implementation according to an embodiment of the present application.

[0257] S2000: Draw the guidance mask map of the N-1th frame.

[0258] S2010, obtaining the rendering instruction stream and associated data of the Nth frame.

[0259] S2020, applying variable rate shading to the rendering instruction stream and associated data obtained in S2010 according to the guidance mask map drawn in S2000.

[0260] S2030: Identify the occluded area of ​​the Nth frame.

[0261] S2040: Draw a guidance mask map for the Nth frame.

[0262] In one embodiment, VRS is implemented based on software, so that VRS can be used on different hardware platforms.

[0263] FIG21 is a schematic diagram of a system architecture according to an embodiment of the present application.

[0264] As shown in FIG. 21 , the application layer 2210 is used to run applications (eg, a game application 2211 , a video playback application 2212 , etc.).

[0265] The graphics API layer 2220 is used to load a graphics driver, which is a driver program capable of drawing graphics, such as OpenGL and Vulkan.

[0266] The operating system (OS) kernel layer 2240 is used to load the system kernel and drivers.

[0267] The chip layer 2250 is the hardware layer of the electronic device, which includes a CPU and a GPU.

[0268] The VRS layer 2230 is used to implement the VRS method flow of the application embodiment. The VRS layer 2230 is included in the instruction reassembly layer of the terminal OS system and includes a data acquisition module 2231, an occlusion region identification module 2232, and a variable rate shading module 2238.

[0269] Data Acquisition Module 2231 is used to intercept graphics API calls from Graphics API Layer 2220, capture the graphics command data stream, collect and calculate rendering-related data, such as the main scene rendering pass and UI rendering pass for the current frame, and cache the relevant data. Data Acquisition Module 2231 utilizes the graphics driver's reserved interface to intercept all driver layer instructions before the engine layer calls the driver layer instructions, and caches them as a command stream. Some of these driver instructions contain information about the main scene rendering pass and UI rendering pass during the drawing process.

[0270] Taking a game application scenario as an example, FIG22 is a schematic diagram of the operation flow of the data acquisition module 2231 according to an embodiment of the present application.

[0271] S2300, the game engine calls the graphics driver interface.

[0272] S2310: The data acquisition module 2231 determines whether the game rendering needs to be optimized.

[0273] If not, the process ends.

[0274] If yes, execute S2320.

[0275] S2320, the data acquisition module 2231 intercepts the graphics instruction stream.

[0276] S2321, the data acquisition module 2231 analyzes the graphics instruction stream and obtains information such as the game main scene rendering channel and the UI rendering channel.

[0277] S2322, the data acquisition module 2231 caches the data acquired in S2321.

[0278] The occlusion area recognition module 2232 performs recognition at the end of the frame, analyzes the touch events and other application window information received in the current frame, calculates the hand occlusion and other application window occlusion areas; analyzes the UI rendering channel at the same time, calculates the UI occlusion area; loads the device device occlusion area; and finally comprehensively draws the guidance mask map (which areas use low-rate shading).

[0279] The occlusion area recognition module 2232 includes a hand occlusion area prediction module 2233 , another window recognition module 2234 , a UI occlusion area recognition module 2235 , a device occlusion recognition module 2236 and an occlusion area drawing module 2237 .

[0280] The variable rate shading module 2238 is used to decide which areas to shade at which rate when drawing the main scene in the next frame based on the guidance mask, reorganize the intercepted instruction stream, and transmit it to the system kernel and driver (operating system (OS) kernel layer 2240).

[0281] Taking a game application scenario as an example, FIG23 is a schematic diagram of the operation flow of the variable rate shading module 2238 according to an embodiment of the present application.

[0282] S2400 , obtaining the graphics instruction stream cache data cached by the data acquisition module 2231 .

[0283] S2410, obtaining depth cache data of the game main scene rendering channel according to the graphics instruction stream cache data.

[0284] S2420 , according to the guidance mask image generated by the occlusion area identification module 2232 , write the corresponding depth value (0 or 1) for the occlusion area in the depth buffer data of the main game scene.

[0285] For example, a mask pixel controls the shading rate of the 8x8 area pixels in the depth cache; when the mask pixel value is 10, the shading rate of the 8x8 pixel area corresponding to the depth cache should be 4x4, which means that only 1 pixel is retained for shading out of 16 pixels, the depth value of the shaded pixel value is set to 1, and the depth values ​​of the remaining 15 pixels that need to be eliminated are set to 0 (0 represents the closest depth and 1 represents the farthest depth).

[0286] S2430, rendering the main game scene. For pixels with a depth value of 0, the pixel will not be shaded through the early-z mechanism, thereby saving the FS execution overhead.

[0287] S2440: After the main scene is rendered, the colors of the uncolored pixels in the occluded area are filled in through a sampling interpolation algorithm according to the guidance mask generated by the occluded area drawing module.

[0288] According to the method of the embodiment of the present application, VRS is implemented in the instruction reassembly layer in the terminal OS system, and there is no need to adapt the application of the application layer. The implementation of VRS does not depend on the application, thereby ensuring that the VRS method of the embodiment of the present application can be ported to any platform and applied to any application, thereby improving the portability and versatility of VRS.

[0289] In the description of the embodiments of the present application, for the convenience of description, the device is described as being divided into various modules according to their functions. The division of each module is merely a division of logical functions. When implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0290] The device proposed in the embodiment of the present application can be fully or partially integrated into a physical entity during actual implementation, or it can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; or they can all be implemented in the form of hardware; or some modules can be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the detection module can be a separately established processing element, or it can be integrated in a chip of an electronic device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. During the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or the instructions in the form of software.

[0291] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0292] An embodiment of the present application further provides an electronic device.

[0293] FIG24 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

[0294] As shown in Figure 24, the electronic device 2500 includes a memory 2502 for storing computer program instructions and a processor 2501 for executing program instructions, wherein, when the computer program instructions are executed by the processor 2501, the electronic device 2500 is triggered to execute the method steps described in the embodiments of the present application.

[0295] In one embodiment of the present application, the above-mentioned one or more computer programs are stored in the above-mentioned memory 2502, and the above-mentioned one or more computer programs include instructions. When the above-mentioned instructions are executed by the above-mentioned electronic device 2500, the above-mentioned electronic device 2500 executes the method steps described in the embodiment of the present application.

[0296] It is understood that the structural description of the electronic device 2500 in the embodiment of the present application does not constitute a specific limitation on the electronic device 2500. In other embodiments of the present application, the electronic device 2500 may include other components besides the processor 2501 and the memory 2502.

[0297] The processor 2501 may be a device on a chip (SOC), and the processor 2501 may include a central processing unit (CPU), and may further include other types of processors.

[0298] The processor involved in processor 2501 may include, for example, a CPU, a DSP, a microcontroller, or a digital signal processor, and may also include a GPU, an embedded neural network processor (NPU), and an image signal processor (ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASICs, or one or more integrated circuits for controlling the execution of the program of the technical solution of this application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

[0299] The processor 2501 may include one or more processing units. For example, the processor may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 2500 may also include one or more processors 2501. The controller may generate an operation control signal based on the instruction opcode and the timing signal to complete the control of instruction fetching and execution.

[0300] In some embodiments, the processor 2501 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc. Among them, the USB interface is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface can be used to connect a charger to charge the electronic device, and can also be used to transmit data between the electronic device and peripheral devices.

[0301] Electronic device 2500 may also include an external memory interface for connecting an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with processor 2501 via the external memory interface to implement data storage. For example, files such as music and videos can be stored on the external memory card.

[0302] Memory 2502 may include a code storage area and a data storage area. The code storage area may store an operating system. The data storage area may store data created during the use of electronic device 2500. Furthermore, memory 2502 may include high-speed random access memory and non-volatile memory, such as one or more disk storage components, flash memory components, and universal flash storage (UFS).

[0303] The memory 2502 may be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any computer-readable medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer.

[0304] The processor 2501 and the memory 2502 may be combined into one processing device, or more commonly, they may be independent components.

[0305] The devices, apparatuses, and modules described in the embodiments of the present application may be implemented by computer chips or entities, or by products having certain functions.

[0306] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, apparatus, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0307] In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this application.

[0308] In one embodiment of the present application, a computer-readable storage medium is further provided, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the embodiment of the present application.

[0309] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program product is run on a computer, it enables the computer to execute the method provided in the embodiment of the present application.

[0310] The embodiment description in this application is described with reference to the flow chart and / or block diagram according to the method, equipment (device) and computer program product of embodiment of the present application.It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of general-purpose computer, special-purpose computer, embedded processing machine or other programmable data processing equipment to produce a machine, so that the instruction executed by the processor of computer or other programmable data processing equipment produces the device for realizing the function specified in one flow chart flow chart or multiple flow charts and / or one block or multiple blocks of block diagram.

[0311] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0312] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0313] It should also be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single or multiple.

[0314] In the embodiments of the present application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, commodity, or apparatus comprising the element.

[0315] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0316] The various embodiments in this application are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the partial description of the method embodiments.

[0317] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments of the present application can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0318] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0319] The above description is merely a specific embodiment of the present application. Any person skilled in the art may easily conceive of variations or substitutions within the technical scope disclosed in this application, and such variations or substitutions shall be within the scope of protection of this application. The scope of protection of this application shall be subject to the scope of protection of the claims.

Claims

1. An image rendering method based on variable rate shading, applied to electronic equipment, characterized in that: The method comprises: Determine an occluded area and a non-occluded area on the image to be displayed of the first application, wherein the occluded area is an occluded area in the image to be displayed, and the non-occluded area is an unoccluded area in the image to be displayed; Based on variable rate shading, the image to be displayed is rendered to generate a display image of the first application, wherein the occluded area is rendered based on at least a first shading rate, and the non-occluded area is rendered based on at least a second shading rate, and the first shading rate is less than the second shading rate.

2. The method according to claim 1, characterized in that The occlusion area includes any one or more combinations of a hand occlusion area, a user interface occlusion area, other application occlusion areas, and a device self-occlusion area, wherein: The hand-blocking area includes a hand-covered area on the display screen determined in response to a touch operation of a user or a portion of the hand-covered area; The user interface blocking area includes, in the image area of ​​the user interface of the first application, an area that blocks the application display content of the first application; The other application blocking area includes the window area of ​​the foreground active window of the second application, or a part of the window area of ​​the foreground active window of the second application; The device self-occlusion area includes an area of ​​the display screen of the electronic device corresponding to the front camera.

3. The method according to claim 2, characterized in that The determining of the blocked area and the non-blocked area on the image to be displayed of the first application includes: In response to a touch event of a user, analyzing the touch event to obtain touch area information; Determining the hand coverage area according to the touch area information; Mark the hand coverage area as the hand obstruction area; or, mark the area in the hand coverage area that overlaps with the display interface of the first application as the hand obstruction area.

4. The method according to claim 2, characterized in that: The determining of the blocked area and the non-blocked area on the image to be displayed of the first application includes: Determine an image area of ​​the user interface according to a display image of a previous frame; When the image area of ​​the user interface causes occlusion of the application content of the first application, the image area of ​​the user interface that causes occlusion is marked as the user interface occlusion area.

5. The method according to claim 4, characterized in that The determining of the blocked area and the non-blocked area on the image to be displayed of the first application includes: Identify the previous frame's UI rendering pass; Sampling the user interface rendering channel to obtain sampled pixels; When the sampled pixel has a color value and the transparency is greater than a preset threshold, the image area corresponding to the sampled pixel is marked as the user interface occlusion area.

6. The method according to claim 2, characterized in that The determining of the blocked area and the non-blocked area on the image to be displayed of the first application includes: Obtaining window information of the foreground active window of the second application; Drawing a window area of ​​the foreground active window according to the window information of the foreground active window of the second application; Mark the window area of ​​the foreground active window as the other application obstruction area; or, mark the area of ​​the window area of ​​the foreground active window that overlaps with the display interface of the first application as the other application obstruction area.

7. The method according to claim 2, characterized in that The determining of the blocked area and the non-blocked area on the image to be displayed of the first application includes: Comparing the shape of a normal display area of ​​the display screen with a rectangular display picture, and drawing the abnormal display area of ​​the display screen according to the comparison result; The non-normal display area is marked as a device self-occlusion area; or, an area in the non-normal display area that overlaps with the display interface of the first application is marked as the device self-occlusion area.

8. The method according to any one of claims 1 to 7, characterized in that The shielding area includes at least a first area; The step of rendering the image to be displayed based on variable rate shading to generate a display image of the first application includes: According to the first shading rate, a first pixel region is selected from the first region; the first region includes a first pixel region and a second pixel region; Coloring the pixels of the first pixel area; and filling the second pixel area with the coloring result based on the coloring result of the first pixel area.

9. The method according to claim 8, characterized in that The shielding area also includes at least a second area; The step of rendering the image to be displayed based on variable rate shading to generate a display image of the first application further includes: The second region is colored using a third coloring rate, where the third coloring rate is different from the first coloring rate and is smaller than the second coloring rate.

10. The method according to claim 9, characterized in that The second area and the first area correspond to different shielding area types respectively.

11. The method according to any one of claims 1 to 10, characterized in that The method comprises: Determine an occluded area and a non-occluded area on a display image of the first application in the N-1th frame; Obtaining a rendering instruction stream and associated data of an image to be displayed of the first application in the Nth frame; Performing display image rendering according to the rendering instruction stream and associated data of the image to be displayed of the first application of the Nth frame to generate a display image of the first application of the Nth frame, wherein the variable rate shading is applied according to the occluded area and the non-occluded area on the image to be displayed of the first application of the N-1th frame; An occluded area and a non-occluded area on a display image of the first application in the Nth frame are determined.

12. An electronic device, characterized in that: The electronic device comprises a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method steps as described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 11.

Images