Load-reducing rendering method and electronic device

Abstract

Provided in the embodiments of the present application is a load-reducing rendering method. An electronic device can identify each level of occlusion area (a finger touch area, a finger occlusion area and a palm occlusion area) on the basis of touch information of the finger touch area when a user touches a screen with fingers, and can perform differentiated load-reducing rendering on each level of occlusion area, for example, a load-reducing rendering method, i.e. rendering elimination, is implemented on the finger touch area, and rendering methods, i.e. resolution reduction, image texture reduction and feature point clipping, can be implemented on the finger occlusion area and the palm occlusion area. With regard to finger occlusion areas with different visibility, different degrees of resolution reduction, image texture reduction and feature point clipping can be implemented. Thus, the load of a GPU rendering game screen can be reduced without affecting the user experience.

Description

Load-reducing rendering method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 25, 2023, with application number 202310754454.7, and the priority of the Chinese patent application entitled “A load-reduced rendering method and electronic device”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of electronic technology, and in particular to a load-reducing rendering method and electronic equipment. Background Art

[0003] Game screen rendering places high demands on the performance of the graphics processing unit (GPU). In addition, long-term, high-resolution game screen rendering will cause severe rendering load on the GPU, which may cause problems such as game screen lag, high device power consumption and heat, and short battery life.

[0004] Summary of the Invention

[0005] In the first aspect, an embodiment of the present application provides a load-reduced rendering method, which may include: an electronic device displays a picture provided by a first application on the screen, detects a touch operation of a user's finger on the screen, and obtains touch information of the touch operation. Then, the electronic device determines a multi-level occlusion area based on the touch information, and the multi-level occlusion area includes: a finger touch area, and a finger occlusion area, wherein the finger occlusion area is covered by the periphery of the finger touch area. In addition, the electronic device performs load-reduced rendering on the pictures in the multi-level occlusion area, wherein the pictures in the occlusion area that are closer to the touch center are rendered with a lower load, and the touch center is the center of the finger touch area. The load-reduced rendering includes one or more of the following: rendering culling, reducing image texture, reducing image resolution, reducing feature points, and reducing shading rate.

[0006] In the first aspect, by implementing the method of the first aspect, the electronic device can more finely determine the areas with different degrees of occlusion and adapt differentiated rendering reduction methods, thereby reducing the GPU rendering load without affecting the user's gaming experience.

[0007] In conjunction with the first aspect, in some embodiments, rendering is performed with reduced load for occluded areas closer to the touch center. Specifically, this may include: rendering culling of the finger touch area; and performing one or more of the following processing on the finger occluded area: reducing image texture, reducing image resolution, reducing feature points, and reducing shading rate. Specifically, the closer the finger occluded area is to the touch center, the greater the reduction in resolution, image texture, and the greater the number of feature points that may be cropped.

[0008] In combination with the first aspect, in some embodiments, the first application may be a game application, and the screen also displays: a steering wheel, skill keys, and touch operations are performed on the steering wheel or skill keys. The steering wheel and skill keys are respectively distributed in the lower left corner, lower right corner, or respectively distributed in the lower right corner, lower left corner of the screen.

[0009] The first application is not limited to game applications, and may also be other applications that provide a graphical user interface and receive user touch input. The method of the first aspect is also applicable to such applications to reduce the rendering load of the graphical user interface when the user touches the screen.

[0010] In combination with the first aspect, in some embodiments, the finger touch area is an elliptical area, and the touch information includes the center coordinates, major and minor axes, and the direction of the elliptical area, wherein the direction of the elliptical area is the extension direction of the major axis.

[0011] There can be N finger occlusion areas, where N is a positive integer. These N finger occlusion areas can be constructed from N elliptical areas that are expanded from the finger touch area. The expanded elliptical areas have the same elliptical orientation as the finger touch area, with their major axis vertices near the center of the screen. A finger occlusion area is the area remaining after subtracting the largest nested elliptical area from the expanded elliptical area.

[0012] When N=1, the largest elliptical area within the expanded elliptical area is the finger touch area. For example, the finger occlusion area 212A in FIG4 is the area left after the elliptical area S1 is expanded and the finger touch area (i.e., the largest elliptical area nested within the elliptical area S1) is removed.

[0013] When N ≥ 2, the N expanded elliptical areas that contribute to the left or right finger occlusion area can be nested layer by layer, gradually increasing in size to construct multiple levels of finger occlusion areas with varying degrees of occlusion. The area remaining after removing the largest elliptical area nested within an elliptical area forms a first-level finger occlusion area. For example, the finger occlusion area 212A in FIG4 is the area remaining after removing the finger touch area from the expanded elliptical area S1, and the finger occlusion area 213A is the area remaining after removing the elliptical area S1 (i.e., the largest elliptical area nested within the elliptical area S2) from the expanded elliptical area S2.

[0014] The sum of the two minor axes of the expanded outermost elliptical area is equal to the width W of the first finger, or the difference between the sum of the two minor axes and the width W of the first finger is less than a first value, such as 2 mm. The first finger may be a thumb.

[0015] When N≥2, the rendering means for reducing the load may include: using different shading rates for the N finger occlusion areas, wherein the game screen uses a lower shading rate for the finger occlusion area closer to the touch center.

[0016] In conjunction with the first aspect, in some embodiments, the multi-level occlusion area may further include a left palm occlusion area. The left palm occlusion area is the area within the rectangular or triangular area in the lower left corner of the screen that is not covered by the left finger touch area or the left finger occlusion area. The diagonal of the rectangular area in the lower left corner is the line segment connecting the center of the left finger touch area to the lower left corner of the screen, and the hypotenuse of the triangular area in the lower left corner is the line segment.

[0017] In conjunction with the first aspect, in some embodiments, the multi-level occlusion area further includes: a right palm occlusion area; the right palm occlusion area is the area within the rectangular or triangular area in the lower right corner of the screen that is not covered by the right finger touch area of ​​the right finger or the right finger occlusion area of ​​the right finger. The diagonal of the rectangular area in the lower right corner is a line segment connecting the center of the right finger touch area to the lower right corner of the screen, and the hypotenuse of the triangular area in the lower right corner is a connecting line segment.

[0018] In a second aspect, the present application provides an electronic device comprising one or more processors and one or more memories; wherein the memories are coupled to the processors, and the one or more memories are used to store computer program code, and the computer program code comprises computer instructions. When the processor executes the computer instructions, the electronic device executes the method described in the first aspect, or any possible implementation of the first aspect.

[0019] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions to enable the electronic device to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0021] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 exemplarily shows an electronic device provided by an embodiment of the present application;

[0023] FIG2 exemplarily shows a scenario in which a user holds an electronic device in landscape mode and plays a game;

[0024] FIG3 exemplarily shows that the finger touch area is simulated as an elliptical area in an embodiment of the present application;

[0025] FIG4 exemplarily shows that multiple nested elliptical areas are expanded based on a finger touch area to construct a multi-level finger occlusion area in an embodiment of the present application;

[0026] FIG5 exemplarily shows an embodiment of the present application in which a rectangular area is expanded based on a finger touch area to construct a palm blocking area;

[0027] FIG6 exemplarily shows a palm occlusion area constructed based on a finger touch area in an embodiment of the present application;

[0028] FIG7 exemplarily shows a load-reducing rendering solution provided by an embodiment of the present application;

[0029] FIG8 exemplarily shows a software system architecture of an electronic device;

[0030] FIG9 shows the interaction between the various software modules in the software system architecture shown in FIG8 ;

[0031] FIG10 shows the overall process of the load-reducing rendering method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0033] FIG1 shows an electronic device 100 provided in an embodiment of the present application.

[0034] The electronic device 100 may be a portable smart device such as a mobile phone, a tablet computer, a handheld computer, or a personal digital assistant (PDA).

[0035] As shown in Figure 1, the electronic device 100 may include: a processor 110, a memory 120, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0036] The sensor module 180 may include a gyro sensor 180B, an acceleration sensor 180E, and a touch sensor 180K.

[0037] The processor 110 can be a processor with powerful processing capabilities and rich storage and peripheral resources, such as an application processor (AP), a neural-network processing unit (NPU), a graphics processing unit (GPU), a modem processor, etc., or it can be a system on a chip (SOC) that integrates multiple processors.

[0038] The processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input and output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0039] The processor 110 may be provided with a cache memory for storing instructions or data that have just been used or are cyclically used by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the cache memory, which can reduce the waiting time of the processor 110 and improve the efficiency of program operation.

[0040] The processor 110 may also be connected to a memory 120. The memory 120 may include a program storage area and a user data storage area. The program storage area may store an operating system and one or more application programs (such as game applications), and the data storage area may store data (such as photos, contacts) created by the user while using the electronic device 100. The memory 120 may be a high-speed random access memory or a non-volatile memory, such as a disk, flash memory, or universal flash storage (UFS). The memory 120 may also be an external memory card, such as a Micro SD card.

[0041] The memory 120 may also store code instructions for the load-reduced rendering method provided in an embodiment of the present application. When the processor 110 reads the code instructions from the memory 120 and runs the code instructions, the electronic device 100 may execute the load-reduced rendering method.

[0042] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0043] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0044] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0045] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0046] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0047] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0048] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0049] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display screen panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniLED, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0050] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0051] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise and brightness. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0052] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0053] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0054] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0055] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0056] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0057] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0058] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0059] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0060] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0061] The headphone jack 170D is used to connect a wired headphone and can be a USB interface, or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0062] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0063] Accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in all directions (generally three axes). It can also detect the magnitude and direction of gravity when electronic device 100 is stationary. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0064] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, in a location different from that of the display screen 194.

[0065] Keys 190 include a power button, volume button, and other buttons. Keys 190 can be mechanical or touch-sensitive. Electronic device 100 can receive key inputs and generate key signal inputs related to user settings and function control of electronic device 100. Motor 191 can generate vibration prompts. SIM card interface 195 is used to connect a SIM card. A SIM card can be connected to and disconnected from electronic device 100 by inserting or removing it from SIM card interface 195.

[0066] The structure shown in FIG1 does not constitute a specific limitation on electronic device 100. Electronic device 100 may include more or fewer components than shown, or may combine or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0067] A game application may be installed on the electronic device 100. To reduce the load of the GPU in rendering the game screen, an embodiment of the present application provides a rendering method for reducing the load. The method will be described in detail below.

[0068] Identify occlusion areas at all levels

[0069] As shown in Figure 2, when a user holds (such as holding it in landscape mode) the electronic device 100 to play a game, certain areas of the game screen displayed on the screen will be blocked by the user's thumb or other hand parts. Generally, the lower left corner area of ​​the screen is designed as the roulette area, and the lower right corner area of ​​the screen is designed as the skill area. The user needs to manipulate the roulette area and the skill area, which will block the screen, resulting in this part of the game screen being invisible or having poor visibility. For example, the pad of the thumb touches and presses on the touch screen, causing the area pressed by the pad to be invisible to the user, while the part of the thumb other than the pad is suspended in the air on the touch screen at a close distance, resulting in poor visibility of the game screen area below it. In addition, the base of the thumb and the palm part connected to it will usually also contact or block the lower left corner area and the lower right corner area of ​​the screen, causing the lower left corner area and the lower right corner area of ​​the game screen to be invisible to the user or have poor visibility.

[0070] The load reduction rendering method provided in the embodiment of this application will accurately identify these occluded areas in different levels, laying the foundation for subsequent differentiated rendering reduction. The occlusion areas at each level may include: finger touch area, finger occlusion area, and palm occlusion area. The method for accurately identifying the occlusion areas at each level is described as follows:

[0071] Finger touch area recognition

[0072] In this article, the finger touch area refers to the area of ​​the screen directly touched by the finger. The game screen displayed within it is invisible to the user and has the highest degree of obstruction. Here, fingers include but are not limited to human fingers, and may also include robot fingers and artificial fingers that simulate human fingers.

[0073] Identification of the finger touch area can be achieved based on touch events reported by the screen (specifically, a touch screen). When a finger press is detected, the screen collects touch information, such as the location, shape, and size of the touch area. This information can be included in the touch event. Based on this touch information, the electronic device 100 can identify the finger touch area.

[0074] Specifically, as shown in FIG3 , the finger touch area can be simulated as an ellipse, and the coordinates of the center (O') of the ellipse, the major and minor axes, and the direction of the ellipse are used as touch information. The direction of the ellipse can refer to the extension direction of the major axis, which can be specifically represented by the angle between the major axis and a side of the screen (such as side XY). The major axis of the ellipse has two vertices, one of which is close to the center of the screen (such as C L 、C R ) and a vertex far from the center of the screen (such as F L 、F R ). Of course, when implementing the embodiments of the present application, people can also regard the extension direction of the minor axis as the direction of the ellipse. However, knowing only the center coordinates and the major and minor axes of the ellipse is not enough to uniquely determine the finger touch area, so it is necessary to introduce the ellipse direction.

[0075] Not limited to what is shown in FIG. 3 , the finger touch area 211A of the left finger and the finger touch area 211B of the right finger do not necessarily need to appear together.

[0076] Finger occlusion area recognition

[0077] When the pad or tip of a finger directly touches the screen, making the finger touch area invisible to the user, the rest of the finger may hover close to the screen, making the screen area below it less visible. This screen area is referred to herein as the finger occlusion area, which does not include the aforementioned finger touch area, to distinguish it from the finger touch area.

[0078] Based on the finger touch area, at least one elliptical area can be expanded outward to construct the finger occlusion area. The area remaining after subtracting the largest elliptical area nested within the expanded elliptical area is the finger occlusion area. The expanded elliptical area and the finger touch area share the elliptical orientation and the long axis vertex near the center of the screen. This also means that the center of the expanded elliptical area is collinear with the center of the finger touch area.

[0079] Specifically, as shown in FIG4 , based on the finger touch area 211A of the left finger, elliptical areas S1 and S2 can be expanded to form the left finger occlusion area. “Based on the finger touch area 211A” may mean that the expanded elliptical areas S1 and S2 share the same elliptical direction and the same major axis vertex C as the finger touch area 211A. L . Major axis vertex C L It is the long axis vertex close to the center of the screen, which reflects the projection position of the left fingertips on the screen.

[0080] Specifically, as shown in FIG4 , based on the finger touch area 211B of the right finger, elliptical areas S1 and S2 can be expanded to form the right finger occlusion area. The expanded elliptical areas S1 and S2 share the same elliptical direction and the long axis vertex C with the finger touch area 211B. R . Major axis vertex C R It is the long axis vertex close to the center of the screen, which reflects the projection position of the right fingertips on the screen.

[0081] Due to the gradually raised three-dimensional structure of the fingertips, the vertical distance from each point on the fingertips to the screen is different. The closer the point or area is to the screen, the more severe the screen occlusion. To this end, the multiple elliptical areas that are expanded and participate in the left or right finger occlusion area can be nested layer by layer, such as S1 nested in S2. They gradually become larger to construct multiple levels of finger occlusion areas with different degrees of occlusion. Among them, the area remaining after subtracting the largest elliptical area nested within an elliptical area forms a first-level finger occlusion area.

[0082] For example, the area remaining after removing the largest elliptical area nested within the expanded ellipse S1 (i.e., the finger touch area 211A (211B)) forms a first-level occlusion area; the area remaining after removing the largest elliptical area nested within the expanded ellipse S2 (i.e., the elliptical area S1) forms another level of occlusion area.

[0083] The visibility of the finger-occluded area decreases as it moves inward toward the touch center. The touch center is the center of the finger's touch area. For example, the visibility of finger-occluded area 212A shown in FIG4 is worse than the visibility of finger-occluded area 213A because it is closer inward to touch area 212A. For another example, the visibility of finger-occluded area 212B shown in FIG4 is worse than the visibility of finger-occluded area 213B because it is closer inward to touch area 212B.

[0084] In particular, the finger touch area 211A ( 211B) can be regarded as a first-level occlusion area with the most severe occlusion degree.

[0085] Based on the gradual reduction in the degree of invisibility of areas 211A (211B), 212A (212B), and 213A (213B) shown in Figure 4, they can be marked as first-level occlusion areas, second-level occlusion areas, and third-level occlusion areas, respectively, and differentiated rendering can be implemented for each level of occlusion areas subsequently.

[0086] The size of the outermost elliptical area expanded to construct the finger occlusion area can be constrained by the finger width W and the screen boundary. Specifically, the sum of the two minor axes of the outermost elliptical area can be equal to or approximately equal to the finger width W, and the elliptical area does not extend beyond the screen boundary. Here, "approximately" can mean smaller than the finger width W, but with a small difference, for example, within 2 mm. Specifically, when only one elliptical area is expanded to construct the finger occlusion area, this elliptical area is the outermost elliptical area, and the sum of its two minor axes can be equal to or approximately equal to the finger width W. Finger width W can be a statistical average of human thumb widths. Finger width W can also be estimated based on the width of the finger touch area (such as the sum of the two minor axes of area 211A), with these two widths being proportional. This is because, generally, the wider the touch area formed by a finger contacting the screen, to some extent, reflects the wider the finger. To improve the accuracy of the estimation of finger width W, further parameters such as user gender, age, and weight can be considered. These parameters can be extracted from the personal information of the logged-in user of electronic device 100. The embodiment of the present application does not limit the method of obtaining these parameters and how to use these parameters to help estimate the finger width.

[0087] Palm occlusion area recognition

[0088] When a finger directly touches the screen and the finger touch area is not visible to the user, in addition to the other parts of the finger being suspended in the air at a close distance on the touch screen, resulting in poor visibility of the screen area below it, the base of the thumb and the palm part connected to it may also usually block the lower left corner and lower right corner of the screen, which may cause poor visibility of the lower left corner and lower right corner of the game screen.

[0089] Herein, the area of ​​the screen blocked by the base of the thumb and the connected palm is referred to as the palm blocking area. The "palm" in "palm blocking area" is not limited to the palm as strictly defined in human anatomy; this term is used herein to distinguish it from the aforementioned finger blocking area.

[0090] Based on the finger touch area, a rectangular area R or a triangular area T can be expanded to construct a palm occlusion area. The palm occlusion area can be the area in the rectangular area R (or triangular area T) that is not covered by the finger touch area and the finger occlusion area.

[0091] Specifically, as shown in FIG5 , based on the finger touch area 211A of the left finger, a rectangular area R1 or a triangular area T1 can be expanded. “Based on the finger touch area 211A” can mean that the diagonal line A of the rectangular area R1 is from the center O of the area 211A. L The line segment between the vertex and the lower left corner of the screen, or the hypotenuse of the triangular area T1 is composed of this line segment.

[0092] Specifically, as shown in FIG5 , based on the finger touch area 211B of the right finger, a rectangular area R2 or a triangular area T2 can be expanded. “Based on the finger touch area 211B” can mean that the diagonal line B of the rectangular area R2 is from the center O of the area 211B. R The line segment between the vertex in the lower right corner of the screen, or the hypotenuse of the triangular area T2 is composed of this line segment.

[0093] Rectangular regions R1 and R2 may not be strict rectangles in the mathematical sense, but rather approximate rectangles. Sides a1, b1, and sides a2, b2 of the rectangles may be modeled or simulated as curved, taking into account the natural curvature of the base of the human thumb and its connection to the palm. Similarly, triangular regions T1 and T2 may not be strict triangles in the mathematical sense, but rather approximate triangles.

[0094] The palm occlusion area may be an area in the rectangular area R (or the triangular area T) that is not covered by the finger touch area and the finger occlusion area.

[0095] As shown in FIG6 , the palm occlusion area 215A of the left palm is the area within the rectangular area R1 (or triangular area T1) that is not covered by the left finger touch area 211A and the left finger occlusion areas 212A and 213A. Similarly, the palm occlusion area 215B of the right palm is the area within the rectangular area R2 (or triangular area T2) that is not covered by the right finger touch area 211B and the right finger occlusion areas 212B and 213B.

[0096] It can be seen that the center O LThe closer to the lower left corner of the screen, the smaller the left palm occlusion area 215A; the center O R The closer to the lower right corner of the screen, the smaller the right palm occlusion area 215B.

[0097] When the distance from the center of the touch area to the left or right bottom corner of the screen is small enough, the left or right palm occlusion area can be ignored. This means that the left or right finger has not fully extended to the middle of the screen, resulting in the left or right palm not blocking the screen. L When the distance to the lower left corner of the screen is less than a certain distance, the left palm blocking area 215A may be ignored. R When the distance to the lower right corner of the screen is less than a specific distance, the right palm occlusion area 215B may be ignored. The specific distance may be the statistical average of the projected lengths of a human thumb on the screen when the thumb is naturally bent or straightened.

[0098] The palm occlusion area has a lower degree of occlusion than the finger touch area, and different rendering can be applied to different areas later. As for whether the palm occlusion area or the finger occlusion area has a higher or lower degree of occlusion, this embodiment of the application does not limit it and can be determined according to actual application needs.

[0099] Differentiated rendering

[0100] After hierarchically identifying the above-mentioned occlusion areas, the load-reducing rendering method provided in the embodiment of the present application can be further adapted to differentiated rendering reduction methods, thereby reducing the GPU rendering load without affecting the user's gaming experience.

[0101] Means of reducing rendering may include but are not limited to: reducing resolution, reducing image texture, feature point cropping, rendering culling, etc. Among them, reducing resolution will reduce image clarity, reducing image texture will reduce image details, feature point cropping means reducing image feature points, and rendering culling means not rendering the culled area. In addition, means of reducing rendering may also include: implementing variable rate shading (VRS) for areas with different degrees of occlusion. VRS can shade in groups of 2 or 4 pixels during rendering, and reuse the shading results for surrounding pixels, reducing the GPU's computing load by selectively rendering and shading the picture.

[0102] In an embodiment of the present application, differentiated down-rendering may refer to using a lower-load rendering for the screen of the occluded area that is closer to the touch center inward, and the touch center is the center of the finger touch area. In this way, a greater degree of down-rendering can be implemented for the occluded area with a higher degree of occlusion. Among them, rendering culling can be considered as the maximum down-rendering, resulting in no content display, which is applicable to the finger touch area. Specifically, the screen of the finger touch area can be rendered culled, and the screen of the finger occluded area can be processed by one or more of the following: reducing image texture, reducing image resolution, reducing feature points, and reducing shading rate.

[0103] For example, as shown in FIG7 , the occlusion levels of finger touch area 211A, finger occlusion area 212A, and finger occlusion area 213A are gradually reduced. Assuming that the occlusion level of palm occlusion area 215A is lower than that of finger occlusion area 213A, as shown in FIG7 , gradually reduced rendering methods can be applied to finger touch area 211A, finger occlusion area 212A, finger occlusion area 213A, and palm occlusion area 215A: rendering culling is applied to finger touch area 211A; VRS is applied to finger occlusion area 212A, finger occlusion area 213A, and palm occlusion area 215A, using 4×4 shading rate, 4×2 shading rate, and 2×2 shading rate, respectively.

[0104] This differentiated rendering reduction not only reduces the GPU rendering load, but can also present different rendering effects for different areas with different degrees of occlusion in a more refined manner without affecting the user's gaming experience.

[0105] The load-reduced rendering method provided in the embodiments of the present application can achieve a seamless development of gaming applications and is widely applicable to various gaming applications without requiring the game application developer to modify functionality. This is due to the software system architecture of the electronic device 100 provided in the embodiments of the present application, which designs a pathway between touch information (carried in input events) and the image processing module.

[0106] FIG8 exemplarily shows the software system architecture of the electronic device 100 .

[0107] As shown in FIG8 , the software system architecture of the electronic device 100 may include an application (APP) layer, an application framework (FWK) layer, and a kernel layer. The following sections will expand on each of these:

[0108] The application layer may include one or more applications, such as a game application, which may provide game screen content.

[0109] The application framework layer may include: input management service (IMS), window management service (WMS), UI framework layer, and image processing module.

[0110] IMS can include: Event Hub, InputReader, and InputDispatcher, which work together to transmit and distribute input events reported by the touch screen to the appropriate window, such as window1 of a game application. Among them, Event Hub can monitor device nodes (such as / dev / input / eventX) through the iNotify and Epoll mechanisms and read the raw input events of the device nodes; InputReader will continuously read the raw input events from EventHub in a loop, process these raw input events, and then hand them over to InputDispatcher; InputDispatcher stores information about all windows in WMS (WMS will update the window information to InputDispatcher in real time), so that InputDispatcher can dispatch input events to the appropriate window.

[0111] WMS can be used to manage all windows (window) in the system, such as window1. Window1 can be the window of a game application, which can be created (instantiated) when the game application is started. Along with the instantiation of window1, a ViewRootImpl object can be instantiated in the UI framework layer, and the ViewRootImpl object can be used to listen to input events dispatched to window1 by InputDispatcher. The embodiment of the present application improves ViewRootImpl. After the improved ViewRootImpl object listens to the input event dispatched to window1 by InputDispatcher, it can also pass the touch information carried in the input event to the image processing module. In addition, since the input event is listened to by the ViewRootImpl object, the game application also receives the input event and understands whether there is a user's finger touching the screen.

[0112] After receiving the touch information passed by the ViewRootImpl object, the image processing module can be used to intercept rendering instructions, identify various levels of occlusion areas, and reduce the rendering load. For the specific implementation of identifying various levels of occlusion areas and reducing the rendering load, please refer to the previous and following articles and will not be repeated here.

[0113] In addition to IMS, WMS and image processing modules, the application framework layer can also include a layer compositor (SurfaceFlinger). SurfaceFlinger can be used to perform layer synthesis processing on the picture frames rendered by the GPU, generate an interface including the picture frames, and send the interface including the picture frames to the display screen for display.

[0114] The kernel layer may include: a touchscreen driver, a GPU driver, etc. The touchscreen driver can be used to control the operation of the touchscreen and receive touch signals reported by the touchscreen, while the GPU driver can be used to control the GPU to perform rendering operations. When a user's finger is detected touching the touchscreen, the touchscreen can simulate the finger pressing area as an elliptical area and report ellipse information to the touchscreen driver, such as the coordinates of the ellipse's center (O'), major and minor axes, and the ellipse's direction. This ellipse information is carried as touch information in the input event reported to the touchscreen driver.

[0115] In addition to the above layers, the system architecture of the electronic device 100 may also include a local service layer. The local service layer may include one or more graphics libraries, such as the open graphics library (OpenGL) and the open graphics library for embedded systems (OpenGL ES). The graphics drawing interface included in the graphics library can be called to configure the GPU to render the picture frame.

[0116] Based on the software system architecture exemplified in Figure 8, Figure 9 illustrates the coordination between the various software modules, particularly the image processing module's process for intercepting rendering instructions, identifying occlusion areas at various levels, and reducing rendering load. A game application may include a main thread and a render thread. The image processing module may include a rendering instruction interception module, a module for identifying occlusion areas at various levels, and a rendering module for reducing rendering load.

[0117] The interaction process of each module can be as follows:

[0118] 1. The main thread of the game application can synchronize the game screen content to the rendering thread.

[0119] 2. After the rendering thread obtains the game screen content, it can call the first graphics drawing interface in the graphics library (such as OpenGL, OpenGL ES, etc.) to issue rendering instructions.

[0120] At the same time, the game screen content to be rendered by the game application can be cached, for example, temporarily stored in cache 1, so as to be read when the GPU is rendering. Cache 1 can be a partial cache area on the memory 120 in the electronic device 100.

[0121] 3. The first graphics drawing interface in the graphics library (such as OpenGL) is called to forward the rendering instruction to the rendering instruction interception module.

[0122] Calling the first graphics drawing interface in the graphics library does not pass the rendering instructions from the rendering thread directly to the GPU, but instead forwards the rendering instructions to the rendering instruction interception module in the image processing module. This is the rendering instruction interception mentioned in this article.

[0123] Specifically, rendering instruction interception can occur during the period when the image processing module receives touch information transmitted by the ViewRootImpl object, during which time the user's finger touches the screen. That is, during the period when the user's finger touches the screen, the game application can call the first graphics drawing interface in the graphics library. In addition, during the period when the user's finger does not touch the screen, the game application can call the second graphics drawing interface in the graphics library to pass the rendering instruction directly to the GPU. This is also the normal rendering mentioned in this article, which is relative to the rendering method that reduces the rendering load.

[0124] In the graphics library, the first graphics drawing interface is a graphics drawing interface introduced in the embodiment of the present application to implement rendering instruction interception, and the second graphics drawing interface is a traditional graphics drawing interface provided by the graphics library.

[0125] 4. After receiving the rendering instruction, the rendering instruction interception module can further pass the rendering instruction to the occlusion area recognition modules at all levels.

[0126] The occlusion region recognition module identifies various levels of occlusion based on touch information, such as the coordinates of the ellipse's center (O'), its major and minor axes, and its orientation. This information serves as the foundation for subsequent differentiated rendering. For details on how to identify these levels of occlusion, please refer to the previous and following sections and will not be elaborated on here.

[0127] The touch information may be transmitted to the image processing module by the ViewRootImpl object. For details, please refer to the relevant content in the embodiment of FIG8 , which will not be described in detail here.

[0128] 5. After identifying the occlusion areas at each level, the occlusion area identification modules at each level may inform the load-reducing rendering module of the identification results (ie, the occlusion areas at each level).

[0129] Here, the recognition result may include indication information of each level of occlusion area (such as finger touch area, finger occlusion area, palm occlusion area), such as position, size, and shape, which indicates which pixels are covered by each level of occlusion area on the screen.

[0130] 6. After knowing the occlusion areas at each level, the load-reducing rendering module can adapt differentiated rendering reduction methods to each level of occlusion area, obtain differentiated rendering reduction solutions, and send rendering instructions to the GPU. The rendering instructions can carry the differentiated rendering reduction solutions.

[0131] A differentiated downscaling scheme can be used to apply a higher degree of downscaling to occluded areas with higher occlusion levels, and a lower degree of downscaling to occluded areas with lower occlusion levels. For example, downscaling with render culling can be applied to area 211A in Figure 4, while downscaling with variable color rates of 4×4, 4×2, and 2×2 can be applied to areas 212A, 213A, and 215A, respectively. For details on how to implement differentiated downscaling, please refer to the relevant content above and below, and will not be repeated here.

[0132] 7. After receiving the rendering instructions sent by the load-reduced rendering module, the GPU can read the game screen content to be rendered from cache 1, render the game screen content according to the differentiated load-reduced rendering scheme, and finally store the rendered screen frame in cache 2.

[0133] Cache 2 may be a cache area specified by a rendering instruction, and it may be a partial cache area on the memory 120 in the electronic device 100 .

[0134] 8. The layer compositor (SurfaceFlinger) takes the rendered frame from cache 2 for layer synthesis. Then, SurfaceFlinger sends the synthesized frame to the frame buffer for storage to complete the display.

[0135] The frame buffer is a storage space, either in video memory or internal memory, used to store rendering data processed by or about to be retrieved from the graphics chip. The contents of the frame buffer correspond to the interface displayed on display screen 194 . It can be simply understood as a cache corresponding to the content displayed on display screen 194 . Modifying the contents of the frame buffer modifies the content on display screen 194 .

[0136] Specifically, each time SurfaceFlinger receives a vertical synchronization (vsync) signal, it takes out the picture frame from cache 2 for layer synthesis and sends it to the display screen for refresh display. The vsync signal is a synchronization signal generated by the hardware module corresponding to the display screen, with a period consistent with the screen refresh rate. The screen refresh rate indicates the number of times the display screen on the electronic device refreshes the display frame within 1 second. Generally, the refresh rate of the display screen of an electronic device is related to the fixed parameters of the display screen and is a fixed value. For example, the refresh rate can be 60 Hz.

[0137] The software system architecture provided in the embodiment of the present application designs a path from touch information (carried in input events) to the image processing module, and can also achieve low latency, so that there is no difference in user experience between reduced-load rendering and ordinary rendering.

[0138] Typically, games use triple buffering, requiring three frames from rendering to display, two of which are cached. For a 60-frame game (with a frame rate of approximately 16.7 milliseconds), triple buffering will result in a delay of approximately 50 milliseconds. For example, if the delay between a user touch and the game screen displaying on the screen does not exceed 80 milliseconds, this requires that the delay between the touch information being transmitted to the image processing module and the processing delay of the image processing module executing the rendering instruction interception, identifying various levels of occlusion areas, and outputting a differentiated, reduced-load rendering solution must not exceed a total of 30 milliseconds. The fast path from touch information to the image processing module provided by the embodiments of the present application can meet this requirement. The embodiments of the present application can achieve a touch information transmission time of less than 25 milliseconds and a processing time of no more than 5 milliseconds. However, for low-frame-rate games, such as a 30-frame game, since the frame rate is already 33 milliseconds, triple buffering takes approximately 100 milliseconds from rendering to display, a significant delay. Considering the user experience, electronic devices can choose not to perform reduced-load rendering and only perform normal rendering.

[0139] FIG10 shows the overall process of the load-reducing rendering method provided by an embodiment of the present application.

[0140] S11. The game begins.

[0141] The electronic device can start the game application due to user operation (such as clicking the desktop icon of the game application), and needs to render the game screen provided by the game application and display the game screen provided by the game application on the display screen.

[0142] As mentioned above, when a user holds (such as holding it horizontally) an electronic device to play a game, certain areas of the game screen displayed on the screen will be blocked by the user's thumb or other hand parts. Generally, the lower left corner area of ​​the screen is designed as the roulette area, and the lower right corner area of ​​the screen is designed as the skill area. The user needs to manipulate the roulette area and the skill area, which will block the screen, resulting in this part of the game screen being invisible or having poor visibility. For example, the pad of the thumb touches and presses on the touch screen, causing the area pressed by the pad to be invisible to the user, while the part of the thumb other than the pad is suspended in the air at a close distance on the touch screen, causing the visibility of the game screen area below it to be poor. In addition, the base of the thumb and the palm part connected to it will usually contact or block the lower left corner area and the lower right corner area of ​​the screen, causing the lower left corner area and the lower right corner area of ​​the game screen to be invisible to the user or have poor visibility.

[0143] The reduced-load rendering method provided by this embodiment of the application accurately identifies these obscured areas in a graded manner, laying the foundation for subsequent differentiated reduced-rendering. This is not limited to gaming applications; this embodiment of the application can also be applied to other applications that provide graphical user interfaces and accept user touch input, reducing the rendering load of the graphical user interface when a user touches the screen.

[0144] S12. The electronic device may determine whether a touch operation is detected on the touch screen. If so, it may execute S13-S15; otherwise, it may execute S16. That is, while the user's finger is touching the screen, the electronic device performs load-reduced rendering; while the user's finger is not touching the screen, the electronic device performs normal rendering.

[0145] The game screen displayed on the screen may also include a directional wheel and skill keys. The directional wheel can be used to control the movement of a character or other subject in the game, and the skill keys can be used to perform controls such as launching game skills. The directional wheel and skill keys can be located in the lower left corner or lower right corner of the screen, or in the lower right corner or lower left corner of the screen, respectively.

[0146] The touch operation may specifically be an operation on a directional wheel or a skill key. That is, the electronic device may trigger execution of S13-S15 upon detecting a touch operation on the directional wheel or a skill key, or upon detecting a touch operation at the lower right corner or the lower left corner of the screen.

[0147] In addition, before determining whether to execute S13-S15 or S16, the electronic device can also determine whether the game application is a low frame rate game application, such as a 30-frame game. If it is not a low frame rate game application, it determines to execute S13-S15, otherwise it executes S16.

[0148] In this paper, the time delay requirement between the user touching the screen and the game screen displaying (maximum delay T max ) to define whether a game application is a low frame rate game application. Specifically, if the latency of a game application executing triple buffering has reached or exceeded T max , it can be considered a low frame rate gaming application. Or, more strictly, if the latency of a gaming application executing triple buffering exceeds T max -T reserve , it can be considered as a low frame rate game application. reserve This represents the latency reserved for transmitting touch information to the image processing module and the image processing latency, for example, 30 milliseconds. This value is affected by the processing performance of the relevant hardware and can be adjusted according to the actual implementation scenario. The latency of triple buffering in gaming applications is determined by the game application's frame rate, R, and is equal to 3*(1000 / R) milliseconds.

[0149] S13. The electronic device can obtain touch information.

[0150] Touch information may include information such as the position, shape, and size of the finger touch area, which may be carried in a touch event. Based on the touch information, the electronic device can identify the finger touch area. Specifically, as shown in FIG3 , the finger touch area can be simulated as an ellipse, and the center (O') coordinates, major and minor axes, and ellipse direction of the ellipse are used as touch information. Among them, the ellipse direction may refer to the extension direction of the major axis, which may be specifically represented by the angle between the major axis and a side of the screen (such as side XY).

[0151] S14. The electronic device may identify different levels of occlusion areas based on the touch information.

[0152] Each level of occlusion area may include: finger touch area, finger occlusion area, and palm occlusion area.

[0153] The identification of the finger touch area can be achieved based on touch events reported by the screen (specifically, a touch screen). The finger touch area is the area of ​​the screen directly touched by the finger. The game screen displayed within it is invisible to the user, has the highest degree of obstruction, and has the lowest visibility. The finger touch area can be simulated as an ellipse.

[0154] There can be N finger occlusion areas (N is a positive integer), constructed by expanding N elliptical areas outward from the finger touch area. The expanded elliptical areas have the same elliptical orientation as the finger touch area, and their long axis vertices near the center of the screen coincide. A finger occlusion area is the area remaining after subtracting the largest elliptical area nested within an expanded elliptical area.

[0155] When N=1, the largest elliptical area within the expanded elliptical area is the finger touch area. For example, the finger occlusion area 212A in FIG4 is the area left after the finger touch area (i.e., the largest elliptical area nested within the elliptical area S1) is subtracted from the expanded elliptical area S1.

[0156] When N ≥ 2, the N expanded elliptical areas that contribute to the left or right finger occlusion area can be nested layer by layer, gradually increasing in size to construct multiple levels of finger occlusion areas with varying degrees of occlusion. The area remaining after removing the largest elliptical area nested within an elliptical area forms a first-level finger occlusion area. For example, the finger occlusion area 212A in FIG4 is the area remaining after removing the finger touch area from the expanded elliptical area S1, and the finger occlusion area 213A is the area remaining after removing the elliptical area S1 (i.e., the largest elliptical area nested within the elliptical area S2) from the expanded elliptical area S2.

[0157] The closer the finger is to the touch center, the less visible the area blocked by the finger becomes. The touch center is the center of the finger's touch area.

[0158] Specifically, as shown in FIG4 , the elliptical areas S1 and S2 expanded based on the left finger touch area 211A are divided into two levels of left finger occlusion areas after removing the largest ellipse nested within them. The expanded elliptical areas S1 and S2 share the same elliptical direction and the same major axis vertex C as the touch area 211A. L .

[0159] Specifically, as shown in FIG4 , the elliptical areas S1 and S2 expanded based on the touch area 211B of the right finger, after removing the largest ellipse nested within them, are the areas 212B and 213B left respectively, which are the two levels of right finger occlusion areas with different occlusion degrees. Among them, the expanded elliptical areas S1 and S2 share the same elliptical direction and the same major axis vertex C as the touch area 211B. R .

[0160] The size of the outermost elliptical region expanded to form the finger occlusion area can be constrained by the finger width W and the screen border. Specifically, the sum of the two minor axes of the outermost elliptical region can be equal to or approximately equal to the finger width W, and the elliptical region does not extend beyond the screen border. Specifically, when only one elliptical region is expanded to form the finger occlusion area, this elliptical region becomes the outermost elliptical region, and the sum of its two minor axes can be equal to or approximately equal to the finger width W.

[0161] The palm occlusion area may be constructed by expanding the finger touch area into a rectangular area R or a triangular area T. The palm occlusion area may be an area in the rectangular area R (or triangular area T) that is not covered by the finger touch area or the finger occlusion area.

[0162] Specifically, as shown in FIG5 , the area of ​​the rectangular area R1 or triangular area T1 expanded based on the finger touch area 211A of the left finger, which is not covered by the finger touch area 211A of the left finger and the finger blocking areas 212A and 213A of the left finger, is the palm blocking area 215A of the left palm. “Based on the finger touch area 211A” may mean that the diagonal line A of the rectangular area R1 is from the center O of the area 211A. L The line segment between the vertex and the lower left corner of the screen, or the hypotenuse of the triangular area T1 is composed of this line segment.

[0163] Specifically, as shown in FIG5 , the area of ​​the rectangular area R2 or triangular area T2 expanded based on the finger touch area 211B of the right finger that is not covered by the finger touch area 211A of the left finger and the finger blocking areas 212A and 213A of the left finger is the palm blocking area 215B of the right palm. “Based on the finger touch area 211B” may mean that the diagonal line B of the rectangular area R2 is from the center O of the area 211B. R The line segment between the vertex in the lower right corner of the screen, or the hypotenuse of the triangular area T2 is composed of this line segment.

[0164] When the distance from the center of the touch area to the lower left or right corner of the screen is small enough to not consider the area blocked by the left or right palm, the left or right finger does not fully extend to the middle of the screen, resulting in the left or right palm not blocking the screen.

[0165] S15. The electronic device can render images in various levels of occlusion areas with reduced load.

[0166] Specifically, the electronic device may employ reduced-load rendering for the occluded area closer to the touch center. Means for reducing rendering load may include, but are not limited to, one or more of the following: rendering culling, reducing image texture, reducing image resolution, reducing feature points, and reducing shading rate.

[0167] For example, the electronic device may implement a greater degree of resolution reduction, a greater degree of image texture reduction, and cropping of more feature points for the occluded area that is closer to the touch center.

[0168] Among them, rendering culling can be considered as the maximum reduction in rendering, resulting in no content being displayed. Specifically, the electronic device can implement rendering culling, a rendering method that reduces the load, on the finger touch area, and perform one or more of the following processing on the images in the finger occlusion area and the palm occlusion area: reducing the resolution, reducing the image texture, and cropping feature points. Among them, the closer the finger occlusion area is to the touch center, the greater the degree of reduction in resolution, image texture, and cropping of feature points.

[0169] In addition, rendering load reduction methods can also include: implementing VRS for areas with different degrees of occlusion. For example, different shading rates can be used for finger occlusion areas with different degrees of occlusion. The images of finger occlusion areas closer to the touch center use a lower shading rate, and the images of finger occlusion areas further out use a higher shading rate. If the degree of occlusion of the palm occlusion area is considered higher than that of the finger occlusion area, the shading rate used for the palm occlusion area can be lower than the shading rate used for the finger occlusion area.

[0170] In this way, users can observe that after downscaling is implemented, the images in various occluded areas are blurrier than those in other screen areas where downscaling is not implemented. Furthermore, the image becomes blurrier the closer to the touch center. This is because downscaling is implemented in these occluded areas, and their images may be inferior to those in other screen areas where downscaling is not implemented, in one or more aspects, such as image texture, resolution, number of feature points, and shading rate.

[0171] S16. The electronic device may perform normal rendering.

[0172] For ordinary rendering, please refer to the relevant description in the previous embodiment, which will not be repeated here.

[0173] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0174] An embodiment of the present application further provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned various method embodiments.

[0175] The present application also provides a chip system, comprising a processor coupled to a memory, the processor executing a computer program stored in the memory to implement the steps of any method embodiment of the present application. The chip system can be a single chip or a chip module composed of multiple chips.

[0176] The term "user interface (UI), or interface for short" in the specification and drawings of this application refers to the media interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as <head> and <body>. A node corresponds to a control or attribute in the interface, and the node is presented as user-visible content after parsing and rendering. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is a source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code, such as HTML through <body>. 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0177] A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operations that uses graphics. It can be an icon, window, control, or other interface element displayed on the display of an electronic device. Controls can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.

[0178] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk).

[0179] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0180] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A rendering method for reducing load, characterized in that: include: The electronic device displays a picture provided by the first application on the screen; The electronic device detects a touch operation of a user's finger on the screen and obtains touch information of the touch operation; The electronic device determines a multi-level shielding area based on the touch information, wherein the multi-level shielding area includes: a finger touch area and a finger shielding area, wherein the finger shielding area covers the periphery of the finger touch area; The electronic device performs reduced-load rendering on the image of the multi-level occlusion area, wherein the image of the occlusion area that is closer to the touch center is rendered with a reduced load, and the touch center is the center of the finger touch area. The reduced-load rendering includes one or more of the following: rendering culling, reducing image texture, reducing image resolution, reducing feature points, and reducing shading rate.

2. The method according to claim 1, characterized in that The rendering of the image in the blocked area closer to the touch center adopts a lower load, including: rendering and culling the image in the finger touch area, and performing one or more of the following processing on the image in the finger blocked area: reducing image texture, reducing image resolution, reducing feature points, and reducing shading rate.

3. The method according to claim 1 or 2, characterized in that The first application includes a game application, and the screen also displays: a steering wheel and a skill key, the touch operation acts on the steering wheel or the skill key, and the steering wheel and skill key are respectively distributed in the lower left corner, lower right corner of the screen or respectively distributed in the lower right corner and lower left corner of the screen.

4. The method according to any one of claims 1 to 3, characterized in that The finger touch area is an elliptical area, and the touch information includes the center coordinates, the major and minor axes, and the direction of the elliptical area, wherein the direction of the elliptical area is the extension direction of the major axis.

5. The method according to claim 4, characterized in that The finger occlusion areas include N areas, where N is a positive integer, and the N finger occlusion areas are constructed based on N elliptical areas expanded from the finger touch area, wherein a finger occlusion area is an area left after the largest elliptical area nested inside is subtracted from an expanded elliptical area; the expanded elliptical area has the same elliptical direction as the finger touch area, and a long axis vertex close to the center of the screen.

6. The method according to claim 5, characterized in that When N=1, the largest elliptical area nested inside the one expanded elliptical area is the finger touch area.

7. The method according to claim 5 or 6, characterized in that When N is greater than or equal to 2, the expanded N elliptical areas are nested layer by layer and gradually become larger.

8. The method according to claim 7, characterized in that The sum of two short axes of the extended outermost elliptical area is equal to the width W of the first finger, or the difference between the sum of two short axes of the extended outermost elliptical area and the width W of the first finger is less than a first value; the first finger includes the thumb.

9. The method according to claim 7 or 8, characterized in that The N finger occlusion areas use different shading rates, wherein the game screen of the finger occlusion area closer to the touch center uses a lower shading rate.

10. The method according to any one of claims 1 to 9, characterized in that The multi-level occlusion area also includes: a left palm occlusion area; The left palm occlusion area is an area in the rectangular area or triangular area in the lower left corner of the screen that is not covered by the finger touch area of ​​the left finger and the finger occlusion area of ​​the left finger; wherein the diagonal line of the rectangular area in the lower left corner is the line segment from the center of the left finger touch area to the lower left corner of the screen, and the hypotenuse of the triangular area is the line segment.

11. The method according to any one of claims 1 to 10, characterized in that The multi-level occlusion area also includes: a right palm occlusion area; The right palm occlusion area is an area in the rectangular area or triangular area in the lower right corner of the screen that is not covered by the right finger touch area of ​​the right finger and the right finger occlusion area of ​​the right finger; wherein the diagonal line of the rectangular area in the lower right corner is the line segment from the center of the right finger touch area to the lower right corner of the screen, and the hypotenuse of the triangular area is the line segment.

12. An electronic device, characterized in that: The method comprises one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, wherein the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the method as described in any one of claims 1 to 11 is executed.

13. A chip system, the chip system is applied to electronic equipment, the chip system comprises one or more processors, characterized in that: The processor is configured to call computer instructions so as to execute the method according to any one of claims 1 to 11.

14. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the method according to any one of claims 1 to 11 is executed.

Images