Load-reduced rendering method and electronic equipment

Abstract

The embodiment of the invention provides a load-reducing rendering method, which comprises the following steps: when a user touches a screen with fingers, electronic equipment can identify all levels of shielding areas (a finger touch area, a finger shielding area and a palm shielding area) based on touch information of the finger touch area, and differentiated load-reducing rendering is performed on all levels of shielding areas, so that the user experience is improved. For example, a load-reducing rendering mode of rendering and removing a finger touch area can be implemented, and a rendering mode of reducing resolution, reducing image texture and cutting feature points can be implemented for a finger shielding area and a palm shielding area. Wherein for finger shielding areas with different visibility, resolution reduction, image texture reduction and feature point cutting with different degrees can be implemented. In this way, the game picture rendering load of the GPU can be reduced, and the user experience is not affected.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to a load-reducing rendering method and electronic equipment. Background Art

[0002] Game screen rendering has relatively high requirements on the performance of the graphics processing unit (GPU), and long-term high-resolution game screen rendering will cause serious rendering load on the GPU, which may cause problems such as game screen freezes, high device power consumption and heat, and short battery life. Summary of the invention

[0003] In the first aspect, an embodiment of the present application provides a rendering method with reduced load, which may include: an electronic device displays a picture provided by a first application on the screen, detects a touch operation of a user's finger on the screen, and obtains touch information of the touch operation. Then, the electronic device determines a multi-level occlusion area based on the touch information, and the multi-level occlusion area includes: a finger touch area, and a finger occlusion area, wherein the finger occlusion area is covered on the periphery of the finger touch area. In addition, the electronic device performs load-reduced rendering on the pictures in the multi-level occlusion area, wherein the pictures in the occlusion area closer to the touch center are rendered with reduced load, and the touch center is the center of the finger touch area. The load-reduced rendering includes one or more of the following: rendering culling, reducing image texture, reducing image resolution, reducing feature points, and reducing shading rate.

[0004] In the first aspect, by implementing the method of the first aspect, the electronic device can more finely determine the areas with different degrees of occlusion and adapt differentiated rendering reduction methods, thereby reducing the GPU rendering load without affecting the user's gaming experience.

[0005] In combination with the first aspect, in some embodiments, the images of the occluded areas closer to the touch center are rendered with a reduced load, which may specifically include: rendering and culling the images of the finger touch area, and performing one or more of the following processing on the images of the finger occluded area: reducing image texture, reducing image resolution, reducing feature points, and reducing shading rate. Among them, the resolution can be reduced to a greater extent, the image texture can be reduced, and more feature points can be cropped for the finger occluded area closer to the touch center.

[0006] In combination with the first aspect, in some embodiments, the first application may be a game application, and the screen also displays: a steering wheel, a skill key, and touch operations are performed on the steering wheel or the skill keys. The steering wheel and the skill keys are respectively distributed in the lower left corner, lower right corner, or respectively distributed in the lower right corner, lower left corner of the screen.

[0007] The first application is not limited to game applications, and may also be other applications that provide a graphical user interface and receive user touch input. The method of the first aspect is also applicable to such applications to reduce the rendering load of the graphical user interface when the user touches the screen with a finger.

[0008] In combination with the first aspect, in some embodiments, the finger touch area is an elliptical area, and the touch information includes the center coordinates, major and minor axes, and the direction of the elliptical area, wherein the direction of the elliptical area is the extension direction of the major axis.

[0009] There can be N finger occlusion areas, where N is a positive integer. The N finger occlusion areas can be constructed based on the N elliptical areas extended from the finger touch area. The extended elliptical area has the same elliptical direction as the finger touch area, and the long axis vertex is close to the center of the screen. A finger occlusion area is the area left after the largest elliptical area nested inside is subtracted from the extended elliptical area.

[0010] When N=1, the largest elliptical area within the expanded elliptical area is the finger touch area. Figure 4 The finger blocking area 212A in the figure is the area left after the elliptical area S1 is expanded and the finger touch area (i.e., the largest elliptical area nested in the elliptical area S1) is removed.

[0011] When N ≥ 2, the N elliptical areas that are expanded and participate in forming the left or right finger occlusion area can be nested layer by layer, and they gradually become larger to construct multiple levels of finger occlusion areas with different occlusion degrees. Among them, the area left after removing the largest elliptical area nested inside an elliptical area forms a first-level finger occlusion area. For example, Figure 4 The finger occlusion area 212A is the area left after the elliptical area S1 is expanded and the finger touch area is removed, and the finger occlusion area 213A is the area left after the elliptical area S2 is expanded and the elliptical area S1 (ie, the largest elliptical area nested in the elliptical area S2) is removed.

[0012] The sum of the two short axes of the expanded outermost elliptical area is equal to the width W of the first finger, or the difference between the width W of the first finger and the width W of the first finger is less than a first value, such as 2 mm. The first finger may be a thumb.

[0013] When N≥2, the rendering means for reducing the load may include: using different shading rates for the N finger occlusion areas, wherein the game screen of the finger occlusion area closer to the touch center uses a lower shading rate.

[0014] In combination with the first aspect, in some embodiments, the multi-level occlusion area may further include: a left palm occlusion area. The left palm occlusion area is an area in the rectangular area or triangular area in the lower left corner of the screen that is not covered by the finger touch area of ​​the left finger and the finger occlusion area of ​​the left finger. Among them, the diagonal line of the rectangular area in the lower left corner is the line segment from the center of the left finger touch area to the lower left corner of the screen, and the hypotenuse of the triangular area in the lower left corner is the line segment.

[0015] In combination with the first aspect, in some embodiments, the multi-level occlusion area further includes: a right palm occlusion area; the right palm occlusion area is an area in the rectangular area or triangular area at the lower right corner of the screen that is not covered by the right finger touch area of ​​the right finger and the right finger occlusion area of ​​the right finger. The diagonal line of the rectangular area at the lower right corner is a line segment from the center of the right finger touch area to the lower right corner of the screen, and the hypotenuse of the triangular area at the lower right corner is a line segment.

[0016] In a second aspect, the present application provides an electronic device, comprising one or more processors and one or more memories; wherein the memories are coupled to the processors, and the one or more memories are used to store computer program codes, and the computer program codes include computer instructions, and when the processor executes the computer instructions, the electronic device executes the method described in the first aspect, or any possible implementation method of the first aspect.

[0017] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions so that the electronic device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, causes the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0019] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The electronic device provided by the embodiment of the present application is exemplarily shown;

[0021] Figure 2 The example shows a scenario where a user holds an electronic device in landscape mode and plays a game;

[0022] Figure 3It is exemplarily shown that the finger touch area in the embodiment of the present application is simulated as an elliptical area;

[0023] Figure 4 The embodiment of the present application exemplarily shows that multiple nested elliptical areas are expanded based on the finger touch area to construct a multi-level finger blocking area;

[0024] Figure 5 It is exemplarily shown that in the embodiment of the present application, a rectangular area is expanded based on the finger touch area to construct a palm blocking area;

[0025] Figure 6 The palm occlusion area constructed based on the finger touch area in the embodiment of the present application is exemplarily shown;

[0026] Figure 7 A rendering solution for reducing load provided by an embodiment of the present application is exemplarily shown;

[0027] Figure 8 The software system architecture of the electronic device is exemplarily shown;

[0028] Fig. 9 Shows Figure 8 The mutual cooperation between various software modules in the software system architecture shown;

[0029] Fig.10 The overall process of the load reduction rendering method provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0031] Figure 1 An electronic device 100 provided in an embodiment of the present application is shown.

[0032] The electronic device 100 may be a portable smart device such as a mobile phone, a tablet computer, a handheld computer, or a personal digital assistant (PDA).

[0033] like Figure 1As shown, the electronic device 100 may include: a processor 110, a memory 120, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0034] The sensor module 180 may include a gyro sensor 180B, an acceleration sensor 180E, and a touch sensor 180K.

[0035] Processor 110 can be a processor with powerful processing capabilities and rich storage and peripheral resources, such as an application processor (AP), a neural-network processing unit (NPU), a graphics processing unit (GPU), a modem processor, etc., or it can be a system on chip (SOC) that integrates multiple processors.

[0036] The processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input and output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0037] The processor 110 may be provided with a cache memory for storing instructions or data just used or cyclically used by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the cache memory, which can reduce the waiting time of the processor 110 and improve the program running efficiency.

[0038] The processor 110 can also be connected to the memory 120, which can include a program storage area and a user data storage area, wherein the program storage area can store an operating system and one or more application programs (such as game applications), and the data storage area can store data (such as photos, contacts) created by the user in the process of using the electronic device 100. The memory 120 can be a high-speed random access memory or a non-volatile memory, such as a disk, a flash memory, a universal flash storage (UFS), etc. The memory 120 can also be an external memory card, such as a Micro SD card.

[0039] The memory 120 may also store code instructions of the load-reduced rendering method provided in the embodiment of the present application. When the processor 110 reads the code instructions from the memory 120 and runs the code instructions, the electronic device 100 may execute the load-reduced rendering method.

[0040] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0041] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0042] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0043] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0044] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0045] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0046] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0047] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display screen panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), miniled, microled, micro-oled, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0048] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0049] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0050] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0051] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0052] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0053] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0054] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0055] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0056] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0057] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0058] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0059] The earphone interface 170D is used to connect a wired earphone and can be a USB interface, or a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0060] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0061] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0062] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0063] The button 190 includes a power button, a volume button, etc. The button 190 may be a mechanical button. It may also be a touch button. The electronic device 100 may receive a button input and generate a key signal input related to the user settings and function control of the electronic device 100. The motor 191 may generate a vibration prompt. The SIM card interface 195 is used to connect a SIM card. The SIM card may be connected to or disconnected from the electronic device 100 by inserting the SIM card interface 195 or pulling the SIM card interface 195 out.

[0064] Figure 1 The illustrated structure does not constitute a specific limitation on the electronic device 100. The electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0065] A game application may be installed on the electronic device 100. In order to reduce the load of the GPU in rendering the game screen, an embodiment of the present application provides a rendering method for reducing the load. The method will be described in detail below.

[0066] Identify occlusion areas at all levels

[0067] like Figure 2 As shown, when the user holds (such as holding the screen horizontally) the electronic device 100 to play a game, some areas of the game screen displayed on the screen will be blocked by the user's thumb or other hand parts. Generally, the lower left corner area of ​​the screen is designed as a roulette area, and the lower right corner area of ​​the screen is designed as a skill area. The user needs to manipulate the roulette area and the skill area, which will block the screen, resulting in this part of the game screen being invisible or poor visibility. For example, the fingertip of the thumb touches and presses on the touch screen, causing the area pressed by the fingertip to be invisible to the user, while the thumb part outside the fingertip is suspended in the air on the touch screen at a close distance, resulting in poor visibility of the game screen area below it. In addition, the base of the thumb and the palm part connected to it will usually contact or block the lower left corner area and the lower right corner area of ​​the screen, resulting in the lower left corner area and the lower right corner area of ​​the game screen being invisible to the user or poor visibility.

[0068] The load reduction rendering method provided in the embodiment of the present application will accurately identify these blocked areas in different levels, laying the foundation for subsequent differentiated rendering. The blocked areas at each level may include: finger touch area, finger blocked area, and palm blocked area. The method for accurately identifying the blocked areas at each level is described as follows:

[0069] Finger touch area recognition

[0070] In this article, the finger touch area is the screen area directly touched by the finger, and the game screen displayed in it is invisible to the user and has the highest degree of obstruction. Here, the finger includes but is not limited to a human finger, and may also include a machine finger that simulates a human finger, a simulated finger, etc.

[0071] The identification of the finger touch area can be realized based on the touch event reported by the screen (specifically the touch screen). When a finger is pressed, the screen will collect touch information, such as the position, shape, size and other information of the touch area, which can be carried in the touch event. Based on the touch information, the electronic device 100 can identify the finger touch area.

[0072] Specifically, Figure 3 As shown, the finger touch area can be simulated as an ellipse, and the coordinates of the center (O') of the ellipse, the major and minor axes, and the direction of the ellipse are used as touch information. The direction of the ellipse can refer to the extension direction of the major axis, which can be specifically represented by the angle between the major axis and a side of the screen (such as side XY). The major axis of the ellipse has two vertices, one near the center of the screen (such as C L , C R ) and a vertex far from the center of the screen (such as F L 、F R ). Of course, when implementing the embodiments of the present application, people can also regard the extension direction of the minor axis as the direction of the ellipse. However, knowing only the center coordinates and the major and minor axes of the ellipse is not enough to uniquely determine the finger touch area, so it is necessary to introduce the ellipse direction.

[0073] Not limited to Figure 3 As shown, the finger touch area 211A of the left finger and the finger touch area 211B of the right finger do not necessarily appear together.

[0074] Finger occlusion area recognition

[0075] When the finger pad, finger tip, etc. directly touch the screen and the finger touch area is not visible to the user, other parts of the finger will also be suspended on the screen at a close distance, resulting in poor visibility of the screen area below it. This screen area may be referred to as the finger occlusion area in this article, which does not include the aforementioned finger touch area to distinguish it from the finger touch area.

[0076] Based on the finger touch area, at least one elliptical area can be expanded outward to construct the finger occlusion area. The area left after the largest elliptical area nested in the expanded elliptical area is the finger occlusion area. The expanded elliptical area shares the elliptical direction and the long axis vertex close to the center of the screen with the finger touch area. This also means that the center of the expanded elliptical area is on the same line as the center of the finger touch area.

[0077] Specifically, Figure 4 As shown, based on the finger touch area 211A of the left finger, elliptical areas S1 and S2 can be expanded to form the left finger occlusion area. "Based on the finger touch area 211A" may mean that the expanded elliptical areas S1 and S2 share the same elliptical direction and the long axis vertex C with the finger touch area 211A. L . Major axis vertex C L It is the long axis vertex close to the center of the screen, which reflects the projection position of the left fingertips on the screen.

[0078] Specifically, Figure 4 As shown, based on the finger touch area 211B of the right finger, elliptical areas S1 and S2 can be expanded to form the right finger occlusion area. The expanded elliptical areas S1 and S2 share the same elliptical direction and the long axis vertex C with the finger touch area 211B. R . Major axis vertex C R It is the long axis vertex close to the center of the screen, which reflects the projection position of the right fingertips on the screen.

[0079] Due to the gradually raised three-dimensional structure of the fingertips, the vertical distances from each point on the fingertips to the screen are not the same. The closer the point or area is to the screen, the more serious the screen occlusion is. For this reason, the multiple elliptical areas that are expanded and participate in the left or right finger occlusion area can be nested layer by layer, such as S1 nested in S2, and they gradually become larger to construct multiple levels of finger occlusion areas with different degrees of occlusion. Among them, the area remaining after an elliptical area is subtracted from the largest elliptical area nested inside it forms a first-level finger occlusion area.

[0080] For example, the area remaining after removing the largest elliptical area nested within the expanded ellipse S1 (i.e., the finger touch area 211A (211B)) forms a first-level occlusion area; the area remaining after removing the largest elliptical area nested within the expanded ellipse S2 (i.e., the elliptical area S1) forms another level of occlusion area.

[0081] The visibility of the finger occluded area is worse as it moves inwards towards the touch center. The touch center is the center of the finger touch area. For example, as the finger moves inwards towards the touch area 212A, Figure 4The visibility of the finger-blocked area 212A shown in FIG. 2 is inferior to the visibility of the finger-blocked area 213A. For another example, because the touch area 212B is closer to the inside, Figure 4 The visibility of the finger-occluded region 212B shown in FIG. 2 is inferior to the visibility of the finger-occluded region 213B.

[0082] In particular, the finger touch area 211A (211B) can be regarded as a first-level occlusion area with the most serious occlusion degree.

[0083] based on Figure 4 The invisibility of areas 211A (211B), 212A (212B), and 213A (213B) shown in the figure gradually decreases, and they can be marked as first-level occlusion areas, second-level occlusion areas, and third-level occlusion areas, respectively, and differentiated rendering can be performed on each level of occlusion areas later.

[0084] The size of the outermost elliptical area expanded for constructing the finger occlusion area can be constrained by the finger width W and the screen boundary. Specifically, the sum of the two short axes of the outermost elliptical area can be equal to or approximate to the finger width W, and the elliptical area does not exceed the screen boundary. Here, approximate can mean less than the finger width W, but the difference is small, for example, the difference is within 2 mm. In particular, when only one elliptical area is expanded to construct the finger occlusion area, this elliptical area is the outermost elliptical area, and the sum of its two short axes can be equal to or approximate to the finger width W. The finger width W can be the statistical average of the width of a human thumb. The finger width W can also be estimated based on the width of the finger touch area (such as the sum of the two short axes of area 211A), and the two widths are proportional. Because, generally, the wider the touch area formed by a finger contacting the screen, to some extent, it can reflect that the finger is wider. In order to improve the estimation accuracy of the finger width W, parameters such as user gender, age, weight, etc. can also be further considered, and these parameters can be extracted from the personal information of the logged-in user of the electronic device 100. The embodiment of the present application does not limit the method of obtaining these parameters and how to use these parameters to help estimate the finger width.

[0085] Palm occlusion area recognition

[0086] When a finger directly touches the screen and the finger touch area is not visible to the user, in addition to the other parts of the finger being suspended in the air at a close distance on the touch screen, resulting in poor visibility of the screen area below it, the base of the thumb and the palm part connected to it may also usually block the lower left and lower right corners of the screen, which may cause poor visibility of the lower left and lower right corners of the game screen.

[0087] Herein, the screen area blocked by the base of the thumb and the connected palm part can be referred to as the palm blocking area. Here, the "palm" in the "palm blocking area" is not limited to the palm strictly defined in the field of human anatomy, and this article only uses this name to distinguish it from the aforementioned finger blocking area.

[0088] Based on the finger touch area, a rectangular area R or a triangular area T can be expanded to construct a palm blocking area. The palm blocking area can be an area in the rectangular area R (or triangular area T) that is not covered by the finger touch area and the finger blocking area.

[0089] Specifically, Figure 5 As shown, based on the finger touch area 211A of the left finger, a rectangular area R1 or a triangular area T1 can be expanded. "Based on the finger touch area 211A" can mean that the diagonal line A of the rectangular area R1 is from the center O of the area 211A to the center O of the area 211A. L The line segment between the vertex to the lower left corner of the screen, or the hypotenuse of the triangular area T1 is composed of this line segment.

[0090] Specifically, Figure 5 As shown, based on the finger touch area 211B of the right finger, a rectangular area R2 or a triangular area T2 can be expanded. "Based on the finger touch area 211B" can mean that the diagonal line B of the rectangular area R2 is from the center O of the area 211B. R The line segment between the vertex in the lower right corner of the screen, or the hypotenuse of the triangular area T2 is composed of this line segment.

[0091] The rectangular regions R1 and R2 may not be strict rectangles in the mathematical sense, but may be approximate rectangles, and the sides a1 and b1, and the sides a2 and b2 of the rectangles may be modeled or simulated to be curved in consideration of the natural curvature of the base of the human thumb and the part connected to the palm. Similarly, the triangular regions T1 and T2 may not be strict triangles in the mathematical sense, but may be approximate triangles.

[0092] The palm blocking area may be an area in the rectangular area R (or the triangular area T) that is not covered by the finger touch area and the finger blocking area.

[0093] like Figure 6 As shown, the palm occlusion area 215A of the left palm is an area in the rectangular area R1 (or triangular area T1) that is not covered by the finger touch area 211A of the left finger, the finger occlusion areas 212A and 213A of the left finger. Similarly, the palm occlusion area 215B of the right palm is an area in the rectangular area R2 (or triangular area T2) that is not covered by the finger touch area 211B of the right finger, the finger occlusion areas 212B and 213B of the right finger.

[0094] It can be seen that the center O L The closer to the lower left corner of the screen, the smaller the left palm occlusion area 215A; R The closer to the lower right corner of the screen, the smaller the right palm occlusion area 215B is.

[0095] When the distance from the center of the touch area to the left or right bottom corner of the screen is small enough, the left or right palm occlusion area can be ignored, which means that the left or right finger has not fully extended to the middle of the screen, resulting in the left or right palm not blocking the screen. L When the distance to the lower left corner of the screen is less than a certain distance, the left palm blocking area 215A may be ignored. R When the distance to the lower right corner of the screen is less than a specific distance, the right palm occlusion area 215B may be ignored. The specific distance may be a statistical average of the projection length of a human thumb on the screen when the thumb is naturally bent or straightened.

[0096] The degree of occlusion of the palm occlusion area is lower than that of the finger touch area, and different areas can be subsequently subjected to differential rendering. As for which of the palm occlusion area and the finger occlusion area is higher or lower in terms of occlusion degree, the present application embodiment does not limit it and can be determined according to actual application requirements.

[0097] Differentiated rendering

[0098] After the above-mentioned occlusion areas are graded and identified, the load-reducing rendering method provided in the embodiment of the present application can be further adapted to differentiated rendering reduction methods, thereby reducing the GPU rendering load without affecting the user's gaming experience.

[0099] Means of reducing rendering may include but are not limited to: reducing resolution, reducing image texture, feature point cropping, rendering culling, etc. Among them, reducing resolution will reduce image clarity, reducing image texture will reduce image details, feature point cropping means reducing image feature points, and rendering culling means not rendering the culled area. In addition, rendering reduction means may also include: implementing variable resolution rendering (variable rate shading, VRS) for areas with different degrees of occlusion. VRS can shade in groups of 2 or 4 pixels during rendering, and reuse the shading results for surrounding pixels, reducing the computing load of the GPU by selectively rendering and shading the picture.

[0100] In an embodiment of the present application, differentiated down-rendering may refer to rendering with a lower load for the screen of the occluded area that is closer to the touch center inward, and the touch center is the center of the finger touch area. In this way, a greater degree of down-rendering can be implemented for the occluded area with a higher degree of occlusion. Among them, rendering culling can be considered as the maximum degree of down-rendering, resulting in no content display, which is suitable for the finger touch area. Specifically, the screen of the finger touch area can be rendered culled, and the screen of the finger occluded area can be processed by one or more of the following: reducing image texture, reducing image resolution, reducing feature points, and reducing shading rate.

[0101] For example, Figure 7 As shown, the occlusion degree of the finger touch area 211A, the finger occlusion area 212A, and the finger occlusion area 213A is gradually reduced. It is also assumed that the occlusion degree of the palm occlusion area 215A is lower than the occlusion degree of the finger occlusion area 213A. Then, Figure 7 As shown, the finger touch area 211A, the finger occlusion area 212A, the finger occlusion area 213A, and the palm occlusion area 215A can be subjected to rendering reduction measures of gradually reduced degrees: rendering culling is performed on the finger touch area 211A; VRS is performed on the finger occlusion area 212A, the finger occlusion area 213A, and the palm occlusion area 215A, and 4×4 shading rate, 4×2 shading rate, and 2×2 shading rate are adopted respectively.

[0102] This differentiated down-rendering can reduce the GPU rendering load while more finely presenting different rendering effects for different areas with different degrees of occlusion without affecting the user's gaming experience.

[0103] The load-reduced rendering method provided in the embodiment of the present application can realize the development of game applications without perception, and can be widely applied to various game applications without requiring the game application developer to change functions. This is due to the software system architecture of the electronic device 100 provided in the embodiment of the present application, which designs the path between touch information (carried in input events) and the image processing module.

[0104] Figure 8 The software system architecture of the electronic device 100 is exemplarily shown.

[0105] like Figure 8 As shown, the software system architecture of the electronic device 100 may include an application (APP) layer, an application framework (FWK) layer, and a kernel (kernel) layer. The following expands each part:

[0106] The application layer may include one or more applications, such as a game application, which may provide game screen content.

[0107] The application framework layer may include: input management service (IMS), window management service (WMS), UI framework layer, and image processing module.

[0108] IMS may include: Event Hub, input reader, and input distributor, which cooperate with each other to pass and distribute input events reported by the touch screen to the appropriate window, such as window1 of the game application. Among them, Event Hub can monitor device nodes (such as / dev / input / eventX) through iNotify and Epoll mechanisms, and read the original input events of the device nodes; InputReader will continuously read the original input events in EventHub in a loop, and process these original input events and hand them over to InputDispatcher; InputDispatcher saves the information of all windows in WMS (WMS will update the window information to InputDispatcher in real time), so that InputDispatcher can dispatch input events to the appropriate window.

[0109] WMS can be used to manage all windows in the system, such as window1. Window1 can be a window of a game application, which can be established (instantiated) when the game application is started. Along with the instantiation of window1, a ViewRootImpl object can be instantiated in the UI framework layer, and the ViewRootImpl object can be used to listen to input events dispatched to window1 by InputDispatcher. The embodiment of the present application improves ViewRootImpl. After the improved ViewRootImpl object listens to the input event dispatched to window1 by InputDispatcher, it can also pass the touch information carried in the input event to the image processing module. In addition, since the input event is listened to by the ViewRootImpl object, the game application also receives the input event and understands whether there is a user's finger touching the screen.

[0110] After receiving the touch information transmitted by the ViewRootImpl object, the image processing module can be used to intercept rendering instructions, identify occlusion areas at all levels, and reduce the rendering load. For the specific implementation of identifying occlusion areas at all levels and reducing the rendering load, please refer to the previous and following texts, which will not be repeated here.

[0111] In addition to IMS, WMS and image processing modules, the application framework layer can also include a layer synthesizer (SurfaceFlinger). SurfaceFlinger can be used to perform layer synthesis processing on the picture frames rendered by the GPU, generate an interface including the picture frames, and send the interface including the picture frames to the display screen for display.

[0112] The kernel layer may include: a touch screen driver, a GPU driver, etc. Among them, the touch screen driver may be used to control the operation of the touch screen and receive touch signals reported by the touch screen, and the GPU driver may be used to control the GPU to perform rendering operations. When a user's finger is detected touching the touch screen, the touch screen may simulate the finger pressing area into an elliptical area, and report ellipse information to the touch screen driver, such as the center (O') coordinates of the ellipse, the major and minor axes, the direction of the ellipse, and other information. The ellipse information is carried as touch information in the input event reported to the touch screen driver.

[0113] In addition to the above layers, the system architecture of the electronic device 100 may also include a local service layer. The local service layer may include one or more graphics libraries, such as an open graphics library (OpenGL) and an open graphics library for embedded systems (OpenGL ES). The graphics drawing interface included in the graphics library may be called to configure the GPU to render a picture frame.

[0114] based on Figure 8 The software system architecture is shown as an example. Fig. 9 The figure shows the cooperation between the various software modules, especially the image processing module intercepting rendering instructions, identifying occlusion areas at all levels, and reducing the rendering load. Among them, the game application may include a main thread (Main Thread) and a rendering thread (Render Thread). The image processing module may include a rendering instruction interception module, an occlusion area identification module at all levels, and a rendering module for reducing the load.

[0115] The interaction process of each module can be as follows:

[0116] 1. The main thread of the game application can synchronize the game screen content to the rendering thread.

[0117] 2. After the rendering thread obtains the game screen content, it can call the first graphics drawing interface in the graphics library (such as OpenGL, OpenGL ES, etc.) to issue rendering instructions.

[0118] At the same time, the game screen content to be rendered of the game application can be cached, for example, temporarily stored in cache 1, so as to be read when the GPU is rendering. Cache 1 can be a partial cache area on the memory 120 in the electronic device 100.

[0119] 3. The first graphics drawing interface in the graphics library (such as OpenGL) is called to forward the rendering instruction to the rendering instruction interception module.

[0120] Calling the first graphics drawing interface in the graphics library does not directly pass the rendering instruction from the rendering thread to the GPU, but forwards the rendering instruction to the rendering instruction interception module in the image processing module first, which is the rendering instruction interception mentioned in this article.

[0121] Specifically, the rendering instruction interception can occur during the period when the image processing module receives the touch information transmitted by the ViewRootImpl object, during which the user's finger touches the screen. That is, during the period when the user's finger touches the screen, the game application can call the first graphics drawing interface in the graphics library. In addition, during the period when the user's finger does not touch the screen, the game application can call the second graphics drawing interface in the graphics library to pass the rendering instruction directly to the GPU, which is also the ordinary rendering mentioned in this article, relative to this rendering method that reduces the rendering load.

[0122] In the graphics library, the first graphics drawing interface is a graphics drawing interface introduced in the embodiment of the present application to implement rendering instruction interception, and the second graphics drawing interface is a traditional graphics drawing interface provided by the graphics library.

[0123] 4. After receiving the rendering instruction, the rendering instruction interception module can further pass the rendering instruction to the occlusion area recognition modules at all levels.

[0124] The occlusion area recognition module at each level can identify the occlusion areas at each level based on touch information, such as the coordinates of the center (O') of the ellipse, the major and minor axes, the direction of the ellipse, etc.: finger touch area, finger occlusion area, palm occlusion area, to lay the foundation for subsequent differentiated rendering. For how to identify the occlusion areas at each level, please refer to the relevant content in the previous or later text, which will not be repeated here.

[0125] The touch information can be passed to the image processing module by the ViewRootImpl object. For details, please refer to Figure 8 The relevant contents in the embodiments will not be repeated here.

[0126] 5. After identifying the occlusion areas at each level, the occlusion area identification modules at each level may inform the load-reducing rendering module of the identification results (ie, the occlusion areas at each level).

[0127] Here, the recognition result may include indication information of each level of occlusion area (such as finger touch area, finger occlusion area, palm occlusion area), such as position, size, and shape, which indicates which pixels are covered by each level of occlusion area on the screen.

[0128] 6. After knowing the occlusion areas at each level, the load reduction rendering module can adapt differentiated rendering reduction methods to the occlusion areas at each level, obtain differentiated rendering reduction solutions, and send rendering instructions to the GPU. The rendering instructions can carry differentiated rendering reduction solutions.

[0129] The differentiated down-rendering scheme can be used to implement a higher degree of down-rendering for occluded areas with a higher degree of occlusion, and a lower degree of down-rendering for occluded areas with a lower degree of occlusion. Figure 4 The region 211A in the image is subjected to rendering culling and down-rendering with variable color rates of 4×4, 4×2, and 2×2 is respectively applied to the regions 212A, 213A, and 215A. For how to implement differentiated down-rendering, please refer to the relevant contents in the previous or later texts, which will not be described here.

[0130] 7. After receiving the rendering instruction sent by the load-reduced rendering module, the GPU can read the game screen content to be rendered from cache 1, render the game screen content according to the differentiated load-reduced rendering scheme, and finally store the rendered screen frame in cache 2.

[0131] Cache 2 may be a cache area specified by a rendering instruction, and it may be a partial cache area on the memory 120 in the electronic device 100 .

[0132] 8. The layer synthesizer (SurfaceFlinger) takes the rendered frame from cache 2 for layer synthesis. Then, SurfaceFlinger sends the frame after layer synthesis to the frame buffer for storage to complete the display.

[0133] The frame buffer is a storage space that can be located in the video memory or the internal memory, and is used to store the rendering data processed by the graphics chip or to be extracted. The content of the frame buffer corresponds to the interface display on the display screen 194, and it can be simply understood as the cache corresponding to the content displayed on the display screen 194. Modifying the content in the frame buffer is to modify the content on the display screen 194.

[0134] Specifically, after receiving the vertical synchronization (vsync) signal each time, SurfaceFlinger takes out the picture frame from cache 2 for layer synthesis and sends it to the display screen for refresh display. The vsync signal is a synchronization signal generated by the hardware module corresponding to the display screen with a period consistent with the screen refresh rate. The screen refresh rate indicates the number of times the display screen on the electronic device refreshes the display frame within 1 second. Generally, the refresh rate of the display screen of the electronic device is related to the fixed parameters of the display screen and is a fixed value. For example, the refresh rate can be 60 Hz.

[0135] The software system architecture provided in the embodiment of the present application designs a path from touch information (carried in input events) to the image processing module, and can also achieve low latency, so that there is no difference in user experience between load-reduced rendering and ordinary rendering.

[0136] Generally, the game uses triple buffering, and it takes 3 frames from rendering to display, of which 2 frames are cached. For a 60-frame game (single-frame interval is about 16.7 milliseconds), triple buffering will produce a delay of about 50 milliseconds. Taking the delay between the user's touch operation and the screen display of the game screen as an example, which does not exceed 80 milliseconds, this requires that the delay of the touch information being transmitted to the image processing module and the image processing module executing the rendering instruction interception, the identification of the occlusion areas at all levels, and the processing delay of the output differentiated load-reducing rendering scheme cannot exceed 30 milliseconds in total. The fast path between the touch information and the image processing module provided in the embodiment of the present application can meet this requirement. The embodiment of the present application can achieve the time for the touch information to be transmitted to the image processing module to be less than 25 milliseconds, and the processing time of the image processing module does not exceed 5 milliseconds. However, for low-frame rate games, taking a 30-frame game as an example, since the single-frame interval has reached 33 milliseconds, the rendering under triple buffering takes about 100 milliseconds to display, and the delay is already very significant. Considering the user experience, the electronic device can choose not to perform load-reducing rendering, but only perform ordinary rendering.

[0137] Fig.10 The overall process of the load-reducing rendering method provided by the embodiment of the present application is shown below.

[0138] S11. The game starts.

[0139] The electronic device can start the game application due to user operation (such as clicking the desktop icon of the game application), and needs to render the game screen provided by the game application and display the game screen provided by the game application on the display screen.

[0140] As mentioned above, when a user holds (such as holding it horizontally) an electronic device to play a game, certain areas of the game screen displayed on the screen will be blocked by the user's thumb or other hand parts. Generally, the lower left corner area of ​​the screen is designed as a roulette area, and the lower right corner area of ​​the screen is designed as a skill area. The user needs to manipulate the roulette area and the skill area, which will block the screen, causing this part of the game screen to be invisible or have poor visibility. For example, the fingertip of the thumb touches and presses on the touch screen, causing the area pressed by the fingertip to be invisible to the user, while the part of the thumb other than the fingertip is suspended in the air at a close distance on the touch screen, causing the visibility of the game screen area below it to be poor. In addition, the base of the thumb and the palm part connected to it usually contact or block the lower left corner area and the lower right corner area of ​​the screen, causing the lower left corner area and the lower right corner area of ​​the game screen to be invisible to the user or have poor visibility.

[0141] The load-reducing rendering method provided in the embodiment of the present application will accurately identify these blocked areas in a graded manner, laying the foundation for subsequent differentiated rendering reduction. Not limited to game applications, the embodiment of the present application can also be applied to other applications that provide a graphical user interface and receive user touch input to reduce the rendering load of the graphical user interface when the user's finger touches the screen.

[0142] S12. The electronic device can determine whether a touch operation is detected on the touch screen. If a touch operation is detected, S13-S15 can be executed, otherwise S16 can be executed. That is, when the user's finger touches the screen, the electronic device performs load reduction rendering; when the user's finger does not touch the screen, the electronic device performs normal rendering.

[0143] The game screen displayed on the screen may also display: a direction wheel and a skill key, wherein the direction wheel can be used to control the movement of characters or other subjects in the game, and the skill key can be used to implement control such as launching game skills. The direction wheel and the skill key can be respectively distributed in the lower left corner and the lower right corner of the screen, or respectively distributed in the lower right corner and the lower left corner of the screen.

[0144] The touch operation may be an operation on a steering wheel or a skill key. That is, the electronic device may trigger execution of S13-S15 when a touch operation on a steering wheel or a skill key is detected, or a touch operation is detected at the lower right corner or the lower left corner of the screen.

[0145] In addition, before determining whether to execute S13-S15 or S16, the electronic device may also determine whether the game application is a low frame rate game application, such as a 30-frame game. If it is not a low frame rate game application, determine to execute S13-S15, otherwise execute S16.

[0146] In this paper, the delay requirement between the user's touch operation and the screen display of the game screen (maximum delay T max) to define whether a game application is a low frame rate game application. Specifically, if the latency of a game application executing triple buffering reaches or exceeds T max , it can be considered a low frame rate gaming application. Or, more strictly, if the latency of a gaming application executing triple buffering exceeds T max -T reserve , it can be considered as a low frame rate game application. reserve It indicates the delay reserved for the touch information to be transmitted to the image processing module and the processing delay of the image processing module, for example, 30 milliseconds, which is affected by the processing performance of the relevant hardware and can be adjusted according to the actual implementation scenario. The delay of the game application to perform triple buffering is determined by the frame rate of the game application, R, which is equal to 3*(1000 / R) milliseconds.

[0147] S13. The electronic device can obtain touch information.

[0148] The touch information may include information such as the position, shape, and size of the finger touch area, which may be carried in the touch event. Based on the touch information, the electronic device may identify the finger touch area. Specifically, Figure 3 As shown, the finger touch area can be simulated as an ellipse, and the coordinates of the center (O'), the major and minor axes, and the direction of the ellipse are used as touch information. The direction of the ellipse can refer to the extension direction of the major axis, which can be specifically represented by the angle between the major axis and a side of the screen (such as side XY).

[0149] S14. The electronic device can identify different levels of occlusion areas based on the touch information.

[0150] The occlusion areas at each level may include: a finger touch area, a finger occlusion area, and a palm occlusion area.

[0151] The identification of the finger touch area can be realized based on the touch event reported by the screen (specifically the touch screen). The finger touch area is the screen area directly touched by the finger, and the game screen displayed therein is not visible to the user, has the highest degree of occlusion, and has the worst visibility. The finger touch area can be simulated as an ellipse.

[0152] There can be N (N is a positive integer) finger occlusion areas, which can be constructed by N elliptical areas expanded outward based on the finger touch area. The expanded elliptical area has the same elliptical direction as the finger touch area, and the long axis vertices of the two close to the center of the screen coincide. Among them, a finger occlusion area is the area left after the largest elliptical area nested in the expanded elliptical area is removed.

[0153] When N=1, the largest elliptical area within the expanded elliptical area is the finger touch area. Figure 4The finger blocking area 212A in FIG. 1 is the area left after the elliptical area S1 is expanded and the finger touch area (ie, the largest elliptical area nested in the elliptical area S1) is removed.

[0154] When N ≥ 2, the N elliptical areas that are expanded and participate in forming the left or right finger occlusion area can be nested layer by layer, and they gradually become larger to construct multiple levels of finger occlusion areas with different occlusion degrees. Among them, the area left after removing the largest elliptical area nested inside an elliptical area forms a first-level finger occlusion area. For example, Figure 4 The finger occlusion area 212A is the area left after the elliptical area S1 is expanded and the finger touch area is removed, and the finger occlusion area 213A is the area left after the elliptical area S2 is expanded and the elliptical area S1 (ie, the largest elliptical area nested in the elliptical area S2) is removed.

[0155] The visibility of the finger-occluded area becomes worse as it moves inwards toward the touch center. The touch center is the center of the finger touch area.

[0156] Specifically, Figure 4 As shown in FIG. 2 , the elliptical areas S1 and S2 expanded based on the finger touch area 211A of the left finger, after removing the largest ellipse nested in each of them, respectively leave the areas 212A and 213A, which are the two levels of left finger occlusion areas with different occlusion degrees. Among them, the expanded elliptical areas S1 and S2 share the same elliptical direction and the long axis vertex C with the touch area 211A. L .

[0157] Specifically, Figure 4 As shown in FIG. 2 , the elliptical areas S1 and S2 expanded based on the finger touch area 211B of the right finger, after removing the largest ellipse nested in each of them, the remaining areas 212B and 213B are two levels of right finger occlusion areas with different occlusion degrees. Among them, the expanded elliptical areas S1 and S2 share the same elliptical direction and the long axis vertex C with the touch area 211B. R .

[0158] The size of the outermost elliptical area expanded to construct the finger occlusion area can be constrained by the finger width W and the screen boundary. Specifically, the sum of the two short axes of the outermost elliptical area can be equal to or approximate to the finger width W, and the elliptical area does not exceed the screen boundary. In particular, when only one elliptical area is expanded to construct the finger occlusion area, this elliptical area is the outermost elliptical area, and the sum of its two short axes can be equal to or approximate to the finger width W.

[0159] The palm occlusion area may be constructed by expanding the rectangular area R or the triangular area T based on the finger touch area. The palm occlusion area may be an area in the rectangular area R (or the triangular area T) that is not covered by the finger touch area or the finger occlusion area.

[0160] Specifically, Figure 5 As shown in FIG. 1 , in the rectangular area R1 or triangular area T1 expanded based on the finger touch area 211A of the left finger, the area not covered by the finger touch area 211A of the left finger and the finger blocking areas 212A and 213A of the left finger is the palm blocking area 215A of the left palm. “Based on the finger touch area 211A” may mean that the diagonal A of the rectangular area R1 is from the center O of the area 211A to the left palm. L The line segment between the vertex to the lower left corner of the screen, or the hypotenuse of the triangular area T1 is composed of this line segment.

[0161] Specifically, Figure 5 As shown in FIG. 1 , the area of ​​the rectangular area R2 or triangular area T2 expanded based on the finger touch area 211B of the right finger that is not covered by the finger touch area 211A of the left finger and the finger blocking areas 212A and 213A of the left finger is the palm blocking area 215B of the right palm. “Based on the finger touch area 211B” may mean that the diagonal line B of the rectangular area R2 is from the center O of the area 211B to the left. R The line segment between the vertex in the lower right corner of the screen, or the hypotenuse of the triangular area T2 is composed of this line segment.

[0162] When the distance from the center of the touch area to the left or right lower corner of the screen is small enough to ignore the area blocked by the left or right palm, the left or right finger does not fully extend to the middle of the screen, resulting in the left or right palm not blocking the screen.

[0163] S15. The electronic device can render the images of the occlusion areas at all levels with reduced load.

[0164] Specifically, the electronic device may render the screen in the blocked area closer to the touch center with a reduced load. The means of rendering with reduced load may include but are not limited to one or more of the following: rendering culling, reducing image texture, reducing image resolution, reducing feature points, and reducing shading rate.

[0165] For example, the electronic device may implement a greater degree of resolution reduction, a greater degree of image texture reduction, and cropping of more feature points for the occluded area that is closer to the touch center.

[0166] Among them, rendering culling can be considered as the maximum reduction in rendering, resulting in no content display. Specifically, the electronic device can implement rendering culling, a rendering method that reduces the load, on the finger touch area, and perform one or more of the following processing on the images in the finger occlusion area and the palm occlusion area: reduce the resolution, reduce the image texture, and crop the feature points. Among them, the closer the finger occlusion area is to the touch center, the greater the reduction in resolution, the reduction in image texture, and the more feature points can be cropped.

[0167] In addition, the rendering means of reducing the load may also include: implementing VRS for areas with different degrees of occlusion. For example, different shading rates may be used for finger occlusion areas with different degrees of occlusion. The images of finger occlusion areas closer to the touch center use a lower shading rate, and the images of finger occlusion areas closer to the periphery use a higher shading rate. If it is believed that the degree of occlusion of the palm occlusion area is higher than that of the finger occlusion area, the shading rate used in the palm occlusion area may be lower than that used in the finger occlusion area.

[0168] In this way, users can observe that after the rendering with reduced load is implemented, the images of the occluded areas at all levels are more blurred than the images of other screen areas where the rendering with reduced load is not implemented. Moreover, the images of the areas closer to the touch center are more blurred. Because these occluded areas are rendered with reduced load, their images are inferior to other screen areas where the rendering with reduced load is not implemented in one or more aspects such as image texture, resolution, number of feature points, and shading rate.

[0169] S16. The electronic device may perform normal rendering.

[0170] For ordinary rendering, please refer to the relevant description in the previous embodiment, which will not be repeated here.

[0171] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0172] The embodiments of the present application also provide a computer program product. When the computer program product is executed on an electronic device, the electronic device can implement the steps in the above-mentioned various method embodiments.

[0173] The embodiment of the present application also provides a chip system, the chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the steps of any method embodiment of the present application. The chip system can be a single chip or a chip module composed of multiple chips.

[0174] The term "user interface (UI), or interface for short) in the specification and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. A control, also called a widget, is a basic element of a user interface. Typical controls include a toolbar, menu bar, text box, button, scrollbar, picture and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as . A node corresponds to a control or attribute in the interface. After parsing and rendering, the node is presented as user-visible content. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, HTML is defined by 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0175] The most common form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed in a graphical manner. It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

[0176] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0177] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0178] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A rendering method for reducing load, characterized in that: include: The electronic device displays a picture provided by the first application on the screen; The electronic device detects a touch operation of a user's finger on the screen and obtains touch information of the touch operation; The electronic device determines a multi-level shielding area based on the touch information, wherein the multi-level shielding area includes: a finger touch area and a finger shielding area, wherein the finger shielding area covers the periphery of the finger touch area; The electronic device performs reduced-load rendering on the image of the multi-level occlusion area, wherein the image of the occlusion area that is closer to the touch center is rendered with a reduced load, and the touch center is the center of the finger touch area. The reduced-load rendering includes one or more of the following: rendering culling, reducing image texture, reducing image resolution, reducing feature points, and reducing shading rate.

2. The method according to claim 1, characterized in that The rendering of the image in the blocked area closer to the touch center adopts a lower load, including: rendering and culling the image in the finger touch area, and performing one or more of the following processing on the image in the finger blocked area: reducing image texture, reducing image resolution, reducing feature points, and reducing shading rate.

3. The method according to claim 1 or 2, characterized in that The first application includes a game application, and the screen also displays: a steering wheel and a skill key, the touch operation acts on the steering wheel or the skill key, and the steering wheel and skill key are respectively distributed in the lower left corner, lower right corner of the screen or respectively distributed in the lower right corner and lower left corner of the screen.

4. The method according to any one of claims 1 to 3, characterized in that The finger touch area is an elliptical area, and the touch information includes the center coordinates, the major and minor axes, and the direction of the elliptical area, wherein the direction of the elliptical area is the extension direction of the major axis.

5. The method according to claim 4, characterized in that The finger occlusion areas include N areas, where N is a positive integer, and the N finger occlusion areas are constructed based on N elliptical areas expanded from the finger touch area, wherein a finger occlusion area is an area left after the largest elliptical area nested inside is subtracted from an expanded elliptical area; the expanded elliptical area has the same elliptical direction as the finger touch area, and a long axis vertex close to the center of the screen.

6. The method according to claim 5, characterized in that When N=1, the largest elliptical area nested inside the one expanded elliptical area is the finger touch area.

7. The method according to claim 5 or 6, characterized in that When N is greater than or equal to 2, the expanded N elliptical areas are nested layer by layer and gradually become larger.

8. The method according to claim 7, characterized in that The sum of two short axes of the extended outermost elliptical area is equal to the width W of the first finger, or the difference between the sum of two short axes of the extended outermost elliptical area and the width W of the first finger is less than a first value; the first finger includes the thumb.

9. The method according to claim 7 or 8, characterized in that The N finger occlusion areas use different shading rates, wherein the game screen of the finger occlusion area closer to the touch center uses a lower shading rate.

10. The method according to any one of claims 1 to 9, characterized in that The multi-level occlusion area also includes: a left palm occlusion area; The left palm occlusion area is an area in the rectangular area or triangular area in the lower left corner of the screen that is not covered by the finger touch area of ​​the left finger and the finger occlusion area of ​​the left finger; wherein the diagonal line of the rectangular area in the lower left corner is the line segment from the center of the left finger touch area to the lower left corner of the screen, and the hypotenuse of the triangular area is the line segment.

11. The method according to any one of claims 1 to 10, characterized in that The multi-level occlusion area also includes: a right palm occlusion area; The right palm occlusion area is an area in the rectangular area or triangular area in the lower right corner of the screen that is not covered by the right finger touch area of ​​the right finger and the right finger occlusion area of ​​the right finger; wherein the diagonal line of the rectangular area in the lower right corner is the line segment from the center of the right finger touch area to the lower right corner of the screen, and the hypotenuse of the triangular area is the line segment.

12. An electronic device, characterized in that: The method comprises one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, wherein the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the method as described in any one of claims 1 to 11 is executed.

13. A chip system, the chip system is applied to electronic equipment, the chip system comprises one or more processors, characterized in that: The processor is configured to call computer instructions so as to execute the method according to any one of claims 1 to 11.

14. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the method according to any one of claims 1 to 11 is executed.

Images