Frame insertion method, device, electronic device and readable storage medium
By identifying the time point in the rendering of the user interface on an electronic device, the main screen and the user interface are separated and rendered, and only the main screen is inserted, the problem of frame drop and stuttering of the game screen is solved, and the smoothness and user level is improved Experience.
Patent Information
- Application Number
- CN202310921506.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-24
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2043-07-24
AI Technical Summary
The game screen is prone to frame drops and stuttering on low-performance devices, resulting in distortion of the user interface and affecting the user experience.
By identifying whether the target application starts rendering the user interface, and rendering the main screen and the user interface in different frame caches, only the main screen is interpolated to avoid distortion of the user interface and improve the interpolation effect.
It improves the frame count and fluency of the game screen, improves the user experience, reduces power consumption, and enhances the performance of the device.
Smart Images

Figure CN117695625B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminals, and in particular to a frame insertion method, device, electronic device and readable storage medium. Background Art
[0002] With the rapid development of the gaming industry, more and more users have higher requirements for the smoothness and high frame rate of the game screen. Limited by the performance of the device used by the user, the game screen may have problems such as frame drops and freezes. In this case, interpolation technology can effectively improve the frame rate and smoothness of the game screen.
[0003] Usually, interpolation is to interpolate the entire game screen, which will cause the user interface (UI) to appear distorted after interpolation, affecting the user experience. Summary of the invention
[0004] The present application provides a frame insertion method, device, electronic device and readable storage medium, which determines whether the target parameter of the electronic device meets the first preset condition before controlling the electronic device to enter the long standby mode, and if so, determines to control the electronic device to enter the long standby mode. The problem of inaccurate time of entering the long standby mode, resulting in insignificant effect of reducing power consumption or affecting the user's experience, can be improved.
[0005] In order to achieve the above objectives, this application adopts the following technical solutions:
[0006] In a first aspect, a frame insertion method is provided, which is applied to an electronic device, wherein the electronic device runs a target application, and the screen of the target application includes a main screen and a user interface, and the method includes:
[0007] Identify whether the target application starts to render the user interface. When it is detected that the target application starts to render the user interface, draw the subsequent main screen in the first frame buffer and draw the user interface in the second frame buffer; perform interpolation processing on the main screen in the first frame buffer to obtain the interpolated main screen; mix the interpolated main screen with the user interface in the second frame buffer as the screen output of the target application.
[0008] In the embodiments of the present application, the interpolation method can be applied to electronic devices, including mobile phones, tablet computers, handheld game consoles, wearable devices, augmented reality / virtual reality devices, laptops, ultra-mobile personal computers, netbooks, personal digital assistants, etc.
[0009] In the first aspect, by identifying the time point when the user interface is started to be rendered, and rendering the main screen and the user interface in different frame buffers respectively, only the main screen is interpolated and then mixed to obtain the output screen of the target application. Since the main screen and the UI are separated and only the main screen is interpolated, the UI in the output screen of the target application is not distorted, which improves the effect of interpolation and improves the user experience.
[0010] In some possible implementations, detecting that the target application starts rendering a user interface includes:
[0011] According to the identification of the target application, the corresponding user interface rendering characteristics are obtained; when it is detected that the screen rendering characteristics of the target application meet the corresponding user interface rendering characteristics, it is determined that the target application starts to render the user interface, and the screen rendering characteristics include at least one of the type of drawing call DrawCall during rendering, the number of DrawCalls, the number of color channels, and the drawing state of the target frame cache.
[0012] In some possible implementations, the corresponding user interface rendering feature includes: a preset type of DrawCall is detected, and the preset type of DrawCall includes at least one that meets the first preset condition.
[0013] In some possible implementations, the first preset condition includes: detecting a preset first clear instruction, detecting a preset second clear instruction, detecting that blending is turned on, detecting that deep writing is disabled, detecting that the number of DrawCalls between executing the preset first clear instruction and executing the preset second clear instruction is greater than 0, detecting that the count of the current DarwCall is less than a preset count threshold, and detecting that the count of the frame cache of the last execution of the second clear instruction in the previous frame meets the second preset condition.
[0014] In some possible implementations, the second preset condition includes: the frame buffer index of the current frame is equal to the count of the last frame buffer in which the second clear instruction is executed in the previous frame.
[0015] In some possible implementations, the method further includes: when the number of frames satisfying the first preset condition obtained on the preset type of DrawCall is greater than a preset number threshold, determining that the preset type of DrawCall meets the first preset condition.
[0016] In some possible implementations, the corresponding user interface rendering features include: detecting that blending is turned on, detecting that rendering is to a target frame buffer, detecting that the number of DrawCalls corresponding to the target frame buffer is greater than 1, detecting that the drawing state of the target frame buffer is the last drawn frame buffer, and detecting that there is a frame buffer with a color channel in addition to the target frame buffer.
[0017] In some possible implementations, before obtaining the corresponding user interface rendering features based on the identification of the target application, it also includes: analyzing the target application to obtain the user interface rendering features corresponding to the target application; storing the identification of the target application and the user interface rendering features corresponding to the target application in a preset storage space.
[0018] Acquiring corresponding user interface rendering features according to the identifier of the target application includes: matching and acquiring corresponding user interface rendering features of the target application from a preset storage space according to the identifier of the target application.
[0019] In a second aspect, a frame insertion device is provided, which is applied to an electronic device, wherein the electronic device runs a target application, and the screen of the target application includes a main screen and a user interface, and the device includes:
[0020] an identification module, used for identifying whether the target application starts to render the user interface, and when it is detected that the target application starts to render the user interface, drawing the subsequent main screen in the first frame buffer and drawing the user interface in the second frame buffer;
[0021] The interpolation module is used to perform interpolation processing on the main picture in the first frame buffer to obtain the main picture after interpolation;
[0022] The mixing module is used to mix the main picture after the insertion with the user interface in the second frame buffer as the picture output of the target application.
[0023] In some possible implementations, the identification module is specifically used to obtain corresponding user interface rendering features based on the identification of the target application; when it is detected that the screen rendering features of the target application meet the corresponding user interface rendering features, it is determined that the target application starts rendering the user interface, and the screen rendering features include at least one of the type of drawing call DrawCall during rendering, the number of DrawCalls, the number of color channels, and the drawing status of the target frame cache.
[0024] In some possible implementations, the corresponding user interface rendering feature includes: a preset type of DrawCall is detected, and the preset type of DrawCall includes at least one that meets the first preset condition.
[0025] In some possible implementations, the first preset condition includes: detecting a preset first clear instruction, detecting a preset second clear instruction, detecting that blending is turned on, detecting that deep writing is disabled, detecting that the number of DrawCalls between executing the preset first clear instruction and executing the preset second clear instruction is greater than 0, detecting that the count of the current DarwCall is less than a preset count threshold, and detecting that the count of the frame cache of the last execution of the second clear instruction in the previous frame meets the second preset condition.
[0026] In some possible implementations, the second preset condition includes: the frame buffer index of the current frame is equal to the count of the last frame buffer in which the second clear instruction is executed in the previous frame.
[0027] In some possible implementations, the identification module is further configured to determine that a DrawCall of the preset type meets the first preset condition when the number of frames satisfying the first preset condition obtained on the DrawCall of the preset type is greater than a preset number threshold.
[0028] In some possible implementations, the corresponding user interface rendering features include: detecting that blending is turned on, detecting that rendering is to a target frame buffer, detecting that the number of DrawCalls corresponding to the target frame buffer is greater than 1, detecting that the drawing state of the target frame buffer is the last drawn frame buffer, and detecting that there is a frame buffer with a color channel in addition to the target frame buffer.
[0029] Some possible implementations further include an acquisition module for analyzing the target application to acquire the user interface rendering features corresponding to the target application; and storing the target application identifier and the user interface rendering features corresponding to the target application in a preset storage space.
[0030] The acquisition module is further used to match and acquire the user interface rendering features corresponding to the target application from a preset storage space according to the identifier of the target application.
[0031] In a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program and performs the steps of processing in the first aspect or any one of the methods in the first aspect.
[0032] In a fourth aspect, a chip is provided, characterized in that it includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip executes the steps of processing in the first aspect or any one of the methods in the first aspect.
[0033] In a fifth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the steps of processing in the first aspect or any one of the methods in the first aspect.
[0034] In a sixth aspect, a computer program product is provided, the computer program product comprising: a computer program code, when the computer program code is executed by an electronic device, the electronic device executes the steps of processing in the first aspect or any one of the methods in the first aspect.
[0035] Among them, the beneficial effects of the second to sixth aspects can refer to the first aspect and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 This is a schematic diagram of an application scenario of a frame insertion method provided in an embodiment of the present application;
[0037] Figure 2 It is a hardware structure block diagram of an electronic device provided in an embodiment of the present application;
[0038] Figure 3 is a system structure block diagram of an electronic device provided in an embodiment of the present application;
[0039] Figure 4 is a software structure block diagram of the electronic device provided in the embodiment of the present application;
[0040] Figure 5 is a flowchart of a frame insertion method provided in an embodiment of the present application;
[0041] Figure 6 is a structural block diagram of a frame insertion device provided in an embodiment of the present application;
[0042] Figure 7 It is a schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION
[0043] The technical solution in this application will be described below in conjunction with the accompanying drawings.
[0044] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.
[0045] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.
[0046] With the rapid development of the gaming industry, more and more users have higher requirements for the smoothness and high frame rate of the game screen. However, due to the performance of the device used by the user, when running the game application, the game screen may experience frame drops and freezes. In this case, using interpolation technology to interpolate the game screen can effectively improve the frame drops and freezes, increase the number of frames of the game screen, and improve the smoothness of the game.
[0047] The game screen includes the main scene and the user interface (UI). The main scene includes the main screen of the game, such as the movement and interaction effects of characters and objects, while the UI in the game includes the main menu, game settings, character attribute bar, etc.
[0048] In some solutions, when interpolating a game screen, the entire game screen is interpolated, which may cause a distortion in the user interface (UI) after the interpolation, affecting the user experience.
[0049] In view of this, the present application provides an interpolation method, which is applied to an electronic device, wherein the electronic device runs a target application, and the screen of the target application includes a main screen and a user interface. The method includes: identifying whether the target application starts to render the user interface, and when it is detected that the target application starts to render the user interface, drawing the subsequent main screen in a first frame buffer and drawing the user interface in a second frame buffer; interpolating the main screen in the first frame buffer to obtain the interpolated main screen; mixing the interpolated main screen with the user interface in the second frame buffer as the screen output of the target application.
[0050] In this application, by identifying the time point when the user interface is rendered, and rendering the main screen and the user interface in different frame buffers, only the main screen is interpolated and then mixed to obtain the output screen of the target application. Since the main screen and the UI are separated and only the main screen is interpolated, the UI in the output screen of the target application is not distorted, which improves the effect of interpolation and improves the user experience.
[0051] Figure 1 It is a schematic diagram of an application scenario of an interpolation method provided in an embodiment of the present application.
[0052] refer to Figure 1 First, the application scenarios of the embodiments of the present application are briefly described.
[0053] Figure 1 The electronic device 100 is shown in FIG. 1 . When the electronic device 100 receives an operation instructing to start a game application, the game application is started in response to the operation.
[0054] When the electronic device 100 starts the game application and enters the game scene, it detects whether the game application starts rendering the UI. When it detects that the UI starts rendering, the electronic device 100 renders the main interface of the game application and the UI in different frame buffers. Then, the main interface of the game application is interpolated. Finally, the interpolated main interface is mixed with the UI and output.
[0055] As an example, Figure 1 The game application in is a shooting game. Figure 1 , which also shows the main screen 21 and UI 22 of the game application. The main screen 21 includes screens such as game scenes and game characters. The UI 22 includes settings, shooting, backpacks, aiming, character control controls, such as movement controls, standing, lying, and squatting posture controls, etc. The UI 22 may also include a small map, a character name plate, a status bar, etc.
[0056] Figure 2 It is a hardware structure block diagram of an electronic device provided in an embodiment of the present application.
[0057] As an example, the electronic device may include a mobile phone, a tablet computer, a handheld game console, a wearable device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiments of the present application do not impose any limitation on the specific type of the electronic device.
[0058] refer to Figure 2The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0059] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0060] For example, when the electronic device 100 is a mobile phone or a tablet computer, it may include all the components shown in the figure, or may include only some of the components shown in the figure.
[0061] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0062] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.
[0063] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0064] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.
[0065] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0066] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.
[0067] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.
[0068] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
[0069] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0070] The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the electronic device 100 .
[0071] The modulation and demodulation processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing.
[0072] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.
[0073] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).
[0074] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
[0075] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
[0076] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.
[0077] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.
[0078] The camera 193 is used to capture still images or videos. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.
[0079] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0080] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.
[0081] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.
[0082] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.
[0083] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.
[0084] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.
[0085] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.
[0086] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.
[0087] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.
[0088] The pressure sensor 180A is used to sense pressure signals and convert the pressure signals into electrical signals.
[0089] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.
[0090] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.
[0091] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.
[0092] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.
[0093] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.
[0094] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.
[0095] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.
[0096] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.
[0097] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.
[0098] In an embodiment of the present application, the temperature sensor 180J may include multiple ones for detecting the temperatures at different positions of the electronic device 100, such as being set near the processor to obtain the temperature of the processor, being set near the battery to obtain the temperature of the battery, or being set on the inside of the outer casing of the electronic device 100 to obtain the temperature of the outer casing of the electronic device 100.
[0099] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.
[0100] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.
[0101] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.
[0102] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.
[0103] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.
[0104] For the scenario in the above example, the operating system of the electronic device 100 may include but is not limited to Symbian, Android, Windows, MacOS, iOS, Blackberry, HarmonyOS, Linux or Unix operating systems.
[0105] Figure 3 It is a system structure block diagram of the electronic device provided in the embodiment of the present application.
[0106] As an example, when the frame insertion method provided in the present application is executed on the electronic device 100, the operating system of the electronic device 100 may be Android, and its system structure may refer to Figure 3 .
[0107] Among them, the layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer.
[0108] The application layer can include a series of application packages.
[0109] like Figure 3 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.
[0110] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.
[0111] like Figure 3 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.
[0112] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.
[0113] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
[0114] The view system includes visual controls, such as controls for displaying characters, controls for displaying pictures, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text message notification icon can include a view for displaying characters and a view for displaying pictures.
[0115] The phone manager is used to provide communication functions for electronic devices, such as the management of call status (including answering, hanging up, etc.).
[0116] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.
[0117] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.
[0118] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.
[0119] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.
[0120] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
[0121] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.
[0122] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.
[0123] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
[0124] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing.
[0125] A 2D graphics engine is a drawing engine for 2D drawings.
[0126] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.
[0127] Figure 4 It is a software structure block diagram of the electronic device provided in the embodiment of the present application.
[0128] refer to Figure 4 ,The software structure of the electronic device includes modules such as instruction stream interception, frame rate detection, UI separation, frame insertion, redrawing, frame sending, and display sending.
[0129] Among them, the instruction stream interception module is deployed in the system library and is used to intercept the rendering instructions sent by the application to the GPU.
[0130] The frame rate detection module can be deployed in the application layer, for example, it can be integrated in a game optimization application.
[0131] The modules of UI separation, frame insertion, redrawing, and frame sending are deployed in the graphics processing unit (GPU). The UI separation module is used to render the UI and the main screen in two frame buffers respectively, the frame insertion module is used to insert frames into the main screen, the redrawing module is used to mix the inserted main screen with the UI, and the frame sending module is used to transmit the mixed screen to the display sending module.
[0132] The display module can be deployed in the kernel layer to send the received images to the screen for display.
[0133] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0134] Each functional unit and module in the embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit, and the above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional unit. For example, a "module" may be a software program, a hardware circuit, or a combination of the two that implements the above-mentioned functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a merged logic circuit, and / or other suitable components that support the described functions.
[0135] Therefore, the modules of each example described in the embodiments of the present application can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.
[0136] In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the following method embodiment, which will not be repeated here.
[0137] Figure 5 It is a flowchart of the frame insertion method provided in an embodiment of the present application.
[0138] refer to Figure 5 , the interpolation methods include:
[0139] S501: According to an identifier of a target application, match and obtain user interface rendering features corresponding to the target application from a preset storage space.
[0140] In some possible implementations, the user interface rendering features corresponding to the target application include at least three categories. The first category is to determine the start of user interface rendering based on the features of the graphics clear (glClear) instruction. The second category is to determine the start of user interface rendering based on the second draw call (DrawCall) of frame buffer (FB) 0. The third category is the remaining applications except the first two categories. These applications have no obvious features when rendering the UI, and it is impossible to accurately locate when to start rendering the UI. The interpolation method provided in this application is mainly for the first and second categories, and does not temporarily consider the case where the user interface rendering features of the target application are the third category.
[0141] In some possible implementations, the preset storage space may store the application identifier and the type of user interface rendering feature corresponding to the application identifier. The application identifier may be the application name, the application package name, the unified code of the application, etc. The type of user interface rendering feature may be stored by type. For example, if application A corresponds to the first type of user interface rendering feature, the user interface rendering feature corresponding to application A may be marked as "1", if application B corresponds to the second type of user interface rendering feature, the user interface rendering feature corresponding to application B may be marked as "2", and if application C corresponds to the third type of user interface rendering feature, the user interface rendering feature corresponding to application C may be marked as "3".
[0142] The preset storage space may be a configuration file of the UI separation module. As an example, the configuration file may be stored locally on the electronic device or on a server. When the configuration file is stored on the server, it may be updated at preset intervals according to other configuration files stored on the server to ensure that the application identifier recorded in the configuration file and the type of user interface rendering feature corresponding to the application identifier are the latest versions.
[0143] For newly launched applications, the application ID and the user interface rendering features corresponding to the application ID can be analyzed offline on the server side, and then the application ID and the user interface rendering features corresponding to the application ID can be updated to the configuration file stored on the server.
[0144] S502: Whether the user interface rendering features corresponding to the target application are obtained, if so, execute S503, if not, execute S504.
[0145] In some possible implementations, if the application identifier of the target application is stored in the configuration file shown in S501, the user interface rendering features corresponding to the target application can be obtained according to the configuration file. If the application identifier of the target application is not stored in the configuration file shown in S501, it is necessary to execute S504 to analyze the user interface rendering features of the target application online.
[0146] S503: Whether it is detected that the screen rendering characteristics of the target application meet the corresponding user interface rendering characteristics, if so, execute S505, if not, execute S503 again.
[0147] In some possible implementations, for the first type of user interface rendering features shown in S501, a screen rendering feature may be first obtained, the screen rendering feature including the type and number of draw calls DrawCall during rendering.
[0148] In some possible implementations, the first type of user interface rendering feature may include: detecting a preset type of DrawCall, and the preset type of DrawCall includes at least one that meets a first preset condition.
[0149] The type of DrawCall can be classified according to the name of the DrawCall, and the DrawCall names that need to be identified may include "glDrawArrays", "glDrawElementsBaseVertex", and "glDrawElements". Each DrawCall with a name can be regarded as a class.
[0150] Detecting a preset type of DrawCall means that there is an instruction to draw in the FB through the preset type of DrawCall. As an example, it can be detected by monitoring whether there is a preset type of DrawCall in the instruction called by the target application. If there is, it is confirmed that the preset type of DrawCall is detected.
[0151] Then, for each detected DrawCall of the preset type, it is necessary to determine whether it meets the first preset condition. In some possible implementations, the first preset condition includes: detecting a preset first clear instruction, detecting a preset second clear instruction, detecting hybrid on, detecting disabling deep write, detecting that the number of DrawCalls between executing the preset first clear instruction and executing the preset second clear instruction is greater than 0, detecting that the count of the current DarwCall is less than a preset count threshold, and detecting that the count of the frame buffer of the last execution of the second clear instruction in the previous frame meets the second preset condition.
[0152] In some implementations, the features shown in the first preset condition need to be met in sequence. As an example, the features can be numbered:
[0153] Feature 1: A preset first clearing instruction is detected.
[0154] Feature 2: The preset second clear instruction is detected.
[0155] Feature 3: Hybrid on detected.
[0156] Feature 4: Depth write disabled detected.
[0157] Feature 5: It is detected that the number of DrawCalls between the execution of the preset first clear instruction and the execution of the preset second clear instruction is greater than 0.
[0158] Feature 6: It is detected that the current DarwCall count is less than the preset count threshold.
[0159] Feature 7: It is detected that the count of the last frame buffer for executing the second clear instruction in the previous frame meets the second preset condition.
[0160] The first clearing instruction preset in feature 1 may be "glClear<Color Depth Stencil> " This instruction is used to indicate the start of initialization. During initialization, glClearColor(), glClearDepth(), and glClearStencil() are called to set the default values of the cleared color cache, depth cache, and stencil cache respectively.
[0161] The second clearing instruction preset in feature 2 may be "glClear<Depth Stencil> " This instruction is used to indicate the start of initialization. During initialization, glClearDepth() and glClearStencil() are called to set the default values of the cleared depth buffer and stencil buffer respectively.
[0162] If blending is detected in feature 3, it is necessary to detect whether the value of "Enable" in "GL_BLEND" is "true".
[0163] Detecting the disabling of depth writing in feature 4 requires detecting whether the value of "DepthMask" is "False".
[0164] Feature 5 is used to determine whether the main scene is being drawn. If feature 5 is met, it means that a frame buffer is being used to draw the main scene.
[0165] Feature 6 is used to determine whether there is an opportunity to perform UI separation. Since feature 6 comes after feature 2, feature 2 is satisfied when feature 6 is identified. After feature 2 is satisfied, there is an opportunity to perform UI separation only when the DarwCall count is less than the preset count threshold.
[0166] In feature 7, the second preset condition includes: a frame buffer index (Framebuffer Index) of the current frame is equal to a frame buffer count of the last second clear instruction executed in the previous frame.
[0167] In some possible implementations, when features 1 to 7 are met, it can be determined that the screen rendering features of the target application meet the corresponding user interface rendering features, and S505 can be executed, that is, UI separation is performed.
[0168] Some possible implementations may further include feature 8: when the number of frames satisfying the first preset condition obtained on the preset type of DrawCall is greater than the preset number threshold, it is determined that the preset type of DrawCall meets the first preset condition. If there is a frame that does not meet the first preset condition, the count is restarted until the number of frames satisfying the first preset condition is greater than the preset number threshold, and then UI separation is performed.
[0169] It should be noted that, when identifying the first type of user interface rendering features, the number of DrawCalls is the number of DrawCalls of a preset type.
[0170] In some implementations, the DrawCall instruction may be intercepted in the memory. Taking the interception of the following partial rendering instruction as an example, the identification of the first type of user interface rendering features is described.
[0171] The intercepted rendering instructions include:
[0172] glDrawElements(4488)
[0173] glClear(Depth=<1.000000>,Stencil= )
[0174] glBlendFuncSeparate(GL_SRC_ALPHA, GL_LINES, GL_SRC_ALPHA, GL_LINES)
[0175] glDepthMask(False)
[0176] glEnable(CL_CULL_FACE)
[0177] glDrawElements(90)
[0178] glDrawElements(54)
[0179] Assume that glClear is included before the intercepted instruction<Color Depth Stencil> , in glClear<Color Depth Stencil> and glClear(Depth=<1.000000>, Stencil= ) includes at least one "glDrawElements", which meets features 1, 2, and 5.
[0180] The intercepted instructions include glDepthMask(False), which meets feature 4.
[0181] For feature 3, you can check the value of "Enable" in the "target Blends" item in the glDrawElements(90) instruction. If the value is "true", it can be determined that blending is enabled, which meets feature 3.
[0182] Assume that the current DarwCall count is less than the preset count threshold and the count of the last frame buffer that executes the second clear instruction in the previous frame meets the second preset condition, that is, meets condition 6 and condition 7.
[0183] In this example, it can be determined that the screen rendering characteristics of the target application detected at the glDrawElements(90) instruction meet the corresponding user interface rendering characteristics, that is, UI separation can be performed before glDrawElements(90).
[0184] In some possible implementations, for the second type of user interface rendering features shown in S501, the screen rendering features may be first obtained, and the screen rendering features include the number of DrawCalls, the number of color channels (ColorPass) of the draw call during rendering, and the drawing state of the target frame buffer.
[0185] The second type of user interface rendering features generally include multiple Color Passes. This type of application generally renders the main scene on other FBs first, then copies the image of the main scene on FB0, and then starts rendering the UI.
[0186] The second category of user interface rendering features includes: detecting that blending is turned on, detecting that rendering is to the target frame buffer, detecting that the number of DrawCalls corresponding to the target frame buffer is greater than 1, detecting that the drawing state of the target frame buffer is the last drawn frame buffer, and detecting that there is a frame buffer with a color channel in addition to the target frame buffer.
[0187] In some implementations, the features shown above need to be met in order. As an example, the features can be numbered:
[0188] Feature 1: Hybrid on detected.
[0189] Feature 2: Rendering to a target frame buffer is detected.
[0190] Feature 3: It is detected that the number of DrawCalls corresponding to the target frame buffer is greater than 1.
[0191] Feature 4: It is detected that the drawing state of the target frame buffer is the last drawn frame buffer.
[0192] Feature 5: It is detected that there is a frame buffer with color channels in addition to the target frame buffer.
[0193] Among them, if blending is detected in feature 1, it is necessary to detect whether the value of "Enable" in "GL_BLEND" is "true".
[0194] In feature 2, the target frame buffer may be FB0, that is, feature 2 is satisfied when rendering to FB0 is detected.
[0195] In feature 3, it can be detected whether the number of DrawCalls corresponding to FB0 is greater than 1. If it is greater than 1, feature 3 is satisfied.
[0196] In feature 4, when it is determined that FB0 is the last FB to be drawn, it can be guaranteed that FB0 of the previous frame is valid, that is, feature 4 is satisfied.
[0197] When the main scene is drawn in a FB other than the target frame buffer, there will be a ColorPass in the FB, which means that feature 5 is satisfied.
[0198] In some possible implementations, when features 1 to 5 are met, it can be determined that the screen rendering features of the target application meet the corresponding user interface rendering features, and S505 can be executed, that is, UI separation is performed.
[0199] Some possible implementations may also include feature 6: FB0 of the previous frame has glClear <depthstencil>instruction.
[0200] If feature 6 is met, UI separation is postponed to glClear<Depth Stencil> Afterwards, the DrawCall count of FB0 may be greater than 2. If there is no glClear instruction, UI separation is performed before the second DrawCall.
[0201] Some possible implementations may further include feature 7: when the number of frames satisfying features 1 to 5 is greater than a preset number threshold, it is determined that the screen rendering features of the target application meet the corresponding user interface rendering features, and UI separation is performed. When there is a frame that does not satisfy features 1 to 5, the count is restarted until the number of frames satisfying features 1 to 5 is greater than the preset number threshold, and UI separation is performed.
[0202] In some implementations, the DrawCall instruction may be intercepted in the memory. Taking the interception of the following partial rendering instruction as an example, the identification of the second type of user interface rendering features is described.
[0203] The intercepted rendering instructions include:
[0204] Color Pass#4(1-3Targets+Depth)
[0205] glBindFramebuffer(GL_READ_FRAMEBUFFER, backbuffer FB0)
[0206] glBindFramebuffer(GL_READ_FRAMEBUFFER, Framebuffer 16227)
[0207] glBindFramebuffer(GL_READ_FRAMEBUFFER, Framebuffer 16233)
[0208] glInvalidateFramebuffer(Framebuffer16233)
[0209] glInvalidateFramebuffer(Framebuffer 16233)
[0210] glclear(Color=<0.000000,0.000000,0.000000,0.000000>,Depth=<1.000000>,Stencil=
[0211] glEnableVertexAttribArray(Vertex hrray 25,2)
[0212] glEnableVertexAttribArray(Vertex hrray 25,3)
[0213] glEnableVertexAttribArray(Vertex hrray 25,4)
[0214] glDrawElements(5274)
[0215] glEnable(GL_STENCIL_TEST)
[0216] glEnable(GL_CULL_FACE)
[0217] glDrawElements(1242)
[0218] glDrawElementsInstanced(177,3)
[0219] glDrawElements(1311)
[0220] glDrawElements(3510)
[0221] glDrawElementsInstanced(717,2)
[0222] glDrawElements(1035)
[0223] Colour Pass#25(1Targets+Depth)
[0224] glBindFramebuffer(GL_DRAW_FRAEBUFFER,Backbuffer FB0)
[0225] glInvalidateFramebuffer(Backbuffer FB0)
[0226] glDepthMask(True)
[0227] glclear(Color=<0.000000,0.000000,0.000000,0.000000>,Depth=<1.000000>,Stencil=<0x00>
[0228] glDepthMask(False)
[0229] glDrawElements(3)
[0230] glEnable(GL_SCISSOR_TEST)
[0231] glDepthMask(True)
[0232] glclear(Depth=<1.000000>,Stencil= )
[0233] glEnable(GL_BLEND)
[0234] glDepthMask(False)
[0235] glEnableVertexAttribArray(Vertex Array 25,2)
[0236] glEnableVertexAttribArray(Vertex Array 25,3)
[0237] glEnableVertexAttribArray(Vertex Array 25,4)
[0238] glDrawElements(18)
[0239] glDrawElements(18)
[0240] Referring to the intercepted instructions, the FB bound to Color Pass #4 is not FB0, so the Color Pass renders the main scene in other FBs. Color Pass #25 is bound to FB0, and the main scene is copied in the first DrawCall (glBindFramebuffer(GL_DRAW_FRAEBUFFER, Backbuffer FB0)).
[0241] Since glClear clears the depth and stencil (glclear(Depth=<1.000000>, Stencil= )), feature 6 is satisfied. Therefore, it is checked in the DrawCall following glClear whether features 1 to 5 are satisfied.
[0242] As an example, there is a "glDepthMask(False)" instruction before the glDrawElements(18) instruction. You can check the value of "Enable" in the "target Blends" item in the glDrawElements(18) instruction. If the value is "true", blending is enabled, which meets feature 1. FB0 is bound to Color Pass#25, which meets feature 2. Among the DrawCall instructions intercepted above, the DrawCall instruction corresponding to FB0 is greater than 1, which meets feature 3. The main scene is rendered in other FBs through Color Pass#4, which meets feature 5. Assume that FB0 is the last FB to be drawn, which meets feature 4.
[0243] In this example, it can be determined that the screen rendering characteristics of the target application detected at the glDrawElements(18) instruction meet the corresponding user interface rendering characteristics, that is, UI separation can be performed before glDrawElements(18).
[0244] In this embodiment, the first type of user interface rendering features and the second type of user interface rendering features are used to identify when the target application starts rendering the UI, and UI separation is performed before rendering the UI. The main interface and the UI can be accurately separated to facilitate frame insertion of the main interface separately, thereby improving the effect of frame insertion and bringing a better user experience.
[0245] S504: Analyze the target application to obtain user interface rendering features corresponding to the target application.
[0246] In some possible implementations, the target application identifier can be uploaded to the server, and the server retrieves the corresponding user interface rendering features in the stored configuration file according to the representation of the target application. If retrieved, the retrieved user interface rendering features and the application identifier are sent to the electronic device, and the electronic device obtains the user interface rendering features corresponding to the target application.
[0247] Alternatively, if the target application is not retrieved, the server can analyze the target application online according to the target application identifier to obtain the user interface rendering features of the target application. Then, the retrieved user interface rendering features and the application identifier are sent to the electronic device, and the electronic device obtains the user interface rendering features corresponding to the target application.
[0248] In some possible implementations, after the electronic device receives the user interface rendering feature corresponding to the target application, it may also write the received identifier of the target application and the corresponding user interface rendering feature into a local configuration file.
[0249] S505: Draw the main screen in the first frame buffer, and draw the user interface in the second frame buffer.
[0250] In some possible implementations, after it is determined to perform UI separation according to S503, a first frame buffer for drawing the main screen and a second frame buffer for drawing the user interface may be created respectively.
[0251] Then, the rendering instruction is intercepted, and before the rendering instruction instructs drawing the main picture, the frame buffer for drawing the main picture is replaced with the first frame buffer.
[0252] Before the rendering instruction instructs drawing the user interface, the frame buffer for drawing the user interface is replaced with the second frame buffer.
[0253] As an example, when intercepting rendering instructions, the UI separation module can determine the default address of the drawing function in memory according to the drawing function called by the target application, and modify the address in memory to the preset address, that is, the drawing function will output the drawing instruction to the preset address, identify and modify the drawing instruction at the preset address. After that, the drawing instruction is returned to the default address and executed.
[0254] S506: Perform frame insertion processing on the main picture in the first frame buffer to obtain the main picture after frame insertion.
[0255] In some possible implementations, the interpolation process may call an interpolation interface to calculate and generate an intermediate frame or an extrapolated frame based on the main picture of the previous frame and the main picture of the current frame. The generated intermediate frame or extrapolated frame is the main picture after interpolation.
[0256] In one example, when generating an intermediate frame (i.e., an interpolated frame), the interpolation algorithm provided by the interpolation interface can be called, and the main picture of the previous frame (n-1 frame) and the main picture of the current frame (n frame) are input into the interpolation algorithm. The interpolation algorithm calculates and generates at least one interpolation picture (i.e., an intermediate frame) between the n-1 frame and the n frame based on the main pictures of the n-1 frame and the n frame. The interpolation algorithm for generating the intermediate frame may include: an optical flow algorithm, an interpolation algorithm based on deep learning, an interpolation algorithm based on linear predictive coding (LPC), etc.
[0257] In another example, when generating an extrapolated frame (i.e., an interpolated frame), an interpolation algorithm provided by an interpolation interface is called, and the main picture of the previous frame (n-1 frame) and the main picture of the current frame (n frame) are input into the interpolation algorithm. The interpolation algorithm calculates and generates at least one interpolated picture (i.e., an extrapolated frame) between the n frame and the n+1 frame based on the main pictures of the n-1 frame and the n frame. The interpolation algorithm for generating an intermediate frame may include: an alignment interpolation frame algorithm based on a neural network, etc.
[0258] S507: The redrawing module mixes the inserted main picture with the user interface in the second frame buffer to obtain a picture of the target application.
[0259] In some possible implementations, the redrawing module merges the main screen after interpolation output by the interpolation module with the UI in the second frame buffer to obtain the screen of the target application.
[0260] S508: The redrawing module transmits the screen of the target application to the frame sending module.
[0261] S509: The frame sending module transmits the screen of the target application to the display sending module.
[0262] In some possible implementations, for the intermediate frames, the frame sending module may first buffer and swap the intermediate frames to the SurfaceFlinger thread, and then push the real frames at an appropriate time.
[0263] For example, when the frame sending module receives a new intermediate frame, it will not push the intermediate frame and display it on the screen immediately, but will buffer the intermediate frame into memory first and swap it to the SurfaceFlinger thread. This ensures that the next frame is ready without tearing or abnormal screen due to display delay or synchronization problems.
[0264] In some possible implementations, the SurfaceFlinger thread is responsible for pushing frames and displaying them on the screen. The right time may be when the vertical synchronization (VSync) signal arrives. In this case, the SurfaceFlinger thread switches the buffered intermediate frames with the real frames and pushes the real frames to the screen for display. By buffering the intermediate frames in advance and pushing the real frames at the right time, the smoothness and accuracy of the picture can be ensured.
[0265] In some possible implementations, for the extrapolated frame, the real frame can be sent to SurfaceFlinger first, and then the extrapolated frame can be pushed at an appropriate time.
[0266] As an example, when a new real frame is generated, the real frame can be sent to the SurfaceFlinger thread first, and then displayed on the screen after processing and combination. This ensures that the original real frame is displayed on the screen in time, ensuring the accuracy and stability of the picture.
[0267] When the SurfaceFlinger thread is ready to display the next frame, it can push the extrapolated frame at the right time, which may be hardware performance, prediction algorithm, etc., to ensure that the inserted extrapolated frame can adapt to the content and motion of the original real frame.
[0268] S510: The display module displays the screen of the target application on the screen.
[0269] In some possible implementations, the display module may transmit the received frames to the screen driver and display the frames on the screen.
[0270] It should be understood that the above examples are intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to the specific numerical values or specific scenarios illustrated.
[0271] It is obvious that those skilled in the art can make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of the present application.
[0272] Corresponding to the frame insertion method provided in the above embodiment, Figure 6 This is a structural block diagram of a frame insertion device provided in an embodiment of the present application. For the sake of convenience of explanation, only the parts related to the embodiment of the present application are shown.
[0273] Reference Figure 6 , a frame insertion device is applied to an electronic device, the electronic device runs a target application, the screen of the target application includes a main screen and a user interface, and the device includes:
[0274] An identification module 61 is used to identify whether the target application starts to render the user interface. When it is detected that the target application starts to render the user interface, the subsequent main screen is drawn in the first frame buffer and the user interface is drawn in the second frame buffer;
[0275] The interpolation module 62 is used to perform interpolation processing on the main picture in the first frame buffer to obtain the main picture after interpolation;
[0276] The mixing module 63 is used to mix the main picture after the frame insertion with the user interface in the second frame buffer as the picture output of the target application.
[0277] In some possible implementations, the identification module 61 is specifically used to obtain corresponding user interface rendering features based on the identification of the target application; when it is detected that the screen rendering features of the target application meet the corresponding user interface rendering features, it is determined that the target application starts rendering the user interface, and the screen rendering features include at least one of the type of drawing call DrawCall during rendering, the number of DrawCalls, the number of color channels, and the drawing status of the target frame cache.
[0278] In some possible implementations, the corresponding user interface rendering feature includes: a preset type of DrawCall is detected, and the preset type of DrawCall includes at least one that meets the first preset condition.
[0279] In some possible implementations, the first preset condition includes: detecting a preset first clear instruction, detecting a preset second clear instruction, detecting that blending is turned on, detecting that deep writing is disabled, detecting that the number of DrawCalls between executing the preset first clear instruction and executing the preset second clear instruction is greater than 0, detecting that the count of the current DarwCall is less than a preset count threshold, and detecting that the count of the frame cache of the last execution of the second clear instruction in the previous frame meets the second preset condition.
[0280] In some possible implementations, the second preset condition includes: the frame buffer index of the current frame is equal to the count of the last frame buffer in which the second clear instruction is executed in the previous frame.
[0281] In some possible implementations, the identification module 61 is further configured to determine that a DrawCall of the preset type meets the first preset condition when the number of frames satisfying the first preset condition obtained on the DrawCall of the preset type is greater than a preset number threshold.
[0282] In some possible implementations, the corresponding user interface rendering features include: detecting that blending is turned on, detecting that rendering is to a target frame buffer, detecting that the number of DrawCalls corresponding to the target frame buffer is greater than 1, detecting that the drawing state of the target frame buffer is the last drawn frame buffer, and detecting that there is a frame buffer with a color channel in addition to the target frame buffer.
[0283] Some possible implementations further include an acquisition module 64 for analyzing the target application to acquire the user interface rendering features corresponding to the target application; and storing the target application identifier and the user interface rendering features corresponding to the target application in a preset storage space.
[0284] The acquisition module 64 is further configured to match and acquire the user interface rendering features corresponding to the target application from a preset storage space according to the identifier of the target application.
[0285] It should be understood that the hardware system and chip in the embodiments of the present application can execute the various methods of entering the long standby mode in the aforementioned embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the aforementioned method embodiments.
[0286] An embodiment of the present application also provides another electronic device, including a processor and a memory.
[0287] Memory is used to store computer programs that can be executed on the processor.
[0288] The processor is used to execute the processing steps in the method for entering the long standby mode as described above.
[0289] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored; when the computer-readable storage medium is executed on an electronic device, the electronic device executes the method as shown above.
[0290] The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.
[0291] The computer-readable storage medium may be any available medium that can be accessed by a computer or may include one or more servers, data centers, and other data storage devices that can be integrated with the medium.
[0292] The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (eg, a solid state disk (SSD)).
[0293] The embodiment of the present application also provides a computer program product including computer instructions, which, when executed on an electronic device, enables the electronic device to execute the technical solution shown above.
[0294] Figure 7 It is a schematic diagram of the structure of a chip provided in an embodiment of the present application. Figure 7 The chip shown can be a general-purpose processor or a dedicated processor. The chip includes a processor 701. The processor 701 is used to support the electronic device to execute the technical solution shown above.
[0295] Optionally, the chip further includes a transceiver 702, and the transceiver 702 is used to accept the control of the processor 701 and to support the communication device to execute the technical solution shown above.
[0296] Optional, Figure 7 The chip shown may further include: a storage medium 703 .
[0297] It should be noted that Figure 7 The chip shown can be implemented using the following circuits or devices: one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits that can perform the various functions described throughout this application.
[0298] The electronic device, computer storage medium, computer program product, and chip provided in the above-mentioned embodiments of the present application are all used to execute the methods provided above. Therefore, the beneficial effects that can be achieved can refer to the corresponding beneficial effects of the methods provided above, and will not be repeated here.
[0299] It should be understood that the above is only to help those skilled in the art to better understand the embodiments of the present application, and is not intended to limit the scope of the embodiments of the present application. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples given.
[0300] For example, some steps in each embodiment of the above method may be unnecessary, or some new steps may be added, etc. Or a combination of any two or any multiple embodiments above. Such modified, changed or combined solutions also fall within the scope of the embodiments of the present application.
[0301] It should also be understood that the above description of the embodiments of the present application focuses on emphasizing the differences between the various embodiments. The same or similar points that are not mentioned can be referenced to each other. For the sake of brevity, they will not be repeated here.
[0302] It should also be understood that the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0303] It should also be understood that in the embodiments of the present application, "pre-setting" and "pre-definition" can be achieved by pre-saving corresponding codes, tables or other methods that can be used to indicate relevant information in a device (for example, including an electronic device), and the present application does not limit its specific implementation method.
[0304] It should also be understood that the division of the methods, situations, categories and embodiments in the embodiments of the present application is only for the convenience of description and should not constitute a special limitation. The features of various methods, categories, situations and embodiments can be combined without contradiction.
[0305] It should also be understood that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.
[0306] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.< / depthstencil>
Claims
1. A frame insertion method, characterized in that: Applied to an electronic device, the electronic device runs a target application, the screen of the target application includes a main screen and a user interface, and the method includes: According to the identifier of the target application, obtaining corresponding user interface rendering features; When it is detected that the screen rendering feature of the target application meets the corresponding user interface rendering feature, it is determined that the target application starts to render the user interface, wherein the screen rendering feature includes at least one of the type of the draw call DrawCall during rendering, the number of DrawCalls, the number of color channels, and the drawing state of the target frame buffer, and the corresponding user interface rendering feature includes: detecting a preset type of DrawCall, and the preset type of DrawCall includes at least one DrawCall that meets a first preset condition, and the first preset condition includes: detecting a preset first clear instruction, detecting a preset second clear instruction, detecting blending is enabled, detecting disabling deep writing, detecting that the number of DrawCalls between executing the preset first clear instruction and executing the preset second clear instruction is greater than 0, detecting that the count of the current DrawCall is less than a preset count threshold, and detecting that the count of the last frame buffer that executes the second clear instruction in the previous frame meets the second preset condition; or, the corresponding user interface rendering feature includes: detecting blending is enabled, detecting rendering to the target frame buffer, detecting that the number of DrawCalls corresponding to the target frame buffer is greater than 1, detecting that the drawing state of the target frame buffer is the last drawn frame buffer, and detecting that there is a frame buffer with color channels in addition to the target frame buffer; When it is detected that the target application starts to render the user interface, the subsequent main screen is drawn in the first frame buffer, and the user interface is drawn in the second frame buffer; Performing frame interpolation processing on the main picture in the first frame buffer to obtain a main picture after interpolation; The inserted main picture is mixed with the user interface in the second frame buffer and output as the picture of the target application.
2. The method according to claim 1, characterized in that The second preset condition includes: The frame buffer index of the current frame is equal to the count of the frame buffer that last executed the second clear instruction in the previous frame.
3. The method according to claim 1 or 2, characterized in that: The method further comprises: When the number of frames satisfying the first preset condition obtained in the preset type of DrawCall is greater than the preset number threshold, it is determined that the preset type of DrawCall meets the first preset condition.
4. The method according to any one of claims 1 to 3, characterized in that Before acquiring the corresponding user interface rendering feature according to the identifier of the target application, the method further includes: Analyze the target application to obtain user interface rendering features corresponding to the target application; Storing the identifier of the target application and the user interface rendering characteristics corresponding to the target application in a preset storage space; The acquiring corresponding user interface rendering features according to the identifier of the target application includes: According to the identifier of the target application, the user interface rendering feature corresponding to the target application is matched and obtained from the preset storage space.
5. A frame insertion device, characterized in that: Applied to an electronic device, the electronic device runs a target application, the screen of the target application includes a main screen and a user interface, and the device includes: The identification module is used to obtain the corresponding user interface rendering feature according to the identifier of the target application, and when it is detected that the screen rendering feature of the target application meets the corresponding user interface rendering feature, determine that the target application starts to render the user interface, wherein the corresponding user interface rendering feature includes: detecting a preset type of DrawCall, and the preset type of DrawCall includes at least one DrawCall that meets a first preset condition, and the first preset condition includes: detecting a preset first clear instruction, detecting a preset second clear instruction, detecting hybrid enablement, detecting disabling deep writing, and detecting a period between executing the preset first clear instruction and executing the preset second clear instruction. The number of DrawCalls is greater than 0, the count of the current DarwCall is detected to be less than a preset count threshold, and the count of the last frame buffer that executes the second clear instruction in the previous frame is detected to meet the second preset condition; or, the corresponding user interface rendering characteristics include: detecting that blending is turned on, detecting rendering to the target frame buffer, detecting that the number of DrawCalls corresponding to the target frame buffer is greater than 1, detecting that the drawing state of the target frame buffer is the last drawn frame buffer, and detecting that there is a frame buffer with a color channel in addition to the target frame buffer; when it is detected that the target application starts to render the user interface, the subsequent main screen is drawn in the first frame buffer, and the user interface is drawn in the second frame buffer; A frame insertion module, used for performing frame insertion processing on the main picture in the first frame buffer to obtain the main picture after frame insertion; A mixing module is used to mix the main picture after the frame insertion with the user interface in the second frame buffer as the picture output of the target application.
6. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 4 when executing the computer program.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 4.
Citation Information
Patent Citations
Video frame insertion method and related devices
CN111327959A
Image data processing method and related devices
CN111356026A
Rendering method and device, electronic equipment and computer readable storage medium
CN111798361A
Display system
CN203313319U
Method for displaying user interface using image blending
KR1020110010012A
