Rendering method, electronic device and computer readable storage medium
By adding locks when the rendering thread runs tasks, the application main thread is prohibited from adding new tasks, which solves the problem of rendering delay of electronic devices, realizes timely response to user operations, and improves the user experience.
Patent Information
- Application Number
- CN202310802292.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-30
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-06-30
AI Technical Summary
The time spent by electronic devices when rendering the screen to be displayed is too long, resulting in delayed user operation feedback and affecting user experience.
When the rendering thread runs a task, the main thread of the application is prohibited from adding new tasks to the task queue until the task queue is empty and a rendering instruction is generated is sent to the GPU to ensure that the display responds to user operations in a timely manner.
It effectively reduces the accumulation of tasks in the task queue, improves the display's response speed to user operations, reduces latency, and improves user experience.
Smart Images

Figure CN117689785B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminals, and in particular to a rendering method, an electronic device, and a computer-readable storage medium. Background Art
[0002] Electronic devices display images on the display screen to achieve human-computer interaction with users. In some scenarios, users need electronic devices to give feedback on the display screen in a timely manner according to user operations, but because the electronic device takes too long to render the image to be displayed, the image to be displayed cannot be displayed on the display screen in time, affecting the user experience. Summary of the invention
[0003] The present application provides a rendering method, an electronic device, and a computer-readable storage medium. The electronic device can lock the rendering thread when the rendering thread runs a task in the task queue, so that the main thread of the application cannot add new tasks to the task queue when the rendering thread runs the task. The rendering thread can send the rendering instructions generated after the task is run to the graphics processor for rendering in a timely manner, so that the display screen can display the corresponding picture in time according to the user operation.
[0004] In a first aspect, the present application provides a rendering method, which is applied to an electronic device, and the method includes: the electronic device displays a user interface of a first application on a display screen; the electronic device takes out a first task from a task queue, wherein the task in the task queue is added by the first application; the electronic device runs the taken out first task to obtain a graphics processor GPU rendering instruction, and prohibits the first application from adding tasks to the task queue during the running period of the first task; during the non-running period of the first task, the first application is allowed to add tasks to the task queue when a vertical synchronization VSYNC signal arrives; when the task queue is empty, the electronic device performs rendering based on the rendering instructions generated by the tasks in the running task queue.
[0005] The electronic device can display the user interface of the first application, and the first application in the electronic device can add tasks to the task queue, wherein the above-mentioned tasks can describe the view layout, size, etc. in the user interface of the first application. The task added by the first application to the task queue protects the first task, and the electronic device can run the first task and generate rendering instructions corresponding to the first task. Wherein, during the running of the first task, the first application is prohibited from adding new tasks to the task queue. As long as there are tasks in the task queue, the electronic device needs to take the tasks out of the task queue for running. Only when the task queue is empty will the electronic device send the rendering instructions generated after running the task to the GPU, so that the GPU can render the view in the user interface based on the rendering instructions.
[0006] Since the first application is prohibited from adding new tasks to the task queue when the electronic device is running a task, there will be no accumulation of tasks in the task queue. The electronic device can generate rendering instructions after running the tasks in the task queue in time and send them to the GPU for rendering, so that the display screen displays the corresponding picture.
[0007] In conjunction with the first aspect, in some embodiments, the running time of the task before the first task exceeds the first time duration. The task before the first task may include the task that the electronic device ran after the last time the rendering instruction was sent to the GPU, and before the first task. The electronic device may prohibit the first application from adding new tasks to the task queue only when the sum of the running times of the above tasks exceeds the first time duration.
[0008] In conjunction with the first aspect, in some embodiments, the task before the first task may be the previous task of the first task. The electronic device may record the time of each task execution. When the electronic device executes the previous task of the first task, and the execution time exceeds the first time length, the first application may be prohibited from adding new tasks to the task queue when executing the first task.
[0009] In combination with the first aspect, in some embodiments, the first duration is equal to the time interval between two adjacent vertical synchronization VSYNC signals.
[0010] Among them, the time interval between vertical synchronization VSYNC signals can be equal to the time it takes for the display screen to refresh the display screen once. When the time it takes for an electronic device to run a task exceeds the time interval between two adjacent VSYNC signals, the electronic device does not generate a screen with different screen content based on the tasks in the task queue during the time interval for the display screen to refresh the screen. If the main thread of the application continues to add tasks to the task queue, it may cause a backlog of tasks in the task queue, increasing the time interval for the electronic device to display a new screen based on the tasks in the task queue. Therefore, when the above situation occurs, the electronic device can promptly prohibit the main thread of the application from adding new tasks to the task queue to reduce the delay when the electronic device displays the screen.
[0011] In combination with the first aspect, in some embodiments, the user interface of the first application includes a canvas; the tasks in the task queue are created by the electronic device based on detected user input when displaying the user interface of the first application, the user input is used to draw the first content on the canvas, and the tasks are used to describe drawing information of the first content; the drawing information includes one or more of the following: position, size, shape, and color of the first content.
[0012] In combination with the first aspect, in some embodiments, after the electronic device performs rendering based on the rendering instruction generated by the task in the running task queue, the method further includes: the electronic device adds display of the first content on the canvas.
[0013] Among them, the first application may be a drawing application for drawing patterns. The user interface of the drawing application may include a canvas. After the electronic device receives the user's operation of drawing the first content on the canvas (such as pressing, sliding, releasing, etc. on the display screen), it can generate drawing information describing the first content. For example, when the first content is a curve, the drawing information may include the coordinates of one or more points of the curve on the canvas, as well as the size, color, etc. of the curve; when the first content is a specific pattern (such as a circle), the drawing information may include the coordinates of each point of the circular outline on the canvas, the center of the circle, the size of the circular outline, etc. The electronic device may generate a rendering instruction by running the task, and then the GPU in the electronic device may render the first content based on the rendering instruction. The rendered first content can be displayed on the canvas after being synthesized and sent to the display by the electronic device.
[0014] In a drawing application, the display screen needs to display a corresponding picture on the canvas in a timely manner according to the user's drawing operation on the canvas. The main thread of the application in the electronic device can generate tasks based on the user's operations such as pressing, sliding, and releasing fingers on the display screen, and then add the tasks to the task queue. As long as the user draws on the display screen, the main thread of the application will generate new tasks based on the VSYNC signal. The electronic device can prohibit the main thread of the application from adding new tasks to the task queue when processing tasks, thus avoiding the situation where the electronic device cannot display the user's drawn content on the display screen for a long time according to the user's drawing operation when the user draws on the canvas due to a backlog of tasks in the task queue.
[0015] In combination with the first aspect, in some embodiments, the user input includes a sliding input, and the first content is determined based on a sliding trajectory of the sliding input; before the electronic device locks the first application during the execution of the first task, the method also includes: the electronic device detects that the sliding speed of the sliding input exceeds the first rate.
[0016] When the user's sliding speed on the canvas exceeds the first rate, the content drawn by the user per unit time increases, and the task generated by the application main thread based on the user operation contains more drawing information. The rendering thread needs to spend more time running the task, which makes it more likely to have poor hand tracking. Therefore, the electronic device can timely adjust the logic of the application main thread to add tasks to the task queue when it detects that the user's sliding speed is too fast, to avoid the situation of poor hand tracking.
[0017] In combination with the first aspect, in some embodiments, before the electronic device locks the first application during execution of the first task, the method further includes: the electronic device detects that a refresh rate of the display screen exceeds a first refresh rate.
[0018] When the refresh rate of the display screen is faster, the time interval between two adjacent VSYNC signals is shorter. The time it takes for an electronic device to run a task is more likely to exceed the time interval between two adjacent VSYNC signals, which means that when the electronic device sends a new VSYNC signal, the task generated by the previous VSYNC signal is still running. At this time, if the first application continues to add new tasks to the task queue in response to the VSYNC signal, a backlog of tasks in the task queue may occur. The electronic device can adjust the logic of the application main thread to add tasks to the task queue when the refresh rate exceeds the first refresh rate to avoid the situation where the hand does not follow.
[0019] In combination with the first aspect, in some embodiments, the first application includes an application main thread and a rendering thread, wherein the application main thread is used to add tasks to a task queue, and the rendering thread is used to retrieve and run tasks from the task queue.
[0020] The main thread of the application in the first application can receive the user operation to determine the change of the view caused by the user operation, and can also measure the size of the view, obtain the layout of the view, etc., to obtain the drawing information describing the view. The main thread of the application can generate one or more tasks, and the tasks can include drawing information. Then the main thread of the application can add the task to the task queue, and the rendering thread can take the task from the task queue and run it. This can avoid the main thread of the application spending too many resources to process the image in the display interface, affecting the response time of the first application, etc.
[0021] In combination with the first aspect, in some embodiments, the first application is prohibited from adding tasks to the task queue during the execution of the first task, specifically including: the electronic device locks the rendering thread while the rendering thread is running the first task, and the application main thread is prohibited from adding tasks to the task queue while the rendering thread is locked.
[0022] Among them, the electronic device can control the thread that can operate the task queue in the form of a lock. Only the thread holding the lock can operate the task queue. When the rendering thread runs the task, the rendering thread holds the lock, and the application main thread cannot obtain the lock, and cannot add tasks to the task queue. In this way, the electronic device can more conveniently control the logic of the application main thread to add tasks to the task queue. Optionally, the above lock can be a mutex lock.
[0023] In a second aspect, the present application provides an electronic device, comprising a display screen, a memory, and a processor coupled to the memory; the display screen is used to display an interface, the memory stores a computer program, and when the processor executes the above computer program, the electronic device implements any one of the methods described in the above first aspect.
[0024] In a third aspect, the present application provides a computer-readable storage medium, which stores a computer program or computer instructions, and the aforementioned computer program or computer instructions are executed by a processor to implement any method described in the first aspect above.
[0025] In a fourth aspect, an embodiment of the present application provides a computer program product. When the computer program product is executed by a processor, the method described in any one of the above-mentioned first aspects will be implemented.
[0026] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a memory, wherein the memory is used to store computer programs or computer instructions, and the processor is used to execute the computer programs or computer instructions stored in the memory, so that the chip executes any method described in the first aspect above.
[0027] The solutions provided in the second to fifth aspects are used to implement or cooperate with the corresponding methods provided in the first aspect, and therefore can achieve the same or corresponding beneficial effects as the corresponding methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 is a schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of the present application;
[0029] Figure 2 is a schematic diagram of an interface interaction of an electronic device 100 running a drawing application provided by an embodiment of the present application;
[0030] Figure 3A This is a flow chart of an electronic device displaying a screen provided by an embodiment of the present application;
[0031] Figure 3B It is a schematic diagram of the application main thread and rendering thread processing tasks provided by an embodiment of the present application;
[0032] Figure 4A This is a flow chart of a display screen of an electronic device with strong chirality provided by an embodiment of the present application;
[0033] Figure 4B It is a schematic diagram of a display process of an electronic device with poor chirality when displaying a screen provided by an embodiment of the present application;
[0034] Figure 5 is a flowchart of a rendering method provided in an embodiment of the present application;
[0035] Figure 6 is a display flow chart after an electronic device provided by an embodiment of the present application applies a rendering method;
[0036] Figure 7 is a display flow chart of an electronic device provided by an embodiment of the present application after applying another rendering method;
[0037] Figure 8 It is a flowchart for implementing the rendering method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0038] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.
[0039] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.
[0040] The following first introduces the hardware structure of the electronic device 100 provided in the embodiment of the present application.
[0041] Figure 1 It is a schematic diagram of the structure of the electronic device 100 provided in an embodiment of the present application.
[0042] The following is a detailed description of the embodiment using the electronic device 100 as an example. It should be understood that the electronic device 100 may have more than Figure 1 More or fewer components may be shown, two or more components may be combined, or there may be a different configuration of components. Figure 1 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.
[0043] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a camera 193, a display screen 194, etc. The sensor module 180 may include a pressure sensor 180A, a touch sensor 180B, etc.
[0044] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0045] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0046] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0047] The charging management module 140 is used to receive charging input from a charger, where the charger can be a wireless charger or a wired charger.
[0048] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, etc.
[0049] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
[0050] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
[0051] The camera 193 is used to capture still images or videos. The electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.
[0052] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
[0053] The video codec is used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 may play or record videos in multiple coding formats.
[0054] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0055] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.
[0056] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, applications required for at least one function (such as face recognition function, fingerprint recognition function, mobile payment function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as face information template data, fingerprint information template, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0057] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.
[0058] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.
[0059] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.
[0060] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.
[0061] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals.
[0062] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.
[0063] The touch sensor 180B is also called a "touch panel". The touch sensor 180B can be arranged on the display screen 194, and the touch sensor 180B and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180B is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180B can also be arranged on the surface of the electronic device 100, which is different from the position of the display screen 194.
[0064] like Figure 2 FIG. 1 is a schematic diagram of an interface interaction of an electronic device 100 running a drawing application. A drawing application may be installed in the electronic device 100. When the electronic device 100 runs the drawing application, the following may be displayed: Figure 2 The user interface 200 of the drawing application is shown. The user interface 200 of the drawing application may include a canvas 201 , and the electronic device 100 may receive and respond to user operations on the canvas 201 , and display drawing content on the canvas 201 .
[0065] In some cases, poor chirality may occur when running a drawing application in the electronic device 100. Figure 2 As shown, the electronic device 100 can receive an operation in which the user's finger presses at position A in the canvas 201 and moves to position C via position B. In response to the operation, the electronic device 100 records the trajectory of the user's finger moving from position A to position B, and draws a curve from position A to position B on the canvas (i.e. Figure 2 The solid line from the starting point of the curve to the end point of the curve in the figure). The curve is the drawing content formed when the user's finger moves from position A to position B. Figure 2The dotted line from position B to position C in the figure is the moving track of the user's finger from position B to position C, but the moving track of the user's finger from position B to position C is not drawn on the canvas 201. In other words, the electronic device 100 does not follow the operation of the user's finger to display the user's drawing content on the canvas 201 in time, which is one of the scenarios where the electronic device 100 has poor chirality.
[0066] When an electronic device receives a user operation but cannot respond to the user operation in time and provide feedback on the display screen, it can be said that the electronic device has poor hand tracking, or that the touch delay of the electronic device is high. The reason why the electronic device has poor hand tracking is that the application program is poorly compatible with the graphics processing unit (GPU), or the computing power of the central processing unit (CPU) is weak, which makes it take a long time (for example, more than 300 milliseconds) for the electronic device to convert the layout file of the application program into data that the GPU can recognize and render. As a result, the electronic device cannot send the rendered image to the display screen in time, and the new image content cannot be displayed on the display screen (for example, the touch delay of the electronic device is high). Figure 2 The curve formed by the user's finger from position B to position C), which results in poor chirality of the electronic device.
[0067] For the convenience of description and better understanding, the following embodiments of this application are Figure 2 The rendering method provided by the embodiment of the present application is introduced by taking the scenario of the electronic device running a drawing application as an example. It is not limited to the scenario of the electronic device running a drawing application, but can also be other scenarios in which the electronic device receives and displays a picture on the display screen in response to a user operation. The embodiment of the present application does not limit the scenario in which the rendering method is applied.
[0068] To facilitate understanding of the rendering method provided by the present application, the method for displaying an image on an electronic device is specifically described below.
[0069] Figure 3A This is a flow chart of an electronic device displaying a screen provided by an embodiment of the present application. Figure 3A As shown, the electronic device 100 may include an event response module 301, a drawing application 302, a layer synthesis module 303, a display driver 304 and a display screen 194. The method of displaying a picture by the electronic device 100 may include but is not limited to the following steps:
[0070] S301. The event response module 301 receives a user operation and determines parameters of the user operation.
[0071] S302 , the event response module 301 sends the parameters of the user operation to the drawing application.
[0072] The event response module 301 may include a touch sensor, which may receive a user's touch operation and generate a hardware interrupt. The event response module 301 may then determine the parameters of the user operation according to the hardware interrupt, and the parameters of the user operation may include the type of the user's touch operation (e.g., press, move, release, etc.), the coordinates of the user's touch operation, the timestamp of the user's touch operation, and other information. Finally, the event response module 301 may send the above parameters to the drawing application 302.
[0073] S303 , the application main thread 306 in the drawing application 302 generates a task according to the parameters operated by the user and adds the task to a task queue.
[0074] S304 , the rendering thread 307 in the drawing application 302 takes a task from the task queue and runs it to generate a rendering instruction, and then renders the view corresponding to the task.
[0075] S305 , the rendering thread 307 in the application 302 sends the rendered view to the layer composition module.
[0076] It should be noted that a process is an execution entity of an application. Multiple threads can be concurrent in a process. The application main thread 306 and the rendering thread 307 are two threads in the application 302, and the application main thread 306 and the rendering thread 307 can process different events in parallel.
[0077] The application main thread 306 can determine the view data to be displayed by identifying the view layout in the user interface of the drawing application 302 and measuring the size of the view. It should be noted that the user interface displayed in the electronic device 100 is composed of one or more views. A view can correspond to an area in the display interface, and a control in the user interface is a special view. For example Figure 2 The canvas 201 shown is a view in the user interface. The application main thread 306 can also receive user operations (such as Figure 2 The parameters of the operation of the user moving from position A to position B are determined according to the parameters of the user operation (for example, in Figure 2 A curve from position A to position B is drawn on the canvas 201 as shown. The above process may be executed by the CPU in the electronic device 100 .
[0078] After the application main thread 306 determines the view data to be displayed, it can generate a task according to the view data (for example, it can use the event in the GLSurfaceView object). The task generated by the application main thread 306 can describe the coordinates, size, color and other information of the view.
[0079] like Figure 3B The figure is a schematic diagram of the application main thread 306 and the rendering thread 307 processing tasks. The application main thread 306 can add the generated tasks to the task queue (for example, the application main thread 306 can call the queueEvent method in the GLThread object to add the tasks to the task queue).
[0080] The rendering thread 307 can take tasks from the task queue, and then run the tasks (for example, the event.run method in the GLThread object can be used) to generate rendering instructions that the GPU can recognize. After running the task, the rendering thread 307 can send the rendering instructions to the GPU, and the GPU renders the view to be displayed. The GPU's rendering of the view may include but is not limited to coloring the view, adding lighting effects, shadow effects, texture effects, etc. The rendered view is composed of multiple pixels and can be displayed on a display screen. The rendering thread 307 can then send the rendered view to the layer synthesis module 303.
[0081] by Figure 3B Taking the task queue shown as an example, the tasks 1, 2, 3, and 4 in the task queue can be stored in the task queue by the application main thread 306 in the order of task 1, task 2, task 3, and task 4. The rendering thread 307 can process the tasks in the order of task 1, task 2, task 3, and task 4. When a task is taken out from the task queue by the rendering thread 307, the task queue no longer has the task. For example, when there are tasks 1, 2, 3, and 4 in the task queue, after the rendering thread 307 takes out task 1 from the task queue, the task queue only contains tasks 2, 3, and 4.
[0082] Among them, the application main thread 306 adding tasks to the task queue and the rendering thread 307 taking tasks from the tasks are mutually exclusive events. That is to say, the electronic device 100 only allows one of the application main thread 306 and the rendering thread 307 to operate the task queue at the same time. The electronic device 100 can add a mutex lock to the application main thread 306 during the period when the application main thread 306 adds tasks to the task queue, and the electronic device 100 can also add a mutex lock to the rendering thread 307 during the period when the rendering thread 307 takes tasks from the task queue. When the application main thread 306 holds the mutex lock, the application main thread 306 can perform the operation of adding tasks to the task queue, while the rendering thread 307 cannot take tasks from the task queue. When the rendering thread 307 holds the mutex lock, the rendering thread 307 can take tasks from the task queue, while the application main thread 306 cannot add tasks to the task queue. That is to say, only the thread holding the mutex lock can operate the task queue.
[0083] When there are tasks in the task queue and the application main thread 306 does not hold a mutex lock, the rendering thread 307 will take the task out of the task queue for processing. As long as the application main thread 306 does not hold a mutex lock and there are tasks in the task queue, the rendering thread 307 will always take tasks from the task queue and run them. Among them, the rendering thread 307 is the CPU in the electronic device 100 that runs the task. For the introduction of the above-mentioned GLSurfaceView object and GLThread object, please refer to the subsequent embodiments, which will not be expanded here.
[0084] S306: The layer synthesis module 303 synthesizes the views into layers to generate a picture to be displayed.
[0085] S307 , the layer synthesis module 303 sends the image to be displayed to the display driver 304 .
[0086] S308 , the display driver 304 sends the image to be displayed.
[0087] S309, the display screen 194 displays the picture.
[0088] In some embodiments, the layer synthesis module 303 may include a surface synthesizer (surface flinger) and / or a hardware synthesizer (hardware composer, HWC). The surface synthesizer and the hardware synthesizer can synthesize the rendered view into a picture to be displayed. The layer synthesis module 303 can send the synthesized picture to be displayed to the display driver in the electronic device. The display driver 304 converts the picture to be displayed from a digital signal into an electrical signal and sends it to the display circuit in the display screen 194, thereby driving the display screen 194 to display the picture to be displayed.
[0089] The application main thread 306 adds tasks to the task queue, the layer synthesis module 303 synthesizes layers, and the display driver 304 sends images to the display under the control of the vertical synchronization (VSYNC) signal. The method of VSYNC controlling the operation of the above components can refer to the introduction of the subsequent embodiments, which will not be expanded here.
[0090] Figure 3A The components included in the electronic device 100 are only examples. The electronic device 100 may have Figure 3A More or fewer components may be shown, two or more components may be combined, or there may be a different configuration of components. Figure 3A The steps shown when the various components display the pictures can be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.
[0091] The above-mentioned method of displaying a picture by the electronic device 100 is only an example and is not intended to be a limitation of the embodiments of the present application. In some embodiments, the electronic device 100 may include a plurality of Figure 3A For example, in step S305, the rendering thread 307 may directly send the rendered view to the display driver for display.
[0092] The method is not limited to the drawing application 302, but may also be the main thread and rendering thread of other applications. The rendering method of other applications in the electronic device 100 may refer to the rendering method of the drawing application 302, which will not be described in detail here.
[0093] Figure 4A A schematic diagram of a display process of the electronic device 100 displaying an image with strong chirality is shown.
[0094] like Figure 4A As shown, Figure 4A The main thread of the application adds tasks to the task queue and Figure 3A The rendering thread takes the task from the task queue, the rendering thread processes the task to generate a rendering instruction, and the GPU renders the view according to the rendering instruction. Figure 3A The layer synthesis module synthesizes the rendered view layer with Figure 3A The display driver sends the image to be displayed to the display driver. Figure 3A Corresponding to step S308, Figure 4A For an introduction to the above steps, please refer to Figure 3A The corresponding steps in the description will not be repeated here.
[0095] Figure 4A The VSYNC1 signal, VSYNC2 signal, VSYNC3 signal, VSYNC4 signal and VSYNC5 signal are VSYNC signals sent by the electronic device 100 at different times. The time interval between two adjacent VSYNC signals sent by the electronic device 100 may be related to the refresh rate of the display screen. Taking a display screen with a refresh rate of 60Hz as an example, the time for the display screen to refresh the screen once may be 1000ms / 60≈16.6ms. The time interval between two adjacent VSYNC signals sent by the electronic device 100 may also be 16.6ms. In other words, the time interval between the electronic device 100 sending the VSYNC1 signal and the electronic device sending the VSYNC2 signal may be 16.6ms, and the time interval between the electronic device 100 sending the VSYNC2 signal and the electronic device sending the VSYNC3 signal may also be 16.6ms.
[0096] like Figure 4AAs shown, after the electronic device 100 sends a VSYNC1 signal, in response to the VSYNC1 signal, the application main thread 306 can add task 1 to the task queue. During the a1 time period when the application main thread 306 adds task 1, the application main thread 306 holds the mutex lock, and the rendering thread 307 cannot perform the operation of taking tasks from the task queue during the a1 time period. The mutex lock can be released after the application main thread 306 finishes adding task 1. During the a2 time period, the rendering thread 307 obtains the mutex lock and takes out task 1 from the task queue. The rendering thread 307 releases the mutex lock after taking out task 1. Furthermore, the rendering thread 307 can process task 1 and send the rendering instructions corresponding to task 1 to the GPU, and the GPU can render the view corresponding to task 1. Finally, the rendering thread 307 can send the rendered view to the layer synthesis module.
[0097] After the electronic device 100 sends a VSYNC2 signal, in response to the VSYNC2 signal, the application main thread 306 adds task 2 to the task queue. During the a3 time period when the application main thread 306 adds task 2 to the task queue, the application main thread holds a mutex lock. After the application main thread 306 adds the task to the task queue, the mutex lock can be released. Then the rendering thread 307 obtains the mutex lock and takes out task 2 from the task queue. During the a4 time period when the rendering thread 307 takes out task 2 from the task queue, the rendering thread 307 holds the mutex lock. The rendering thread 307 can process task 2 and then send the rendering instruction corresponding to task 2 to the GPU, and the GPU can render the view corresponding to task 2 according to the rendering instruction. Then the rendering thread 307 can receive the view rendered by the GPU and send the rendered view to the layer synthesis module. When the electronic device 100 sends a VSYNC2 signal, the layer synthesis module can also process the view corresponding to task 1 in response to the VSYNC2 signal to generate a picture 1 to be displayed, and then the layer synthesis module 303 can send the picture 1 to be displayed to the display driver.
[0098] After the electronic device 100 sends the VSYNC3 signal, the application main thread 306 can obtain the view layout, measurement view, etc. in the user interface in response to the VSYNC3 signal, and then generate task 3. The application main thread 306 can add task 3 to the task queue (a5 time period), and the rendering thread 307 can take out task 3 from the task queue (a6 time period) and process task 3. The layer synthesis module 303 can synthesize the view corresponding to task 2 in response to the VSYNC3 signal to generate the screen 2 to be displayed. At the same time, the display driver 304 can send the screen 1 to be displayed in response to the VSYNC3 signal, so that the screen 1 is displayed on the display screen.
[0099] After the electronic device 100 sends the VSYNC4 signal, the layer synthesis module 303 can receive and process the view corresponding to task 3 in response to the VSYNC4 signal to generate the picture 3 to be displayed. The display driver 304 can receive and send the picture 2 to be displayed in response to the VSYNC4 signal. In this way, the picture displayed on the display screen changes from picture 1 to picture 2.
[0100] Figure 4A The method in which the electronic device 100 controls each module or component through the VSYNC signal is only used as an example. In some embodiments, the VSYNC signal for controlling the application main thread and the rendering thread, the VSYNC signal for controlling the layer synthesis module, and the VSYNC signal for controlling the display driver in the electronic device 100 can be controlled by different types of VSYNC signals. Among them, the different types of VSYNC signals sent by the electronic device 100 can be the same or different. The embodiments of the present application do not limit the type of VSYNC signal sent by the electronic device 100.
[0101] In some embodiments, because the rendering thread takes too long to process a task, the electronic device may have poor hand tracking. Figure 4B A display flow chart of the electronic device 100 displaying an image with poor chirality is shown.
[0102] like Figure 4B As shown, after the electronic device 100 sends the VSYNC1 signal, the application main thread can receive and respond to the VSYNC1 signal to add task 1 to the task queue. During the b1 time period when the application main thread 306 adds task 1, the application main thread 306 holds the mutex lock. When the application main thread 306 releases the mutex lock after adding task 1, the rendering thread 307 can acquire the mutex lock and take out task 1 from the task queue (b2 time period). After taking out task 1, the rendering thread 307 will release the mutex lock and process task 1.
[0103] After the electronic device 100 sends out the VSYNC2 signal, the application main thread 306 can receive and acquire the mutex lock in response to the VSYNC2 signal, and then the application main thread 306 can add task 2 to the task queue. The application main thread holds the mutex lock during the b3 time period corresponding to the application main thread 306 adding task 2 to the task queue. After the rendering thread 307 processes task 1, it will determine whether there are unprocessed tasks in the task queue. Since the application main thread 306 has added task 2 to the task queue before the rendering thread 307 processes task 1, the rendering thread 307 will continue to take out task 2 from the task queue for processing, and will not send the rendering instruction corresponding to task 1 to the GPU.
[0104] After the electronic device 100 sends out the VSYNC3 signal, the application main thread 306 can receive and add task 3 to the task queue in response to the VSYNC3 signal. The application main thread 306 holds the mutex lock during the b5 time period corresponding to adding task 3. After processing task 2, the rendering thread 307 will also determine whether there are unfinished tasks in the task queue. At this time, there is task 3 newly added by the application main thread in the task queue. The rendering thread will continue to take out task 3 from the task queue and process task 3, and will not send the rendering instructions corresponding to tasks 1 and 2 to the GPU. Since the GPU does not receive the rendering instructions, it cannot render the view in time, and the layer synthesis module cannot generate the screen to be displayed in time. This will cause the display screen to not update the new screen, which means that the chirality is poor.
[0105] Depend on Figure 4B It can be seen that when the application main thread 306 adds a new task to the task queue before the rendering thread 307 finishes processing the tasks in the task queue, it will cause the accumulation of tasks in the task queue, thereby causing the rendering thread to continue processing tasks without sending the rendering instructions after processing the tasks to the GPU for rendering. In order to solve this problem, the rendering method provided by the embodiment of the present application is described below.
[0106] Figure 5 It is a flowchart of the rendering method provided in an embodiment of the present application.
[0107] like Figure 5 As shown, the rendering method may include but is not limited to the following steps:
[0108] S501, the electronic device displays a user interface of a first application on a display screen;
[0109] In the embodiment of the present application, the drawing application 302 may also be referred to as a first application. The electronic device 100 may run the drawing application 302 and display a user interface of the drawing application 302.
[0110] S502, the electronic device takes out a first task from a task queue, where the task in the task queue is added by a first application;
[0111] The user interface of the drawing application 302 may include a canvas. The electronic device 100 may receive a user operation performed by a user on the canvas. The user operation may be used to draw a first content on the canvas. For example, the first content may be one or more curve segments, a pattern of a specific shape, and the like. The event response module 301 in the electronic device 100 may obtain user operation parameters such as the position where the user touches the display screen and a timestamp through a touch sensor, and then the event response module 301 may send the above parameters to the drawing application 302. The application main thread in the drawing application 302 may generate one or more tasks according to the above parameters of the user operation. The one or more tasks may include a first task, wherein the first task is used to describe the drawing information of the first content, for example, the drawing information may include the position where the first content is displayed on the canvas, the size of the first content, the shape and color of the first content, and the like. Then, the application main thread may add the first task to the task queue.
[0112] S503, the electronic device runs the first task obtained to obtain a graphics processor GPU rendering instruction, and prohibits the first application from adding tasks to the task queue during the running period of the first task; during the non-running period of the first task, the first application is allowed to add tasks to the task queue when a vertical synchronization VSYNC signal arrives;
[0113] The rendering thread in the electronic device 100 can take out the first task from the task queue and run it, and obtain the GPU rendering instruction corresponding to the first task. The GPU rendering instruction is used to instruct the GPU to render the view containing the first content. During the period when the rendering thread in the electronic device 100 runs the first task, the electronic device can add a mutex lock to the rendering thread. In this way, the main thread of the application cannot add a new task to the task queue during the period when the rendering thread runs the first task.
[0114] When the user slides on the display screen at a faster speed, more content is drawn on the canvas per unit time, and the rendering thread spends more time running tasks, which makes it easier to produce poor chirality problems. In some embodiments, the electronic device 100 can first detect the sliding speed of the user's sliding operation on the display screen before adding a mutex lock during the period when the rendering thread runs tasks. The electronic device can lock the rendering thread only when the sliding speed of the user's sliding operation on the display screen exceeds a first rate (that is, prohibiting the main thread of the application in the first application from adding tasks to the task queue when the rendering thread runs the first task). In other words, the electronic device 100 can prohibit the main thread of the application from adding tasks to the task queue while the rendering thread runs tasks only when the user's operation is more likely to cause chirality problems.
[0115] When the refresh rate of the electronic device display is fast, the interval between two adjacent VSYNC signals is short, and it is more likely to have poor chirality. In some embodiments, the electronic device 100 can prohibit the application main thread from adding tasks to the task queue while the rendering thread is running tasks only when the refresh rate of the display exceeds the first refresh rate.
[0116] S504: When the task queue is empty, the electronic device performs rendering based on the rendering instructions generated by running the tasks in the task queue.
[0117] When the task queue is empty, the rendering thread can send the GPU rendering instruction obtained after processing the task to the GPU, and the GPU renders the view containing the first content. The view rendered by the GPU can be sent to the rendering thread, and the rendering thread sends the rendered view to the layer synthesis module to synthesize the view into a picture to be displayed. Finally, the layer synthesis module can send the picture to be displayed to the display driver, and the display driver sends the picture to be displayed. In this way, the first content drawn by the user can be displayed in the user interface of the drawing application 302.
[0118] In some embodiments, the electronic device 100 may prohibit the application main thread from adding tasks to the task queue during the period when the rendering thread executes each task.
[0119] like Figure 6 As shown, after the electronic device 100 sends the VSYNC1 signal, the application main thread 306 receives and responds to the VSYNC1 signal to add task 1 to the task queue. During the c1 time period when the application main thread 306 adds task 1, the application main thread 306 holds the mutex lock. The rendering thread 307 acquires the mutex lock and takes task 1 out of the task queue. Then, the rendering thread 307 runs task 1. During the c3 time period when the rendering thread 307 runs task 1, the rendering thread 307 still holds the mutex lock.
[0120] After the electronic device 100 sends out the VSYNC2 signal, the rendering thread 307 is still running Task 1 while holding the mutex lock. In this way, although the main thread of the application receives the VSYNC2 signal, it cannot obtain the mutex lock, and cannot add new tasks to the task queue in response to the VSYNC2 signal. This avoids the main thread of the application 306 adding new tasks to the task queue when the rendering thread 307 has not yet finished running Task 1, resulting in a backlog of tasks in the task queue. The rendering thread will hold the mutex lock during the c3 time period when processing Task 1, and the rendering thread 307 will release the mutex lock only after running Task 1. Since the event queue is empty, the rendering thread 307 can send the rendering instruction corresponding to Task 1 to the GPU, instructing the GPU to render the view corresponding to Task 1 according to the rendering instruction.
[0121] After the electronic device 100 sends the VSYNC3 signal, the application main thread 306 can obtain the mutex lock in response to the VSYNC3 signal and add task 2 to the task queue. The application main thread 306 holds the mutex lock during the c4 time period when the application main thread 306 adds task 2. Then the rendering thread takes out task 2 from the task queue, and the rendering thread holds the mutex lock during the c5 time period when the rendering thread takes task 2. When the rendering thread takes out task 2, it will process task 2. During the c6 time period when the rendering thread 307 processes task 2, the rendering thread 307 holds the mutex lock. In other words, no matter how long it takes for the rendering thread to run the task, the electronic device will lock the rendering thread when the rendering thread runs the task, so that the application main thread cannot add a new task to the task queue when the rendering thread runs the task. After the rendering thread 307 finishes processing task 2, it will send the rendering instruction corresponding to task 2 to the GPU, and the GPU will render according to the rendering instruction. On the other hand, the layer synthesis module receives and responds to the VSYNC3 signal to synthesize the view corresponding to task 1 after GPU rendering to generate the picture 1 to be displayed, and then the layer synthesis module sends the picture 1 to be displayed to the display driver.
[0122] After the electronic device 100 sends the VSYNC4 signal, the layer synthesis module will receive and synthesize the view corresponding to the task 2 rendered by the GPU in response to the VSYNC4 signal to generate the picture 2 to be displayed, and then the layer synthesis module will send the picture 2 to be displayed to the display driver. In addition, the display driver will send the picture 1 to be displayed in response to the VSYNC4 signal, so that the display screen displays the picture 1.
[0123] After the electronic device 100 sends out the VSYNC5 signal, the display driver will receive and send the picture 2 to be displayed in response to the VSYNC5 signal, and the picture displayed on the display screen will change from picture 1 to picture 2.
[0124] The method for the electronic device 100 to process other tasks while holding a mutex lock when running each task can be referred to Figure 6 The method in which the electronic device displays the screens corresponding to task 1 and task 2 is not described in detail here.
[0125] That is to say, the rendering thread 307 can hold a mutex lock when running each task. In this way, the application main thread 306 cannot add new tasks to the task queue while the rendering thread 307 is running tasks. The rendering thread 307 can complete the tasks in the task queue in time, and then send the rendering instructions obtained by running the tasks to the GPU, and the GPU performs rendering according to the rendering instructions. This avoids the problem of poor chirality of electronic devices due to a backlog of tasks in the task queue.
[0126] In some embodiments, the electronic device may prohibit the application main thread from adding tasks to the task queue while the rendering thread is running the task only when it is determined that the running time of the task before the first task exceeds the first time duration. For example, the electronic device may calculate whether the sum of the running time of one or more tasks before the first task and after the last GPU rendering exceeds the first time duration, and then prohibit the application main thread from adding tasks to the task queue while the rendering thread is running the first task.
[0127] In some embodiments, the electronic device may further prohibit the application main thread from adding tasks to the task queue while the rendering thread is running the first task when the running time of the previous task of the first task exceeds a first time length.
[0128] like Figure 7 As shown, after the electronic device 100 sends the VSYNC1 signal, the application main thread 306 can receive and respond to the VSYNC1 signal to add task 1 to the task queue. During the d1 time period when the application main thread 306 adds task 1, the application main thread 306 holds the mutex lock. Then the rendering thread 307 takes task 1 from the task queue, and during the d2 time period when the rendering thread 307 takes task 1, the rendering thread 307 holds the mutex lock. The rendering thread 307 will process task 1, and during the d3 time period when the rendering thread 307 processes task 1, the rendering thread 307 may not hold the mutex lock.
[0129] After the electronic device 100 sends a VSYNC2 signal, since the rendering thread 307 does not hold the mutex lock, the application main thread 306 can receive and acquire the mutex lock in response to the VSYNC2 signal. Then the application main thread 306 can add task 2 (d4 time period) to the task queue. When the rendering thread finishes processing task 1, the processing time of task 1 (i.e., the time length of the d3 time period) can be recorded. Since the processing time of task 1 exceeds the interval between two adjacent VSYNC signals, the rendering thread 307 can hold the mutex lock when processing the next task. After the rendering thread 307 finishes running task 1, there is task 2 in the task queue that is added to the task queue by the application main thread 306 after the VSYNC2 signal. The rendering thread will take out task 2 from the task queue, and the rendering thread holds the mutex lock during the d5 time period when the rendering thread 307 takes task 2. Since the processing time of Task 1 exceeds the interval between two adjacent VSYNC signals, the rendering thread 307 will first acquire the mutex lock and then run Task 2. Thus, during the d6 time period when the rendering thread 307 runs Task 2, the application main thread 306 cannot add new tasks to the task queue. Here, the time interval between two adjacent VSYNC signals sent by the electronic device 100 can be referred to as the first time period, Task 2 can be referred to as the first task, and Task 1 is the previous task of the first task.
[0130] When the electronic device 100 sends a VSYNC3 signal, since the rendering thread 307 is still running Task 2 while holding the mutex lock, the application main thread 306 cannot add tasks to the task queue in response to the VSYNC3 signal. Therefore, there are no other tasks in the task queue after the rendering thread 307 finishes processing Task 2. The rendering thread 307 can send the rendering instructions obtained by processing Task 1 and Task 2 to the GPU, and the GPU can render the views corresponding to Task 1 and Task 2 according to the rendering instructions, and the rendered views are sent to the layer synthesis module. After the rendering thread 307 completes Task 2, the rendering thread 307 can also record the length of time for processing Task 2 (i.e., the length of the d6 time period).
[0131] When the electronic device 100 sends a VSYNC4 signal, the application main thread 306 can receive and respond to the VSYNC4 signal to add task 3 to the task queue (d7 time period), and the rendering thread 307 can obtain the mutex after the application main thread 306 releases the mutex, and then take out task 3 from the task queue. Since the length of time that the rendering thread 307 runs task 2 (i.e., the length of the d6 time period) does not exceed the interval between two adjacent VSYNC signals, the rendering thread 307 will not hold the mutex (d9 time period) when running the next task of task 2 in the task queue (i.e., task 3), that is, the rendering thread 307 allows the application main thread to add tasks to the task queue during the period when the rendering thread 307 runs task 3. On the other hand, the layer synthesis module can generate the view synthesis layer corresponding to task 1 and task 2 to be displayed in response to the VSYNC4 signal, and then the layer synthesis module can send the display 1 to be displayed to the display driver. After the rendering thread 307 completes the running of task 3, the rendering thread 307 can also record the length of time for processing task 3 (i.e., the length of the d9 time period). The rendering thread 307 can determine that the length of the time period d9 does not exceed the time interval between two adjacent VSYNC signals, so the rendering thread 307 does not hold the mutex lock when processing the next task of Task 3 in the task queue.
[0132] When the electronic device 100 sends the VSYNC5 signal, the display driver can send the picture 1 to be displayed, and then the picture 1 can be displayed on the display screen.
[0133] In this embodiment, the method for the electronic device to process other tasks can refer to Figure 7 The method by which the electronic device processes Task 1, Task 2, and Task 3 is not described in detail here.
[0134] When the rendering thread runs a task that exceeds the time interval between two adjacent VSYNC signals, the application main thread adds a new task to the task queue, which will cause the task to accumulate in the task queue. Therefore, the rendering thread 307 can run the next task of the task whose running time exceeds the time interval between two adjacent VSYNC signals while holding the mutex lock. In this way, it can be avoided that when the rendering thread has failed to complete the task between two adjacent VSYNC signals, the application main thread adds a new task to the task queue, resulting in a more serious accumulation of tasks in the task queue.
[0135] Figure 8 A flowchart for implementing the rendering method provided in an embodiment of the present application is shown.
[0136] like Figure 8 As shown, after the drawing application 302 in the electronic device is run, a GLSurfaceView object is created. The GLSurfaceView object can be used to create a rendering environment, and then implement image rendering, display and other functions in the rendering environment.
[0137] The GLsurfaceView object will first call the setRenderer method. In the setRenderer method, the GLsurfaceView object will set the renderer. For example, GlsurfaceView can set the renderer version, rendering mode, etc. After GlsurfaceView sets the renderer, it will generate a GLThread object and start the rendering thread.
[0138] After the rendering thread is started, the GLThread object will call the run method. The run method will first set the thread name of the rendering thread, and then call the guardedRun method in the run method. The rendering thread can call the remove method of the mEventQueue object in the guardedRun method to remove the task (event) from the task queue, and then call the event.run method to run the task and generate rendering instructions that the GPU can recognize. When the rendering thread determines that the task queue is empty in the guardedRun method, it will further call the onDrawFrame method in the guardedRun method to send the rendering instructions obtained after running the task (event) to the GPU, and the GPU will render the view according to the rendering instructions. Finally, the rendering thread can call the eglSwapBuffer method in the guardedRun method to synthesize and display the rendered image so that it is displayed on the display.
[0139] Among them, the task processed by the rendering thread can be added by the drawing application 302. The drawing application 302 can receive the VSYNC signal sent by the electronic device. In response to the VSYNC signal, the drawing application can call the doFrame function to process the user input in the application, the application animation, and the measurement of the controls in the interface to obtain the size, layout and other data of the view. The main thread of the application can generate a task (event) based on the above data. Furthermore, the application can call the queueEvent method of the GLThread object, and the add method of the mEventQueue object will be further called in the queueEvent method to add the task (event) to the task queue.
[0140] In some embodiments, a tag value may be set in the guardedRun method of the GLThread object. False represents 0, True represents 1, and the initial value of the tag may be False. The rendering thread may record the running time of the task each time it runs a task (event). When the running time of the task exceeds the time interval between two adjacent VSYNC signals, the tag may be assigned a value of True. When the rendering thread takes the next task from the task queue, if the value of the tag is True, the task is run with a mutex lock, otherwise the task is run without a mutex lock.
[0141] Combination Figure 7 In the embodiment shown, the rendering thread is started after the drawing application 302 is run in the electronic device 100. After the electronic device 100 sends a VSYNC1 signal, the application main thread calls the doFrame function to generate task 1, and then adds task 1 to the task queue by calling the queueEvent method of the GLThread object. The rendering thread can call the remove method of the mEventQueue object to remove task 1 from the task queue, and then call the event.run method to run task 1. In the initial case, the Tag value is False, and the rendering thread does not hold the mutex lock during the d3 time period when executing the event.run method to run task 1.
[0142] After the electronic device 100 sends out the VSYNC2 signal, the main thread of the application can continue to call the doFrame function to generate Task 2 and add Task 2 to the task queue. After the rendering thread runs the event.run method corresponding to Task 1, it can record the running time of the event.run. Since the running time of the event.run corresponding to Task 1 exceeds the time interval between two adjacent VSYNC signals, the rendering thread can assign the Tag value to True. After the rendering thread takes Task 2 out of the task queue, since the Tag value is True, the rendering thread will lock when calling the event.run method to run Task 2.
[0143] When the electronic device 100 sends a VSYNC3 signal, the main application thread can call the doFrame function to generate Task 3. After running Task 2, the rendering thread can record the running time of Task 2. Since the running time of Task 2 does not exceed the time interval between two adjacent VSYNC signals, the rendering thread can assign the Tag value to False. In this way, when the rendering thread calls the event.run method to run Task 3, it will not be locked.
[0144] The rendering method when the electronic device 100 processes more tasks can refer to the rendering methods of the above-mentioned tasks 1, 2 and 3, which will not be repeated here.
[0145] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
[0146] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.
[0147] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.
[0148] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.
Claims
1. A rendering method, the method being applied to an electronic device, characterized in that: include: The electronic device displays a user interface of a first application on a display screen, wherein the first application includes an application main thread and a rendering thread, wherein the application main thread is used to add tasks to a task queue, and the rendering thread is used to retrieve and run tasks from the task queue; The electronic device takes out a first task from the task queue through the rendering thread, where the task in the task queue is added by the application main thread of the first application program; The electronic device obtains a graphics processor GPU rendering instruction by executing the first task obtained by the rendering thread, and the electronic device locks the rendering thread during the period when the rendering thread executes the first task, and the main application thread of the first application is prohibited from adding tasks to the task queue during the period when the rendering thread is locked; During the non-running period of the first task, the application main thread of the first application is allowed to add tasks to the task queue when a vertical synchronization VSYNC signal arrives; When the task queue is empty, the electronic device performs rendering through the rendering thread based on the rendering instructions generated by running the tasks in the task queue.
2. The method according to claim 1, characterized in that The running time of the task before the first task exceeds the first time duration.
3. The method according to claim 2, characterized in that The task before the first task is a previous task of the first task.
4. The method according to claim 2, characterized in that: The first duration is equal to the time interval between two adjacent vertical synchronization VSYNC signals.
5. The method according to any one of claims 1 to 4, characterized in that The user interface of the first application includes a canvas; the tasks in the task queue are created by the electronic device based on detected user input when displaying the user interface of the first application, the user input is used to draw first content on the canvas, and the tasks are used to describe drawing information of the first content; The drawing information includes one or more of the following: the position, size, shape, and color of the first content.
6. The method according to claim 5, characterized in that After the electronic device performs rendering based on the rendering instruction generated by running the task in the task queue, the method further includes: the electronic device additionally displays the first content on the canvas.
7. The method according to claim 5, characterized in that The user input includes a sliding input, and the first content is determined according to a sliding trajectory of the sliding input; before the electronic device locks the rendering thread during the execution of the first task by the rendering thread, the method further includes: the electronic device detects that the sliding speed of the sliding input exceeds a first rate.
8. The method according to claim 5, characterized in that Before the electronic device locks the rendering thread during the period when the rendering thread executes the first task, the method further includes: the electronic device detects that a refresh rate of a display screen exceeds a first refresh rate.
9. An electronic device, characterized in that: The electronic device comprises: a display screen, a memory, and a processor coupled to the memory; the display screen is used to display an interface, the memory stores a computer program, and when the processor executes the computer program, the electronic device implements the method as described in any one of claims 1 to 8.
10. A computer-readable storage medium comprising computer instructions, characterized in that: When the computer instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 8.
