Image processing method and electronic device

Abstract

Embodiments of the present application relate to the technical field of image processing, and provide an image processing method and an electronic device. The method can effectively reduce a delay in displaying on a display screen response content corresponding to an input operation by a user, thereby improving responsiveness to manual operations. The method comprises: after a first Vsync signal is generated, receiving a first rendering instruction issued by a first application; on the basis of the first rendering instruction, acquiring a first rendering result; on the basis of the first rendering result, performing synthesis processing to acquire a first image for transmission and display, the first image for transmission and display corresponding to an N-th image frame; and, after a second Vsync signal is generated, controlling a display screen to display the N-th image frame on the basis of the first image for transmission and display, the second Vsync signal being generated after the first Vsync signal, and there being no Vsync signal generated after the first rendering result is acquired and before the synthesis processing is performed on the basis of the first rendering result.

Description

Image processing method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on December 21, 2023, with application number 202311780834.4 and invention name “A Method and Electronic Device for Image Processing”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of image processing technology, and in particular to an image processing method and electronic device. Background Art

[0003] Currently, electronic devices can perform processing such as rendering and synthesis on images before displaying them.

[0004] For example, after the application program issues a rendering instruction, the electronic device can perform rendering processing according to the rendering instruction and obtain the corresponding rendering result. The electronic device can also synthesize the rendering result when the next synchronization signal (such as a Vsync signal) arrives, thereby obtaining the display image of the frame image. The display screen of the electronic device can display according to the display image when the next Vsync signal arrives. In this way, the frame image can be displayed on the display screen.

[0005] As a result, there's a delay of multiple Vsync signal cycles between when an application issues a rendering command and when the corresponding frame of image appears on the display. Some rendering commands issued by applications may include responses to user inputs, forcing users to wait a long time before seeing responses on the display. This can lead to poor hand tracking.

[0006] Summary of the Invention

[0007] The present application provides an image processing method and electronic device, which can effectively reduce the delay in displaying corresponding response content on a display screen after a user input operation, thereby improving hand tracking.

[0008] To achieve the above technical objectives, this application adopts the following technical solutions:

[0009] In a first aspect, an image processing method is provided, which is applied to an electronic device, wherein the electronic device is provided with a display screen and a first application is also installed in the electronic device. The method comprises: after generating a first synchronous Vsync signal, receiving a first rendering instruction issued by the first application, wherein the first rendering instruction is used to instruct the electronic device to perform rendering processing on the Nth frame image. According to the first rendering instruction, a first rendering result is obtained. Synthesis processing is performed according to the first rendering result to obtain a first display image. The first display image corresponds to the Nth frame image. After generating a second Vsync signal, the display screen is controlled to display the Nth frame image according to the first display image. The second Vsync signal is generated after the first Vsync signal. After obtaining the first rendering result, before performing synthesis processing according to the first rendering result, no Vsync signal is generated.

[0010] Based on this, after obtaining the rendering result, the electronic device can directly perform synthesis processing based on the rendering result without waiting for the next Vsync signal to arrive. This saves the time of at least one Vsync cycle. In this way, after receiving the rendering instruction issued by the first application, the electronic device can display the corresponding interface on the display screen more quickly, thereby significantly improving the tracking performance.

[0011] Optionally, before generating the first synchronous Vsync signal, the method further includes: receiving a first operation, where the first operation is an operation on the first interface, where the first interface is a display interface of the first application. In this way, the first rendering instruction indicates that the rendered content may include a response to the first operation.

[0012] Optionally, the method further includes: generating first operation information according to the first operation, wherein the first operation information indicates an operation type of the first operation and location information of the first operation, and sending the first operation information to the first application.

[0013] Optionally, a first buffer queue is configured in the electronic device, and the first buffer queue corresponds to the first application. After obtaining the first rendering result according to the first rendering instruction, the method further includes: sending first information corresponding to the first rendering result to the first buffer queue. The first information includes any one of the following: image information of the first rendering result. An identifier of the buffer buffer storing the first rendering result. A first file identifier, which indicates the storage location of the first rendering result in the memory of the electronic device. Thus, a logical implementation of backward transmission of rendering results is provided. For example, the logical implementation can be based on the buffer queue configured in the electronic device. After obtaining the rendering result, the corresponding buffer can be queued to the buffer queue. Therefore, when the rendering result needs to be synthesized later, the corresponding buffer can be dequeued.

[0014] Optionally, after sending the first information corresponding to the first rendering result to the first buffer queue, the method further includes: configuring a first identification field corresponding to the first buffer queue to a second value. The second value of the first identification field indicates that new information is queued in the first buffer queue.

[0015] Optionally, before the first information corresponding to the first rendering result is sent to the first buffer queue, the first identification field is configured to have a first value. The first identification field having the first value indicates that the available information in the first buffer queue is empty.

[0016] This application provides a solution for managing buffer queues. For example, each buffer queue can be configured with a corresponding identification field. Different values ​​of this identification field indicate whether there are available buffers in the buffer queue, or the number of available buffers in the buffer queue. Thus, based on the changes in the values ​​of different identification fields, it is possible to monitor the entry of new buffers into each buffer queue.

[0017] Optionally, performing synthesis processing based on the first rendering result to obtain a first image for display includes: obtaining the first information from the first buffer queue, and performing synthesis processing on the first rendering result indicated by the first information to obtain the first image for display.

[0018] Optionally, the electronic device is configured with an application whitelist, the application whitelist including at least one application information, where the application information of different applications is different. Before obtaining the first information from the first buffer queue, the method further includes: determining that the application information corresponding to all buffer queues whose identification fields have the second value is included in the application whitelist.

[0019] Optionally, determining that application information corresponding to all buffer queues whose identification fields are the second value is included in the application whitelist includes: determining that application information corresponding to the first buffer queue is included in the application whitelist.

[0020] Optionally, the application information includes a package name of the application. The information of the first buffer queue includes: the package name of the first application corresponding to the first buffer queue. The determining that the information of the first buffer queue is included in the application whitelist includes: determining that the package name of the first application corresponding to the first buffer queue is included in the application whitelist.

[0021] Optionally, the method further includes: determining, according to the name of the first buffer queue, a package name of the first application corresponding to the first buffer queue.

[0022] This provides a mechanism for triggering early synthesis processing. In this example, the electronic device can be pre-configured with a whitelist of applications that require early synthesis processing. In this way, if a new buffer is queued in the buffer queue of an application on the whitelist, the buffer can be dequeued in advance for synthesis and display. This allows the rendering instructions of the applications on the whitelist to be reflected on the display screen more quickly.

[0023] Optionally, a second application is also installed in the electronic device. The method also includes: after generating the third Vsync signal, receiving a second rendering instruction issued by the second application, the second rendering instruction is used to instruct the electronic device to perform at least partial rendering processing of the M-th frame image. According to the second rendering instruction, a second rendering result is obtained. After generating the fourth Vsync signal, synthesis processing is performed according to the second rendering result to obtain a second display image. The second display image corresponds to the M-th frame image. After generating the fifth Vsync signal, the display screen is controlled to display the M-th frame image according to the second display image. The fourth Vsync signal is generated after obtaining the second rendering result.

[0024] Optionally, after obtaining the second rendering result according to the second rendering instruction, the method further includes: determining that the application information of the second application is not included in the application whitelist.

[0025] Therefore, for applications not in the application whitelist, the electronic device may not trigger the pre-synthesized XSync solution, thereby ensuring that within the same Vsync cycle, the electronic device (such as the SF module of the electronic device) does not trigger multiple synthesis processes.

[0026] Optionally, after obtaining the second rendering result, the method further includes: storing second information of the second rendering result in a second buffer queue, where the second buffer queue is a buffer queue corresponding to the second application.

[0027] Optionally, the method further includes: after generating the third Vsync signal, receiving a third rendering instruction issued by the first application, wherein the second rendering instruction is used to instruct the electronic device to render at least a portion of the M-th frame image. According to the third rendering instruction, obtaining a third rendering result. Storing third information of the third rendering result in the first buffer queue. The third rendering result is obtained before the fourth Vsync signal is generated.

[0028] Optionally, after generating the fourth Vsync signal, performing synthesis processing based on the second rendering result to obtain a second image for display includes: after generating the fourth Vsync signal, obtaining the third rendering result based on the third information of the first buffer queue; obtaining the second rendering result based on the second information of the second buffer queue; and synthesizing the second rendering result and the third rendering result to obtain the second image for display.

[0029] Optionally, after storing the third information of the third rendering result in the first buffer queue, the method further includes: determining that application information corresponding to a buffer queue having at least one identification field with a second value is not included in the application whitelist.

[0030] Optionally, determining that application information corresponding to at least one buffer queue having an identification field with a second value is not included in the application whitelist includes: determining that application information corresponding to the second buffer queue is not included in the application whitelist.

[0031] In a second aspect, the present application further provides an electronic device comprising: a memory and one or more processors. The memory and processors are coupled. In some implementations, the electronic device may also be configured with a display screen. The memory is used to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device executes the technical solution provided in the first aspect and any possible implementation thereof. In this way, the display screen of the electronic device can display images more quickly.

[0032] In a third aspect, the present application further provides a chip system, which is applied to an electronic device. The chip system may include one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected via a circuit. The interface circuit is configured to receive a signal from the memory of the electronic device and send the signal to the processor. The signal includes computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device implements the technical solution provided in the first aspect and any possible implementation thereof.

[0033] In a fourth aspect, the present application also provides a computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the technical solution provided in the above-mentioned first aspect and any possible implementation thereof.

[0034] In a fifth aspect, the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the technical solution provided in the above-mentioned first aspect and any possible implementation thereof.

[0035] It can be understood that the solutions provided in the second to fifth aspects of the present application can respectively correspond to the first aspect and any possible design thereof, so the beneficial effects that can be achieved are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a logical diagram of interface interaction and display;

[0037] FIG2 is a logic diagram of the interaction between modules within an electronic device;

[0038] FIG3 is a logic diagram of a multi-frame image processing process;

[0039] FIG4 is a logic diagram of a multi-frame image processing process after the solution provided in an embodiment of the present application takes effect;

[0040] FIG5 is a schematic diagram of the composition of an electronic device provided in an embodiment of the present application;

[0041] FIG6 is a schematic diagram of the composition of an electronic device provided in an embodiment of the present application;

[0042] FIG7 is a schematic diagram of an inter-module interaction provided in an embodiment of the present application;

[0043] FIG8 is a schematic diagram of an inter-module interaction provided by an embodiment of the present application;

[0044] FIG9 is a schematic diagram of an inter-module interaction provided in an embodiment of the present application;

[0045] FIG10 is a schematic diagram of a flow chart of an interaction between modules provided in an embodiment of the present application;

[0046] FIG11 is a logic diagram of a multi-frame image processing process provided by an embodiment of the present application;

[0047] FIG12 is a schematic diagram of display effects of different frame images provided by an embodiment of the present application;

[0048] FIG13 is a schematic diagram of a flow chart of an interaction between modules provided in an embodiment of the present application;

[0049] FIG14 is a schematic diagram of a flow chart of an interaction between modules provided in an embodiment of the present application;

[0050] FIG15 is a schematic diagram of the composition of an electronic device provided in an embodiment of the present application;

[0051] FIG16 is a schematic diagram showing the composition of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0053] When some applications installed in an electronic device are running, images can be displayed on the display screen of the electronic device. For example, these applications may include game applications, etc. Thus, when the game application is running, the game screen can be displayed on the display screen of the electronic device.

[0054] For example, the electronic device is a mobile phone, and an application 1 is installed in the electronic device, and the application 1 is a game application.

[0055] 1 , the electronic device may display an icon of application 1 on a main interface. The user may instruct the electronic device to run the application 1 by inputting an operation 101 (such as a click operation) on the icon of application 1.

[0056] In response to the user's operation 101, the electronic device may run the application 1. Thereafter, the application 1 may instruct the electronic device to render and display the game screen by issuing a rendering instruction.

[0057] In the example shown in FIG. 1 , after the application 1 is run, the electronic device may display an interface 102 on the display screen.

[0058] For example, if the game screen of application 1 includes continuously displayed images, the interface 102 of the electronic device can sequentially display the first to fifth frames of images as shown in FIG1 , as well as subsequent images. Thus, by displaying the continuous images, the user can obtain the perception of a continuous game screen.

[0059] It should be noted that, in the embodiment of the present application, the first frame to the fifth frame can be any five consecutive frames during the game running process. The first frame does not necessarily refer to the first image displayed after the game is running.

[0060] 2 , an example of the internal processing mechanism of each frame of image (such as the first frame of image) is given.

[0061] In the example shown in FIG. 2 , the electronic device may be configured with a drawing module 21 , a buffer queue 22 , a synthesis module 23 and a display screen 24 .

[0062] The drawing module 21 is used to render images according to rendering instructions issued by an application (such as application 1).

[0063] The buffer queue 22 may be used to temporarily store rendering results obtained after rendering an image.

[0064] The synthesis module 23 can be used to obtain the rendering results from the buffer queue 22 when a pre-configured timing in the electronic device arrives, and process the rendering results (such as synthesis processing). The synthesis module 23 can also be used to send the synthesized image data to the display screen 24 so that the display screen 24 can display the image accordingly.

[0065] In the example of FIG2 , an example of interaction between various modules during the process of processing and displaying the first frame of image is provided.

[0066] As shown in Figure 2, application 1 may send a rendering instruction 201 to the rendering module 21. For example, the rendering instruction 201 may be used to instruct the rendering module 21 to render the first frame of image.

[0067] The rendering module 21 can call a component with image rendering capability (such as a graphics processing unit (GPU)) in the electronic device to render the image according to the rendering instruction 201. In this way, the rendering module 21 can obtain the rendering result 202 of the first frame image corresponding to the rendering instruction 201.

[0068] The rendering module 21 may store the rendering result 202 of the first frame image in the buffer queue 22 to prepare for the subsequent display of the first frame image.

[0069] The synthesis module 23 can retrieve the stored image from the buffer queue 22 when a pre-configured timing arrives. For example, the pre-configured timing can correspond to the arrival of a synchronization (Vsync) signal generated by the electronic device. Thus, the synthesis module 23 can retrieve the rendering result 202 from the buffer queue 22 based on the arrival of the Vsync signal.

[0070] After obtaining the rendering result 202 , the synthesis module 23 may perform synthesis processing on the rendering result 202 , thereby obtaining the display image 203 of the first frame image.

[0071] Thus, the synthesis module 23 can send the display image 203 to the display screen 24 to realize the display of the first frame image.

[0072] It is understandable that, for the processing and display process of other frame images (such as the second frame to the fifth frame, etc.), reference can be made to the processing mechanism of the first frame image shown in FIG. 2 .

[0073] It should be noted that, in some cases, the timing of the application (such as application 1) issuing the rendering instruction 201 and the timing of the display screen 24 displaying can also be controlled based on the Vsync signal. Take the Vsync signal including signals V31 to V36 as an example.

[0074] Refer to Figure 3. Application 1 can issue a rendering instruction (such as rendering instruction 201) for the first frame of image after signal V31 arrives. Correspondingly, rendering module 21 can perform rendering processing on the first frame of image after signal V31 arrives, and complete the enqueueing of rendering result 202 of the first frame of image into buffer queue 22.

[0075] Next, when signal V32 arrives, the synthesis module 23 can obtain the already queued rendering result 202 from the buffer queue 22. The synthesis module 23 can synthesize the rendering result 202 and complete the synthesis process of the first frame image before the next Vsync signal (such as signal V33) arrives. As a result, before the arrival of signal V33, the display screen 24 can receive the display image 204 of the first frame image. In this application, signal V32 can also be referred to as the first Vsync signal.

[0076] In this way, the display screen 24 can display the first frame image according to the image 204 after the signal V33 arrives.

[0077] It is understandable that each module component in the electronic device can process subsequent frame images according to the above-mentioned similar processing mechanism.

[0078] Illustratively, FIG3 also provides processing examples of the second frame image and the third frame image.

[0079] For the second frame, application 1 can issue the corresponding rendering instructions after the signal V32 arrives. Accordingly, after the signal V32 arrives, rendering module 21 can begin rendering the second frame. In this example, the rendering time of the second frame can be greater than one Vsync cycle. One Vsync cycle can be the time difference between the arrival of two adjacent Vsync signals.

[0080] In this way, when the signal V33 arrives, the second frame image will not be put into the buffer queue 22 in time because it cannot be rendered. Correspondingly, when the signal V33 arrives, the synthesis module 23 will not be able to obtain the rendering result of the second frame image from the buffer queue 22 for synthesis processing. Accordingly, the display screen will not obtain the display image of the second frame image before the next Vsync signal (such as signal V34) arrives. As a result, the display screen 24 cannot display the second frame image in the display period between the signal V34 and the signal V35. For example, the display screen 24 can continue to display the first frame image in the display period between the signal V34 and the signal V35.

[0081] In the example shown in Figure 3 , the second frame of image can be rendered and queued before the arrival of signal V34. Thus, after the arrival of signal V34, the synthesis module 23 can retrieve the rendering result of the second frame from the buffer queue 22 for synthesis processing. Before the arrival of signal V35, the display screen 24 can receive the image of the second frame for display. Therefore, the display screen 24 can switch to displaying the second frame of image when signal V35 arrives.

[0082] FIG3 also shows the processing and display mechanism of the third frame image. In the example shown in FIG3 , the rendering instruction of the third frame image can be issued by the application 1 after the signal V34 arrives, and the rendering processing is performed by the drawing module 21 to obtain the corresponding rendering result. The rendering result can be placed in the buffer queue 22 and wait for consumption before the signal V35 arrives. In this way, the synthesis module 23 can obtain the rendering result of the third frame image from the buffer queue 22 after the signal V35 arrives, perform synthesis processing, and obtain the corresponding display image. The display image of the third frame image can be transmitted to the display screen 24 before the signal V36 arrives. Therefore, the display screen 24 can switch to display the third frame image after the signal V36 arrives.

[0083] It can be seen that in the example shown in FIG3 , since the rendering time of the second frame image is longer than one Vsync cycle, the display screen 24 eventually displays the first frame image continuously within two display cycles.

[0084] It should be noted that in some cases, the rendering instructions issued by application 1 may be related to an action already entered by the user. For example, if the user enters the action of releasing skill A on the game screen, the rendering instructions issued by application 1 may instruct the electronic device to render and display the corresponding screen effect after releasing skill A.

[0085] For example, let's say Application 1 receives user input between signals V31 and V32, indicating the user is releasing Skill A. The second frame of the image then includes the visual effects associated with releasing Skill A. Obviously, the sooner the electronic device displays the second frame, the sooner the user can see the visual effects associated with releasing Skill A on the interface, resulting in a better hand-tracking experience.

[0086] However, in conjunction with the example in Figure 3, since the rendering time of the second frame image is longer than one Vsync cycle, the second frame image, which should be displayed within the display cycle of signal V34 to signal V35, is delayed to the next display cycle (such as signal V35 to signal V36). In this way, after the user inputs the operation, there is a delay of at least 3 Vsync cycles (such as the Vsync cycle between signal V32 and signal V35) from the start of rendering to the display of the second frame image on the display screen. Correspondingly, the user cannot see the screen effect of the operation in time. The display of other frame images also has varying degrees of large delays. Therefore, the problem of poor chirality in the existing solution is manifested. In this application, the screen effect corresponding to the operation can also be referred to as response content.

[0087] To address the aforementioned issues, the technical solutions provided by the embodiments of the present application enable the synthesis module to acquire and synthesize the queued image data in advance, regardless of the arrival of the next Vsync signal. This allows the display screen to obtain the synthesis result (such as the display image) for display earlier.

[0088] For example, referring to FIG4 , there is an example of the processing and display logic of each frame image in the scenario shown in FIG3 after the solution provided in the embodiment of the present application takes effect.

[0089] As shown in Figure 4 , after the solution provided in this embodiment of the present application takes effect, for the first frame of image, application 1 can issue the corresponding rendering instructions after receiving signal V31. The drawing module can then render the first frame of image, obtain the rendering results, and queue them. This process is similar to the current solution implementation shown in Figure 3 .

[0090] Different from the implementation in the current solution, in the example of Figure 4, the synthesis module 23 does not need to wait until the next Vsync signal arrives to obtain data from the buffer queue 22 for subsequent processing. Correspondingly, the synthesis module 23 can directly obtain the queued data from the buffer queue 22 when new data is available in the buffer queue 22. For example, the synthesis module 23 can obtain the rendering result of the newly queued 1st frame image from the buffer queue 22. Correspondingly, the synthesis module 23 can perform synthesis processing on the rendering result of the 1st frame image, and before the signal V32 arrives, send the display image of the 1st frame image obtained by the synthesis processing to the display screen 24. As a result, the display screen 24 can display the received 1st frame image when the next Vsync signal (such as signal V32) arrives.

[0091] Therefore, compared to the existing solution shown in Figure 3, the solution provided by this application advances the display timing of the first frame image from the display period between signals V33 and V34 shown in Figure 3 to the display period between signals V32 and V33. This means that there is only a delay of one Vsync period from the start of rendering to the display of the first frame image on the display screen, making the display of the first frame image more timely.

[0092] Based on a similar processing mechanism to the first frame, the chirality of other frame images can also be significantly improved.

[0093] Take the second frame image as an example. As shown in Figure 4, application 1 can issue a rendering instruction for the second frame image after the signal V32 arrives. Correspondingly, similar to the scene example in Figure 3, the rendering time of the second frame image can be greater than one Vsync cycle. For example, after the arrival of signal V33 and before the arrival of signal V34, the rendering of the second frame image can be completed. In this application, the synthesis module 23 can directly obtain the rendering result of the second frame image from the buffer queue 22 for synthesis processing after the second frame image is rendered into the queue, without waiting for the next Vsync signal (such as signal V34) to arrive. Thus, before the arrival of signal V34, the display screen 24 can obtain the display image of the second frame image. Correspondingly, after the arrival of signal V34, the display screen 24 can display the second frame image.

[0094] As a result, the display timing of the second frame image is advanced from the display cycle between signals V35 and V36 shown in Figure 3 to the display cycle between signals V34 and V35. This way, even if the rendering time of the second frame image is longer, the delay from the start of rendering to the display of the second frame image on the screen is only two Vsync cycles. This makes the display of the second frame image more timely. This allows users to see the visual effects after releasing Skill A earlier, thereby significantly improving the tracking experience during the operation of Application 1.

[0095] The solution provided in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0096] It should be noted that the solution provided in the embodiment of the present application can be applied to electronic devices. The electronic device may include a mobile phone, a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (ULTRA-MOBILE PERSONAL COMPUTER, UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, or at least one of a smart city device. The embodiment of the present application does not impose any special restrictions on the specific type of the electronic device. In the present application, the electronic device may be configured with a display screen for displaying images.

[0097] In some embodiments, the electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) connector, a charging management module, a power management module, a battery, an antenna 1, an antenna 2, a mobile communication module, a wireless communication module, an audio module, a speaker, a receiver,

[0098] Microphone, headphone jack, sensor module, button, motor, indicator, camera module, display, and subscriber identification module (SIM) card interface, etc. The sensor module may include pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor, bone conduction sensor, etc.

[0099] The processor may include one or more processing units, for example, an application processor (AP), a modem processor (Modem), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor (BP or BBP), and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0100] The processor can generate operation control signals based on instruction opcodes and timing signals to complete the control of instruction fetching and execution.

[0101] The processor may also include a memory for storing instructions and data. In some embodiments, the memory in the processor may be a cache memory. This memory can store instructions or data that have been used by the processor or that are frequently used. When the processor needs to use the instruction or data, it can directly access it from the memory. This avoids duplicate accesses, reduces processor latency, and thus improves system efficiency.

[0102] It should be noted that the structures illustrated in the embodiments of this application do not constitute specific limitations on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than those in the above examples, or may combine or separate certain components, or arrange the components differently. Each component may be implemented in hardware, software, or a combination of software and hardware.

[0103] In some embodiments, based on the above components, an electronic device can implement display functions using a GPU, a display screen, and an application processor. A GPU is a microprocessor for image processing that connects the display screen and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs that execute program instructions to generate or modify display information.

[0104] A display screen is used to display images, videos, and the like. The display screen includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, an electronic device may include one or more display screens.

[0105] The internal memory can be used to store computer executable program code, which includes instructions. The internal memory may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a game application, etc.), etc. The data storage area can store data created during the use of the electronic device (such as image data, audio data, phone book, etc.). In addition, the internal memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor executes various functional methods or data processing of the electronic device by running instructions stored in the internal memory and / or instructions stored in a memory provided in the processor.

[0106] A touch sensor, also known as a "touch control device," can be located on a display screen. The touch sensor and the display screen together form a touch screen, also known as a "touch screen." The touch sensor is used to detect touch operations applied to or near the touch sensor. The touch sensor can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided through the display screen. In other embodiments, the touch sensor can also be located on the surface of the electronic device, in a location different from the display screen.

[0107] In some embodiments of the present application, the touch sensor can be implemented by a touch panel (TP). Taking the display screen as an LCD as an example. In some implementations, the TP can be integrated with the LCD to form a TP-LCD. In other implementations, the TP can be separated from the LCD. In this way, the user can input an operation by touching the corresponding position of the screen displayed on the LCD (such as a game screen). Correspondingly, the TP can receive the operation and send the operation information (such as the operation type, location information, etc.) to the application for subsequent processing.

[0108] In the above example, the composition of an electronic device in the present application is provided. The present application embodiment also provides the composition of another electronic device. This composition can be used to describe the software composition logic of the electronic device.

[0109] For example, refer to FIG5 , which is a schematic diagram of the composition of another electronic device provided in an embodiment of the present application.

[0110] In the example shown in FIG5 , the software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. Taking the system as an example, the software structure of the electronic device is illustrated.

[0111] As shown in Figure 5, the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, The system is divided into five layers, from top to bottom: application layer, application framework layer, Android runtime (ART) and native C / C++ library, hardware abstraction layer (HAL) and kernel layer.

[0112] The following explains them separately.

[0113] The application layer can include a series of application packages. This layer can also be referred to as the application layer or the APP layer. As shown in Figure 5, an application package can include applications such as games, calendars, maps, WLAN, music, SMS, calls, navigation, Bluetooth, and videos. For example, the game application package can correspond to the game application (Application 1) in the previous example.

[0114] The application framework layer, also known as the framework layer, provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0115] As shown in FIG5 , the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, and the like.

[0116] The window manager provides window management services (WMS). WMS can be used for window management, window animation management, surface management, and as a transfer station for the input system.

[0117] Content providers are used to store and retrieve data and make it accessible to applications. This data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0118] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and the like. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon can include a view for displaying text and a view for displaying pictures. As an example, one or more surface views (Surface) can be configured in the view system. Each surface view can be configured with one or more textures (textures). Each texture can be used to store part or all of the image content in a frame of image. It should be noted that in the specific implementation process of the present application, the image data in the surface view and / or texture can be stored in the memory of the electronic device. Correspondingly, the electronic device can manage, such as write and read, each data through the file identifier of each data in the memory.

[0119] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0120] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0121] The Activity Manager can provide Activity Management Service (AMS), which can be used to start, switch, and schedule system components (such as activities, services, content providers, and broadcast receivers) as well as manage and schedule application processes.

[0122] The input manager can provide an input management service (IMS), which can be used to manage system input, such as touch screen input, key input, sensor input, etc. The IMS takes events from input device nodes and distributes the events to appropriate windows through interaction with the WMS. In some embodiments, the IMS may include components such as IMSReader and IMSDispatcher (IMS dispatch module). As an example, after receiving a user operation, the TP of the electronic device can send the operation information to the IMSReader.

[0123] IMSReader can send this operation information to IMSDispatcher for centralized management and distribution. For example, IMSDispatcher can send this operation information to the foreground application (such as a game application) so that the game application can respond to the user input. For example, the game application generates rendering instructions for the next frame based on the user input.

[0124] The Android runtime consists of the core libraries and the Android runtime. The Android runtime is responsible for converting source code into machine code. It primarily utilizes ahead-of-time (AOT) and just-in-time (JIT) compilation technologies.

[0125] The core library is mainly used to provide basic Java class library functions, such as basic data structures, mathematics, IO, tools, databases, networks, etc. The core library provides an API for users to develop Android applications.

[0126] Native C / C++ libraries can include multiple functional modules, such as surface manager, media framework, libc, OpenGL ES, SQLite, Webkit, etc.

[0127] The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media framework supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library supports a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. OpenGL ES provides 2D and 3D graphics drawing and manipulation within applications. SQLite provides a lightweight relational database for electronic device applications.

[0128] The hardware abstraction layer (HAL) runs in user space, encapsulates kernel layer drivers, and provides a calling interface to the upper layer. For example, the HAL may include a display module, an audio module, a camera module, a Bluetooth module, etc. In some embodiments, the HAL may also include a touch module.

[0129] The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver (such as DRMDriver), a camera driver, an audio driver, and a Bluetooth driver. In some embodiments, the kernel layer may also include a touch driver.

[0130] It should be noted that the software composition shown in FIG5 is only an exemplary description and does not constitute a limitation on the electronic device involved in the embodiments of the present application. In other embodiments, the electronic device may also have other software compositions.

[0131] For example, referring to Figure 6, which is a schematic diagram of the composition of another electronic device provided by an embodiment of the present application, the example in Figure 6 shows both upper-layer components composed of software modules and various possible configurations of hardware components.

[0132] As shown in Figure 6, in this example, the application layer may include multiple applications. For example, the multiple applications may include application 1, application 2, etc. Among them, application 1 may be a game application. Application 2 may be a small window application or a multi-window application. For example, application 2 may provide the user with relevant controls (such as brightness adjustment, communication function configuration, etc.) during the operation of application 1 by displaying a window on the interface. In some embodiments, application 2 may be a system-level application such as a game center installed in an electronic device.

[0133] The framework layer of the electronic device can be configured with a drawing module, SF module, view system, IMS module, etc.

[0134] The rendering module is used to render an image in response to a rendering instruction issued by an application program (such as application 1), and may correspond to the rendering module 21 in the aforementioned example.

[0135] In some implementations, the drawing module may perform rendering processing according to the rendering instruction through a GPU configured in the hardware layer of the electronic device, thereby obtaining a corresponding rendering result.

[0136] In an embodiment of the present application, a rendering module (or application) can act as a buffer producer. A buffer can be a storage space configured in a memory. Each buffer can store data corresponding to a rendering result. Thus, after the rendering module completes the rendering process of a frame image and obtains a rendering result, it can store the rendering result in a corresponding buffer and enqueue the buffer.

[0137] In some implementations, enqueuing the Buffer may correspond to storing the rendering result data (such as image information) stored in the Buffer into a corresponding buffer queue (BufferQueue).

[0138] In other implementations, enqueuing a buffer may correspond to storing an identifier (e.g., a BufferID) corresponding to the buffer in a buffer queue. It is understood that a BufferID uniquely represents a buffer. Thus, when an electronic device needs to use the buffer, it can obtain the BufferID to determine the address information corresponding to the buffer in the memory and read the rendering result stored in the buffer using the address information.

[0139] In some other implementations, enqueuing a buffer may correspond to storing a file identifier for storing the rendering result in the buffer in a buffer queue. The file identifier may include address information indicating the data stored in the buffer. Thus, when the electronic device needs to use the buffer, it can determine the address information corresponding to the buffer in the memory using the file identifier and read the rendering result stored in the buffer using the address information.

[0140] In the following example, the file identifiers of each Buffer are stored in the BufferQueue as an example.

[0141] The SF module is also known as SurfaceFlinger. The SF module can be used to generate a Vsync signal based on information such as the currently displayed application and the refresh rate of the current display. In some implementations, the SF module can correspond to the synthesis module 23 shown in Figure 4.

[0142] The SF module can act as a consumer of produced buffers, obtaining queued buffers from the buffer queue for consumption and display. Exemplarily, the processing mechanism for this consumption and display can involve the SF performing synthesis processing based on the rendering results corresponding to the buffers obtained from the buffer queue to obtain the corresponding display image. The SF can then directly or indirectly transmit this display image to the electronic device's display screen for display of the next frame.

[0143] In the embodiment of the present application, the SF module may have the ability to monitor the buffer queue.

[0144] For example, the SF module can monitor whether there is a new buffer in the buffer queue. If there is a new buffer in the buffer queue, the SF can directly obtain the new buffer for consumption and display without waiting for the next Vsync signal.

[0145] A view system can also be configured in the framework layer. The view system can include one or more surface views (Surface). When the drawing module performs rendering processing on a frame of image, it can perform drawing based on one or more Surfaces. In this way, after completing the rendering operation of a frame of image, the image obtained by the rendering processing can be stored in one or more Surfaces in the view system. In an embodiment of the present application, a buffer can correspond to one or more Surfaces. In the case where a Buffer corresponds to multiple Surfaces, the multiple Surfaces can respectively store part of the content of the frame image corresponding to the Buffer. The collection of all the contents in the multiple Surfaces can constitute the content of a frame of image corresponding to the Buffer. In this way, after completing the rendering processing of a frame of image, the drawing module can realize the enqueueing of a Buffer and obtain one or more Surfaces storing the rendering results in the view system.

[0146] Correspondingly, when the SF module consumes the Buffer, it obtains the file identifier of the Buffer from the BufferQueue, and then obtains the image data stored in the Surface corresponding to the Buffer from the address indicated by the file identifier for synthesis processing.

[0147] As shown in Figure 6, an IMS module can also be configured in the framework layer. In conjunction with the description in Figure 5, the IMS module can include an IMSReader module and an IMS dispatch module. The IMS dispatch module is also called an IMSDispatcher.

[0148] In some implementations of this application, the IMS module can be used to implement the transmission of user input operation information from the lower layer to the upper layer. This process will be described in detail later.

[0149] A hardware hybrid rendering module (HWC) can be configured in the abstract layer of the electronic device. In some embodiments, the HWC can be used to perform synthesis processing of multiple Surfaces alone. In other embodiments, the HWC can cooperate with the SF module of the framework layer to implement synthesis processing. In other embodiments, the synthesis processing can also be performed by the SF module alone, and the corresponding HWC is used to transmit the display image obtained after the synthesis processing to the display screen.

[0150] In the following description, it is taken as an example that the synthesis process is performed by the SF module alone.

[0151] In other embodiments of the present application, the abstraction layer may also be configured with a compositor to facilitate compositing in different scenarios.

[0152] In the electronic device configuration shown in Figure 6, the core layer can be configured with touch drivers, display drivers, and other components. The touch driver can be used to implement upper-layer software control of the touchpad (TP) and transmit information from the TP to upper-layer software modules. The display driver can be a DRMDriver. The display driver can directly or indirectly receive images sent from the SF module and control the corresponding display screen to display the images sent.

[0153] The electronic device may be configured with various hardware components in the hardware layer as shown in FIG6 to support the above-mentioned software modules to implement corresponding functions.

[0154] For example, the electronic device may be configured with a touchpad, a display screen, a GPU, a memory, and the like.

[0155] In some embodiments of the present application, when multiple applications are running in an electronic device, each application may be configured with a corresponding buffer queue in a memory.

[0156] For example, if application 1 and application 2 are running at the same time, a buffer queue BQ1 corresponding to application 1 may be configured in the memory. The buffer queue BQ1 is used to store the buffer obtained after the drawing module executes the rendering instruction issued by application 1.

[0157] The memory may also be configured with a buffer queue BQ2 corresponding to application 2. The buffer queue BQ2 is used to store the buffer obtained after the drawing module executes the rendering instruction issued by application 2.

[0158] In some implementations, the multiple buffer queues can be managed and maintained by a general buffer module (such as a BufferTX module). In different implementations, the BufferTX module can be configured in the framework layer, abstract layer, kernel layer, or memory of the electronic device. The embodiments of the present application do not limit the configuration selection of the BufferTX module.

[0159] In the example shown in FIG6 , the BufferTX module is configured in the framework layer as an example.

[0160] In some embodiments, the BufferTX module can configure an identification field for each configured buffer queue. The value of the identification field can be a first value, a second value, etc. When the identification field is a first value (such as the first value is 0), it indicates that there is no available buffer in the corresponding buffer queue, that is, the buffer queue is empty. When the identification field is a second value (such as the second value is 1), it indicates that there is an available buffer in the corresponding buffer queue. In some implementations, the second value can also be used to indicate the number of available buffers in the current buffer queue. For example, if the second value is 1, it indicates that the number of available buffers in the corresponding buffer queue is 1.

[0161] As an example, when there is a buffer corresponding to application 1 entering the buffer queue BQ1, the identification field corresponding to the buffer queue BQ1 in the BufferTX module may be configured as 1, indicating that there is a buffer queued in the buffer queue BQ1.

[0162] After SF obtains a buffer from buffer queue BQ1 for consumption, the available buffer in buffer queue BQ1 is empty. The corresponding identification field of buffer queue BQ1 in the BufferTX module is configured to 0. This indicates that the available buffer in buffer queue BQ1 is empty and no buffer can be consumed.

[0163] Similarly, the BufferTX module can also manage and maintain the buffer queues corresponding to other applications (such as the buffer queue BQ2) based on a similar mechanism.

[0164] Therefore, the SF module can determine whether there is a new buffer entering the buffer queue through the various identification fields configured in the BufferTX module.

[0165] In other embodiments of the present application, a display queue (not shown in FIG6 ) may also be configured in the memory. The display queue may be managed by a display driver. For example, the display driver may place each display image into the display queue upon receipt of a display image. Thus, when the next Vsync signal arrives, the display driver retrieves the display image that first enters the display queue from the display queue and sends it to the display screen for display.

[0166] The solutions provided in the embodiments of the present application can all be applied to the electronic devices provided in the aforementioned FIG. 5 or FIG. 6 .

[0167] In the following description, the implementation of the solution provided in the embodiment of the present application is illustrated by taking the application of the solution to the electronic device shown in FIG6 as an example.

[0168] For example, refer to Figure 7, which shows an example of inter-module interaction provided by an embodiment of the present application. By implementing the solution shown in Figure 7, it is possible to support reporting of user input operations to the application while the application is running. In the example shown in Figure 7, the currently running application is Application 1.

[0169] As shown in Figure 7, the solution may include:

[0170] S701: The touch panel of the electronic device receives an operation OP1 input by a user and sends operation information 71 to a touch driver.

[0171] For example, when the display screen of an electronic device displays a certain interface during the operation of application 1, an operation OP1 input by a user may be received. As a possible implementation, this operation OP1 may include clicking a control displayed in the current interface. Through this operation OP1, the user can instruct the currently controlled game character to perform a corresponding action, such as using skill A corresponding to the control.

[0172] Correspondingly, the touch panel can generate operation information 71 according to the operation OP1. In some embodiments, the operation information 71 may include the operation type of the operation OP1 (such as a click operation or a slide operation), the position information corresponding to the operation OP1 on the display screen, and the like.

[0173] Thus, the touch panel can realize the transmission of the currently received operation information 71 to the upper-layer application module through touch driving.

[0174] S702: The touch driver sends the operation information to the IMSReader module.

[0175] S703: The IMSReader module sends the operation information 71 to the IMS scheduling module.

[0176] In some embodiments, the IMSReader module can monitor and obtain information reported by each driver. For example, the IMSReader can obtain operation information 71 reported by the touch driver.

[0177] In this way, the IMSReader module can transmit the operation information 71 to the IMS scheduling module for centralized processing.

[0178] In some embodiments, the IMS scheduling module may determine that the operation information 71 is an operation input by the user to the application 1 based on the fact that the interface currently displayed on top is the interface corresponding to the application 1.

[0179] S704 : The IMS scheduling module sends the operation information 71 to the application 1 .

[0180] Thus, when the user inputs the operation OP1 during the use of the application 1 , the application 1 can obtain the operation information 71 input by the user, and thus the application 1 can perform subsequent operations according to the operation information 71 .

[0181] For example, the application 1 may instruct the various modules in the electronic device to cooperate with each other according to the operation information 71 and display an image corresponding to the operation information 71 .

[0182] Consider the example in Figure 4 . For example, application 1 receives operation information 71 during the Vsync period between signals V31 and V32. Based on this operation information 71, application 1 can instruct the electronic device to render and display the second frame of image after signal V32 arrives. This second frame of image can include the visual effects corresponding to operation OP1, such as the visual effects corresponding to skill A.

[0183] As a possible implementation, referring to Figure 8, another example of inter-module interaction provided by an embodiment of the present application, by implementing the solution shown in Figure 8, the image corresponding to the operation information 71 (such as the second frame image) can be produced and included in the column.

[0184] As shown in Figure 8, the solution may include:

[0185] S801: Application 1 sends a rendering instruction 81 to a drawing module. In this application, the rendering instruction 81 may also be referred to as a first rendering instruction.

[0186] For example, application 1 can generate the rendering instruction 81 based on the operation information 71. Thus, through the rendering instruction 81, application 1 can instruct the electronic device to render a frame image (such as the second frame image) including the screen effect corresponding to operation OP1. The second frame image can also be replaced with the Nth frame image.

[0187] S802: The drawing module calls the GPU to perform rendering processing according to the rendering instruction 81 to obtain a rendering result 82. In this application, the rendering result 82 may also be referred to as a first rendering result.

[0188] S803: The rendering module stores the rendering result 82 in the surface view S1.

[0189] Exemplarily, the drawing module may send the rendering instruction 81 to the GPU so that the GPU performs rendering processing on the second frame image according to the rendering instruction 81 .

[0190] Through this rendering process, one or more textures corresponding to the second frame image can be obtained.

[0191] In this example, it is taken as an example that the rendering result 82 corresponding to the rendering instruction 81 is stored in the surface view S1.

[0192] S804: The rendering module stores the file identifier 84 in the buffer queue BQ1. The buffer queue BQ1 may also be referred to as a first buffer queue.

[0193] The file identifier 84 may correspond to the rendering result 82 .

[0194] In some embodiments, the file identifier 84 may indicate the storage location of the rendering result 82 in the electronic device. It is understood that, based on the aforementioned description of frame buffer production enqueue and consumption dequeue, the storage location of the rendering result 82 in the electronic device may also correspond to the location of the buffer (e.g., Buffer 1) configured for the current rendering instruction 81 in memory.

[0195] Thus, in this example, by storing the file identifier 84 in the buffer queue BQ1 , the production of the Buffer1 corresponding to the rendering result 82 can be achieved.

[0196] In conjunction with the description of the BufferTX module in the preceding example, in some embodiments of the present application, the BufferTX module may be configured with an identification field 1 for buffer queue BQ1. After Buffer1 corresponding to step S804 is enqueued to buffer queue BQ1, the identification field 1 of buffer queue BQ1 in the BufferTX module may be configured to 1. This indicates that a new buffer has been enqueued in buffer queue BQ1 and is awaiting consumption. This identification field 1 may also be referred to as a first identification field.

[0197] With reference to the example in FIG. 4 , the rendering instruction 81 is used as an example to instruct the rendering of the second frame image.

[0198] Thus, application 1 can execute S801 after signal V32 arrives. Correspondingly, after signal V33 arrives, the drawing module completes the enqueueing of Buffer 1 corresponding to S804.

[0199] In an embodiment of the present application, the SF module can be configured to obtain the buffer from the buffer queue for consumption and display without waiting for the next Vsync signal to arrive after a new buffer is queued.

[0200] For example, referring to Figure 9, which is a schematic diagram of another module interaction provided by an embodiment of the present application, the electronic device can realize fast consumption and display of newly queued buffers by implementing the solution shown in Figure 9.

[0201] As shown in Figure 9, the solution may include:

[0202] S901. The SF module obtains the file identifier 84 from the buffer queue BQ1.

[0203] For example, after the file identifier 84 is queued, the SF module can sense that a new buffer has been queued, and then directly obtain the buffer for consumption and display.

[0204] As a possible implementation, the SF module can determine whether there is a new buffer queue by monitoring the identification field of the buffer queue configured for each currently running application in the BufferTX module.

[0205] For example, SF may determine that a new buffer is queued in the buffer queue BQ1 based on the fact that the identification field 1 is configured as 1.

[0206] Therefore, the SF module can directly obtain the Buffer1 from the buffer queue BQ1 for subsequent processing.

[0207] In this example, the buffer enqueue is illustrated by placing the file identifier 84 into the buffer queue. Accordingly, the information of the newly enqueued buffer obtained by the SF module from the buffer queue BQ1 can include the file identifier 84. The file identifier 84 can correspond to the storage address of Buffer1 or the storage address of the rendering result 82.

[0208] S902 , the SF module obtains the rendering result 82 in the surface view S1 according to the file identifier 84 .

[0209] Exemplarily, the SF module may obtain the rendering result 82 drawn in the representation view S1 according to the address indicated by the file identifier 84 .

[0210] Therefore, the SF module can perform synthesis processing based on the rendering result 82, and then obtain the complete content of the frame image indicated by the rendering instruction 81.

[0211] For example, the SF module may obtain a corresponding display image 91 through synthesis processing based on the rendering result 82. The display image 91 may also be referred to as a first display image.

[0212] It should be noted that in the description of Figures 8 and 9 , the rendered frame image corresponding to rendering instruction 81 is stored in surface view S1 as an example. In other embodiments, the rendering results of the drawing module according to the rendering instruction can be stored in multiple surface views. In this way, the SF module can obtain the rendering results from the multiple surface views based on the dequeued file identifier and perform synthesis processing to obtain the content of a complete frame image.

[0213] S903. The SF module transmits the display image 91 to the HWC.

[0214] S904. HWC transmits the display image 91 to the display driver.

[0215] In this example, after the SF module completes the synthesis process, it can transmit the acquired display image 91 to the display driver through the HWC to wait for display.

[0216] S905: The display driver transmits the display image 91 to the display screen, and the display screen displays the display image 91.

[0217] For example, after receiving display image 91, the display driver can store it in a display queue. This allows the display driver to retrieve the first display image received from the display queue for display after the next Vsync signal arrives. In this example, after display image 91 enters the display queue, the display queue only includes display image 91.

[0218] In this way, after the next Vsync signal arrives, the display driver can obtain the display image 91 from the display queue and transmit the display image 91 to the display screen for display.

[0219] Combined with the example in Figure 4. After the arrival of signal V33 and before the arrival of signal V34, the drawing module can complete the rendering of the corresponding rendering result of the second frame image into the queue. Based on the solution implementation shown in Figure 9, the SF module (i.e., the synthesis module 23 shown in Figure 4) can directly consume the rendering result of the second frame image for display after the second frame image is rendered into the queue, without waiting for the arrival of signal V34. Therefore, before the arrival of signal V34, the SF module can synthesize and obtain the corresponding display image (such as display image 91) based on the rendering result of the second frame image, and transmit it to the display driver for display. In this way, when the next Vsync signal (such as signal V34) arrives, the display driver can control the display screen to display the second frame image based on the display image 91.

[0220] In order to more clearly illustrate the solution provided by the embodiment of the present application, the rendering and display process of the above-mentioned second frame image is explained below through the inter-module interaction process diagram shown in Figure 10. In the process diagram shown in Figure 10, the storage of rendering results in the buffer queue is used for explanation. Combined with the above-mentioned description of Buffer production enqueue, in a specific implementation, the enqueueing of the rendering result shown in Figure 10 to the buffer queue can also be achieved by enqueuing the corresponding file identifier to the buffer queue. Correspondingly, when Buffer consumption and display are required, after obtaining the file identifier from the buffer queue, the rendering result can be obtained according to the storage address indicated by the file identifier.

[0221] As shown in Figure 10, the solution may include:

[0222] S1001: The touch panel receives an operation OP1, which may also be referred to as a first operation.

[0223] Illustratively, before the touch panel receives the operation OP1, the first frame of image may be displayed on the display screen.

[0224] Thus, the operation OP1 may be an operation input by the user on the first frame image. Thus, the touch panel may generate operation information 71 based on the operation OP1 input by the user. In some embodiments, the operation information 71 may include the operation type and location information of the operation OP1. The operation information 71 may also be referred to as first operation information.

[0225] S1002 : The touch panel sends the operation information 71 to application 1 .

[0226] For example, in combination with the description in FIG. 7 , the touch panel may send the operation information 71 to the application 1 through the touch driver and the IMS module.

[0227] In this example, after the application 1 receives the operation information 71 , the next Vsync signal (eg, signal V32 ) arrives.

[0228] S1003 : Application 1 generates rendering instruction 81 .

[0229] For example, application 1 may generate a new rendering instruction according to the arrival of signal V32 and instruct other components of the electronic device to render a new graphic.

[0230] In some embodiments, the application 1 may generate a rendering instruction 81 according to the received operation information 71 upon receipt of the signal V32 .

[0231] S1004 , application 1 sends rendering instruction 81 to the drawing module.

[0232] Thus, the application 1 can instruct the drawing module to draw a new image through the rendering instruction 81. For example, the new image can be an image subsequent to the currently displayed first frame image, such as the second frame image.

[0233] S1005 , the drawing module performs rendering processing and obtains a rendering result 82 .

[0234] Exemplarily, the drawing module can work with a GPU configured in the electronic device to perform rendering processing according to the rendering instruction 81 , thereby obtaining a corresponding rendering result 82 .

[0235] It should be noted that, in combination with the example in Figure 4, in this example, the rendering processing corresponding to the rendering instruction 81 executed by the drawing module can span two Vsync cycles and complete the rendering processing before the end of the second Vsync cycle (such as the Vysnc cycle between signal V33 and signal V34).

[0236] S1006 : The drawing module queues the rendering result 82 into the buffer queue BQ1 .

[0237] The buffer queue BQ1 may be a buffer queue configured corresponding to the application 1. Thus, the production queue of the rendering result corresponding to the second frame image can be realized.

[0238] S1007. The BufferTX module determines that a new buffer is queued.

[0239] Exemplarily, the BufferTX module can be used to monitor the entry of new buffers into each buffer queue.

[0240] In this example, after the buffer queue BQ1 receives the enqueue of the rendering result 82 , the BufferTX module may determine that a new Buffer is enqueued in the buffer queue BQ1 waiting to be consumed.

[0241] S1008. The BufferTX module configures the identification field 1 to 1.

[0242] The identification field 1 may be a field in the BufferTX module that identifies an available buffer in the buffer queue BQ1.

[0243] In this example, when a new buffer (such as the buffer corresponding to rendering result 82) is queued in buffer queue BQ1, the BufferTX module can configure the identification field 1 corresponding to buffer queue BQ1 from 0 to 1. This indicates that the available buffer in buffer queue BQ1 changes from empty to 1.

[0244] S1009. The SF module determines that the identification field 1 is configured as 1.

[0245] In this example, the SF module can monitor changes in each identification field in the BufferTX module.

[0246] For example, the SF module may be configured as 1 according to the identification field 1 in the BufferTX module, and determine that there is one buffer to be consumed in the corresponding buffer queue BQ1.

[0247] S1010 , the SF module obtains the rendering result 82 in the buffer queue BQ1 corresponding to the identification field 1 .

[0248] For example, the SF module can store the correspondence between each identification field and the corresponding buffer queue. Thus, upon determining that identification field 1 changes from 0 to 1 in the BufferTX module, the SF module can determine that a new buffer has been queued in buffer queue BQ1 based on this correspondence. As a result, the SF module can directly retrieve the queued buffer from buffer queue BQ1 for consumption and display without waiting for the next Vsync signal.

[0249] As an implementation, the SF module may obtain the queued rendering result 82 from the buffer queue BQ1 .

[0250] S1011. The SF module performs synthesis processing based on the rendering result 82 to obtain the display image 91.

[0251] Thus, the SF module can trigger the display of the second frame image without waiting for the next Vsync signal (such as signal V34). For example, before the signal V34 arrives, the SF module can complete the synthesis processing of the second frame image and obtain the corresponding display image 91.

[0252] S1012: The SF module transmits the display image 91 to the display module. In this application, the display module may include a display driver and a display screen.

[0253] S1013: The display module stores the display image 91 in the display queue.

[0254] For example, the SF module transmits the display image 91 to the display driver in the display module, so that the display driver can store the received display image 91 in the display queue, waiting for display.

[0255] S1014 , the display module executes the process of dequeuing the display image 91 according to the signal V34 , and displays the second frame of image according to the display image 91 .

[0256] Exemplarily, the display driver of the display module can obtain the stored display image 91 from the display queue according to the arrival of the signal V34, and control the display screen of the display module to display according to the display image 91.

[0257] In this example, the image 91 sent for display may correspond to the second frame image. In this way, the switching display from the lower frame to the second frame image can be realized on the display screen.

[0258] It can be understood that, through the implementation of the solution shown in FIG10 , even if the rendering processing of the second frame image takes a long time, the delay from application 1 issuing rendering instruction 81 to the corresponding second frame image switching display on the display screen is only two Vsync cycles. Therefore, if the second frame image includes the response effect of the user input operation (such as operation OP1), the user can see the effect of the input operation on the display screen more quickly, thereby improving the display tracking during game operation.

[0259] Figures 8 through 10 above provide detailed descriptions of the rendering and display mechanism for a single frame of image. It is understood that the rendering and display of other frames of image can also be implemented by referring to the various solutions provided in the embodiments of this application. This allows the display screen to more quickly display the corresponding response effects of the user's input operations, thereby achieving the purpose of improving hand tracking.

[0260] When the solution provided in the embodiment of the present application is effective, the effect shown in Figure 4 is obtained. Compared with the implementation effect of the existing solution shown in Figure 3, significant benefits can be obtained in both the synthesis process of the SF module and the display process on the display screen.

[0261] Through experimental verification, the following Table 1 provides a comparison of the delays of the synthesis process and the display process before and after the solution provided in the embodiment of the present application is effective, taking a 120Hz frame rate as an example.

[0262] Table 1

[0263] In the example of Table 1, the synthesis delay may refer to the delay from the issuance of the rendering instruction for the current frame image to the start of the synthesis process of the current frame image. The display delay may refer to the delay from the issuance of the rendering instruction for the current frame image to the display of the current frame image on the display screen.

[0264] It can be seen that after the solution provided in the embodiment of the present application takes effect, the synthesis delay and display delay can be significantly shortened, thereby achieving the effect of improving the chirality.

[0265] It should be noted that in the example shown in Figure 10 , the SF module can consume newly queued buffers from any application's corresponding buffer queue, without waiting for the next Vsync signal. In some examples, this implementation is also referred to as enabling the XSync scheme. That is, after the XSync scheme is enabled, the SF module consumes newly queued buffers without waiting for the next Vsync signal.

[0266] In other embodiments, after determining that a new buffer is queued (eg, after executing S1009 ), the SF module may determine whether to enable the XSync solution based on relevant information of the buffer queue BQ1 corresponding to the identification field 1 .

[0267] As a possible implementation, an electronic device (such as an SF module of an electronic device) may be configured with an application whitelist. The application whitelist may include the package name of at least one application. For example, the package name of the at least one application may correspond to the package name of a preset game application.

[0268] After executing S1009 and before executing S1010 , the SF module may determine that a new buffer is queued in the current buffer queue BQ1 based on the configuration of the identification field 1 being 1.

[0269] The SF module can determine whether the application corresponding to the buffer queue BQ1 is included in the application whitelist based on the name of the buffer queue BQ1. For example, if the name of the buffer queue BQ1 includes the package name of application 1 and application 1 is included in the application whitelist, then the application corresponding to the buffer queue BQ1 is included in the application whitelist.

[0270] Therefore, the SF module can determine to enable the XSync solution based on the application corresponding to the buffer queue with the new buffer being included in the application whitelist. That is, before the signal V34 arrives, S1010 is executed to realize the consumption of the newly queued buffer in the buffer queue BQ1.

[0271] In other embodiments, if the application corresponding to the buffer queue where the new buffer is queued is not included in the application whitelist, the SF module may not enable the XSync solution.

[0272] In this way, the SF module can wait for the next Vsync signal (such as signal V34) to arrive, and then obtain the queued Buffer for consumption.

[0273] As an example, refer to FIG11 , which illustrates another scenario provided in an embodiment of the present application.

[0274] As shown in Figure 11, the rendering instructions for frames 1 through 3 can all be issued by Application 1. For example, consider Application 1 included in the application whitelist. Combined with the above description of enabling the XSync solution, the SF module can execute the compositing process for any of frames 1 through 3 without waiting for the next Vsync signal.

[0275] For the fourth frame, application 1 can issue a rendering instruction to render image 4a when signal V35 arrives. In addition, application 2 can also issue a rendering instruction to render image 4b ​​when signal V35 arrives.

[0276] For example, application 1 is included in the application whitelist, but application 2 is not included in the application whitelist.

[0277] In this way, for image 4a, the SF module can enable the XSync solution for fast synthesis. For image 4b, the SF module can disable the XSync solution and wait for the next Vsync signal to arrive according to the native mechanism before synthesizing the image 4b.

[0278] As an example, take the process of the SF module processing image 4a as an example.

[0279] After image 4a is rendered and queued, the SF module can enable the XSync scheme based on application 1 being included in the application whitelist. This allows the SF module to dequeue the rendered image 4a for compositing without waiting for signal V36 to arrive. In the example shown in Figure 11 , the compositing process for image 4a can be completed during the Vsync period between signals V36 and V37. This allows the display screen to capture the display image of image 4a before signal V37 arrives and then display image 4a after signal V37 arrives. This image 4a corresponds to the fourth frame displayed on the display screen.

[0280] Take the process of the SF module processing image 4b ​​as an example.

[0281] After image 4b ​​is rendered and queued, the SF module may not enable the XSync solution, as application 2 is not included in the application whitelist. For example, after image 4b ​​is rendered and queued, the SF module may wait for the next Vsync signal (e.g., signal V36). When signal V36 arrives, the SF module is already compositing image 4a. This compositing process for image 4b ​​can continue, waiting for the next Vsync signal. For example, the SF module may continue compositing the rendering results of image 4b ​​after signal V37 arrives.

[0282] It should be noted that the above example illustrates the situation where neither Application 1 nor Application 2 issues rendering instructions when Signal V36 arrives after Signal V35. Thus, after Signal V37 arrives, the buffer queue BQ2 corresponding to Application 2 still contains the unconsumed rendering result of Image 4b. Therefore, after Signal V37 arrives, the SF module can dequeue the rendering result of Image 4b ​​from Buffer Queue BQ2 for consumption.

[0283] Generally, the rendering instructions issued by the application can be continuous.

[0284] For example, in the scenario shown in Figure 11, when signal V35 arrives, application 1 issues rendering instructions for image 4a, and application 2 issues rendering instructions for image 4b. When signal V36 arrives, application 1 can continue to issue rendering instructions for image 5a. Application 2 can continue to issue rendering instructions for image 5b.

[0285] Thus, in the example shown in FIG11 , when the signal V37 arrives, the buffer queue BQ2 may not include the rendering result of the image 4 b .

[0286] As an example, FIG11 shows the corresponding rendering results in the buffer queue BQ1 and the buffer queue BQ2 at time T1 and time T2.

[0287] Here, time T1 may be the time when image 5a is rendered into the queue. Time T2 may be the time when image 5b is rendered into the queue. Time T1 and time T2 may be two moments in the Vsync period corresponding to signal V36 to signal V37.

[0288] As shown in Figure 11, at time T1, since image 5a has already been rendered and queued, the rendering result of image 5a can be stored in buffer queue BQ1 and awaiting consumption. Correspondingly, image 5b has not yet been rendered, so the rendering result of image 4b ​​can continue to be stored in buffer queue BQ2 and awaiting consumption.

[0289] At time T2, since image 5b has been rendered into the queue, the buffer queue BQ2 can store the rendering result of image 5b waiting for consumption.

[0290] Therefore, based on the scenario of the fifth frame image as shown in FIG11 and the implementation description of enabling the XSync solution in the aforementioned description, in some embodiments of the present application, in the multi-application (or window) display scenario, the SF module determines whether to trigger the XSync solution for a newly enqueued buffer based on a comprehensive determination of the available buffers in the buffer queues of multiple applications.

[0291] For example, take the 4th frame image as an example.

[0292] After the rendering result of image 4a is queued, the rendering result of image 4b ​​has not yet been completed. Thus, the configured identification fields in the BufferTX module may include: identification field 1 of buffer queue BQ1 is configured to 1, and identification field 2 of buffer queue BQ2 is configured to 0. In other words, the currently available buffer is included in buffer queue BQ1.

[0293] Therefore, the SF module can trigger the XSync solution for the newly queued Buffer in the buffer queue BQ1 (as shown in Figure 4a) according to the name of the identification field 1 configured as 1, including a package name in the application whitelist.

[0294] Take the 5th frame image as an example.

[0295] After the rendering result of image 5a is queued, the rendering result of image 5b has not yet been completed. Therefore, the configured identification fields in the BufferTX module may include: identification field 1 of buffer queue BQ1 is configured as 1, and identification field 2 of buffer queue BQ2 is configured as 1. The identification field 2 of buffer queue BQ2 is configured as 1 because the rendering result of image 4b ​​has not yet been consumed.

[0296] Therefore, the SF module may not trigger the XSync solution for the currently available buffer according to the buffer queues where the currently available buffers are located, including the buffer queue BQ2 that is not in the application whitelist.

[0297] In this way, the SF module can wait for the arrival of the next Vsync signal (such as signal V37), obtain all currently available buffers for unified synthesis processing, and obtain the corresponding image for display of the 5th frame.

[0298] For example, when signal V37 is received, the SF module can obtain the rendering results of the currently available image 5a in buffer queue BQ1 and the rendering results of image 5b in buffer queue BQ2. Based on the rendering results of image 5a and image 5b, the SF module synthesizes the rendering results to obtain the display image of frame 5 and sends it to the display driver.

[0299] When the signal V38 arrives, the display driver can display the image of the fifth frame.

[0300] As an example, refer to Figure 12. The rendering result of image 4a is shown as 1201. Correspondingly, when the electronic device displays the fourth frame of image on the display screen, it can display the display image after the synthesis processing according to the rendering result of image 4a.

[0301] In this example, application 2 can display the small window 1202 by instructing the electronic device to render and display image 4b ​​and image 5b.

[0302] In this way, based on the processing mechanism of the fourth and fifth frames of images in Figure 11, the effect of the fourth frame of image including the game screen corresponding to application 1 can be obtained as shown in Figure 12. The display of the fifth frame can simultaneously include the game screen indicated by application 1 and the small window screen indicated by application 2.

[0303] To more clearly illustrate the processing mechanism for the 4th and 5th frames in a multi-application (window) scenario, refer to Figures 13 and 14, which are another schematic diagram of the flow of module interaction provided in an embodiment of the present application. By implementing this solution, the electronic device can accurately process the 4th and 5th frames, obtaining the display effects shown in Figures 11 and 12.

[0304] FIG13 provides the processing logic within the Vsync period of the signal V35 to the signal V36.

[0305] As shown in Figure 13, the solution may include:

[0306] S1301: The touch panel receives operation OP2.

[0307] Exemplarily, the operation OP2 may be an operation input by a user for the application 2 .

[0308] For example, taking application 2 as a small window application, the operation OP2 may correspond to an operation such as dragging, sliding, or clicking a control corresponding to application 2 on the current display interface.

[0309] Through operation OP2, the user can instruct the electronic device to display the small window of application 2.

[0310] It is understandable that when the electronic device receives the operation OP2 during the running of application 1, the electronic device can simultaneously display the game screen of application 1 and the small window interface corresponding to the operation OP2 in the subsequent image display.

[0311] S1302 : The touch panel sends operation information 72 to application 2 .

[0312] For example, after receiving operation OP2, the touch panel can generate corresponding operation information 72. Similar to the process of sending operation information 71 to application 1 after operation OP1 is input, in this example, the touch panel can send the operation information 72 to application 2 through the touch driver and IMS module.

[0313] Thus, after the next Vsync signal arrives, application 1 can normally issue rendering instructions to instruct the electronic device to render the game screen. Application 2 can also issue rendering instructions to instruct the electronic device to render and subsequently display the small window.

[0314] For example, the electronic device can implement rendering, synthesis, and other processing of the current game screen (such as image 4a) through the following processing mechanism S1303a-S1313a. The electronic device can also implement corresponding processing of the small window (such as image 4b) through the following processing mechanism S1303b-S1309b.

[0315] The following explains them separately.

[0316] S1303a: Application 1 generates a rendering instruction 131. The rendering instruction 131 may also be referred to as a third rendering instruction.

[0317] For example, the application 1 may generate a rendering instruction 131 for the image 4 a after the signal V35 arrives.

[0318] S1304a, application 1 sends rendering instruction 131 to the drawing module.

[0319] S1305a: The rendering module performs rendering processing according to the rendering instruction 131 and obtains a rendering result 133. The rendering result 133 may also be referred to as a third rendering result.

[0320] S1306a: The drawing module sends the rendering result 133 to the buffer queue BQ1.

[0321] S1307a. The BufferTX module updates the identification field 1.

[0322] S1308a: The SF module confirms that the identification field 1 in the BufferTX module is updated.

[0323] For example, the processing mechanism of S1304a to S1308a may refer to S1004 to S1009 in Figure 10. The specific implementations may be referenced to each other and will not be described in detail here.

[0324] S1309a: The SF module determines that the application 1 corresponding to the available buffer is in the application whitelist.

[0325] In this example, the SF module can determine the status of all currently available buffers based on various identification fields in the BufferTX module.

[0326] For example, the SF module may determine that the currently available Buffer is only included in the buffer queue BQ1 corresponding to the identification field 1 according to the identification field configured as 1 in the BufferTX module only including the identification field 1.

[0327] In this way, the SF module can determine that the application corresponding to the currently available Buffer is in the application whitelist based on the fact that the name of the buffer queue BQ1 includes application 1 and the package name of application 1 is included in the application whitelist.

[0328] Therefore, the SF module can start the XSync solution for the currently available buffer. In other words, the SF module can consume the existing buffer in the buffer queue BQ1 without waiting for the arrival of the next Vsync signal.

[0329] S1310a: The SF module obtains the rendering result 133 from the buffer queue BQ1.

[0330] S1311a, the SF module performs synthesis processing based on the rendering result 133.

[0331] S1312a: The SF module completes the synthesis process of the rendering result 133 and obtains the display image 92. In this example, in combination with the description of FIG11 , the SF module can complete the synthesis process of the rendering result 133 after the signal V36 arrives.

[0332] S1313a, the SF module transmits the display image 92 to the display module.

[0333] In this way, after the next Vsync signal arrives, the display screen can display the image 92.

[0334] This allows for rapid display of the game screen corresponding to image 4a.

[0335] As shown in FIG. 13 , the electronic device may further process instructions related to application 2 through S1303 b - S1309 b .

[0336] S1303b: Application 2 generates a rendering instruction 132. The rendering instruction 132 may also be referred to as a second rendering instruction.

[0337] For example, the application 2 may generate a rendering instruction 132 based on the operation information 72. The rendering instruction 132 may be used to instruct the electronic device to render a small window interface. The small window interface may correspond to the image 4b ​​in the above example.

[0338] S1304b: Application 2 sends rendering instruction 132 to the drawing module.

[0339] S1305b: The drawing module performs rendering processing according to the rendering instruction 132 and obtains the rendering result 134.

[0340] As shown in Figure 13, in this example, application 1 and application 2 can generate and issue corresponding rendering instructions independently according to the arrival of signal V35. In this way, the drawing module can execute the rendering processing corresponding to application 1 and application 2 respectively through different rendering threads.

[0341] It is understandable that in different rendering processes, since the rendering contents indicated by the rendering instructions are different, the corresponding time consumption is also different. For example, the timing of obtaining the rendering result 133 by the drawing module may be earlier than the timing of obtaining the rendering result 134.

[0342] In this way, the rendering result 133 can enter the buffer queue earlier than the rendering result 134, thereby triggering the above-mentioned S1306a and subsequent processing processes.

[0343] In contrast, the drawing module may obtain the rendering result 134 through the step shown in S1305b and then continue processing according to the following steps.

[0344] S1306b: The drawing module sends the rendering result 134 to the buffer queue BQ2.

[0345] The buffer queue BQ2 may be a buffer queue configured for the application 2 in the memory of the electronic device, thereby completing the production and queuing of a new buffer of the rendering result 134 .

[0346] S1307b. The BufferTX module updates the identification field 2.

[0347] S1308b. The SF module confirms that the identification field 2 in the BufferTX module is updated.

[0348] S1309b: The SF module determines that application 2 corresponding to the available buffer is not in the application whitelist.

[0349] It is understandable that, in combination with the processing in S1311a, after the rendering result 133 corresponding to S1311a is dequeued for synthesis processing, the available buffer in the buffer queue BQ1 may be empty.

[0350] Correspondingly, the BufferTX module may adjust the identification field 1 corresponding to BQ1 to 0 after S1311a.

[0351] Therefore, after the processing of S1307b, the BufferTX module updates the identification field 2 (e.g., adjusts the identification field 2 to 1), and the identification field 1 is 0 and the identification field 2 is 1. That is, the available buffer indicated by the current BufferTX module is the buffer corresponding to BQ2 corresponding to the identification field 2.

[0352] In step S1309b, the SF module can determine, based on the indication in the BufferTX module that the available buffer is the buffer corresponding to BQ2, that the name of BQ2 corresponding to the available buffer includes Application 2, which is not in the application whitelist. Therefore, the SF module may not enable the Xsync solution for the currently available buffer. In other words, the SF module may wait for the next Vsync signal to arrive and attempt to synthesize the currently available buffer.

[0353] In this way, the processing of the fourth frame image can be completed within the Vsync period from the signal V35 to the signal V36.

[0354] Then, the electronic device may continue to perform the processing for the 4th frame image and the 5th frame image in the subsequent Vsync period as described in FIG. 14 .

[0355] As shown in Figure 14, the solution may include:

[0356] S1401a , application 1 generates rendering instruction 141 .

[0357] With reference to the example of FIG. 11 , after the signal V36 arrives, the application 1 may further generate a rendering instruction 141 , thereby instructing the electronic device to perform processing such as rendering of the image 5 a .

[0358] S1402a, application 1 sends rendering instruction 141 to the drawing module.

[0359] S1403a: The drawing module performs rendering processing according to the rendering instruction 141 and obtains the rendering result 143.

[0360] S1404a: The drawing module sends the rendering result 143 to the buffer queue BQ1.

[0361] S1405a. The BufferTX module updates the identification field 1.

[0362] S1406a: The SF module detects that the identification field 1 is updated.

[0363] S1407a: The SF module determines that the applications corresponding to the available buffers include application 2 which is not in the application whitelist.

[0364] It can be understood that, in conjunction with the description in FIG13 , before the rendering result 143 corresponding to S1404a is enqueued, in the BufferTX module, the rendering result 133 has been dequeued, so the identification field 1 is 0. The rendering result 134 is waiting for the next Vsync signal to arrive and has not yet been dequeued, so the identification field 2 is 1.

[0365] In this way, after the rendering result 143 corresponding to S1404a is queued, the BufferTX module may include: identification field 1 is 1, and identification field 2 is 1.

[0366] The SF module may determine that a new buffer is queued according to S1406a, and then determine whether to trigger the Xsync solution according to the corresponding steps of S1407a.

[0367] Exemplarily, the SF module may determine, based on the identification field in the BufferTX module, that the currently available buffers may include one buffer in the buffer queue BQ1 and one buffer in the buffer queue BQ2.

[0368] The SF module can determine, based on the application 1 corresponding to the name of the buffer queue BQ1, that the buffer corresponding to the buffer queue BQ1 is a buffer generated by an application in the application whitelist.

[0369] The SF module can determine, based on the application 2 corresponding to the name of the buffer queue BQ2, that the buffer corresponding to the buffer queue BQ2 is not a buffer generated by an application in the application whitelist.

[0370] In this way, since there is a buffer produced by an application not in the application whitelist among the currently available buffers, the SF module can not trigger the XSync solution. Correspondingly, the SF module can wait for the next Vsync signal to arrive and then consume the two produced buffers.

[0371] In addition, as shown in FIG13 , during the Vsync period from signal V35 to signal V36 , the electronic device may further perform the following processing:

[0372] S1401b: Application 2 generates a rendering instruction 142. The rendering instruction 142 may be used to instruct the rendering of the image 5b.

[0373] S1402b: Application 2 sends rendering instruction 142 to the drawing module.

[0374] S1403b: The drawing module performs rendering processing according to the rendering instruction 142 and obtains the rendering result 144.

[0375] S1404b: The drawing module sends the rendering result 144 to the buffer queue BQ2.

[0376] S1405b. The BufferTX module updates the identification field 2.

[0377] S1406b: The SF module detects that the identification field 2 is updated.

[0378] S1407b: The SF module determines that the applications corresponding to the available buffers include application 2 which is not in the application whitelist.

[0379] Thus, through the implementation of the solution from S1401b to S1407b, the electronic device can respond to the rendering instruction 142 of application 2 and perform corresponding processing within the Vsync period. For example, the drawing module can complete the enqueueing of the corresponding rendering result 144 into the buffer queue BQ2.

[0380] In some embodiments of the present application, for example, buffer queue BQ2 is configured with one valid buffer. Thus, buffer queue BQ2 can contain at most one available buffer at a time. Since buffer queue BQ2 already has unconsumed rendering result 134, rendering result 144 can overwrite rendering result 134 after it is queued.

[0381] Correspondingly, after S1404 b , the buffer queue BQ2 may include the rendering result 144 .

[0382] Since there is a produced buffer in the buffer queue BQ2, the identification field 2 in the BufferTX module can be 1.

[0383] In the present application, the SF module may perform a judgment such as S1407a or S1407b each time it detects a new update of the identification field to determine whether to trigger the XSync solution.

[0384] In S1407b, the SF module may determine that the currently available buffers include the buffer generated by application 2, but application 2 is not included in the application whitelist, so the SF module does not trigger the XSync solution.

[0385] Then, after the next Vsync signal (such as signal V37) arrives, the electronic book can execute S1408-S1412.

[0386] S1408 , the display module displays according to the sent image 92 .

[0387] Thus, the display screen of the electronic device can display the fourth frame image corresponding to the sent image 92. The fourth frame image can correspond to the rendering result of the image 4a shown in FIG11, that is, the game screen.

[0388] Combined with the description in S1407b, since all currently produced buffers need to wait for the next Vsync signal to arrive before being dequeued for consumption, after the signal V37 arrives, the solution may also include:

[0389] S1409 , the SF module obtains the rendering result 143 from the buffer queue BQ1 .

[0390] S1410 , the SF module obtains the rendering result 144 from the buffer queue BQ2 .

[0391] S1411, the SF module performs synthesis processing based on the rendering results 143 and 144 to obtain the display image 93.

[0392] S1412. The SF module transmits the display image 93 to the display module.

[0393] In this way, the image 93 can include the screen content that application 1 and application 2 instruct the electronic device to render after the signal V36 arrives. For example, the image 93 can include image 5a (such as a game screen) and image 5b (such as a small window screen).

[0394] Thus, as shown in S1413, after the next Vsync signal (such as signal V38) arrives, the display module can display according to the display image 93. In this way, the complete fifth frame image can be displayed on the display screen.

[0395] It should be noted that, in some other embodiments of the present application, the electronic device may further be configured with a switch button for enabling the Xsync solution. Through the switch button of the Xsync solution, the effectiveness of the Xsync solution can be generally controlled.

[0396] Exemplarily, when the switch button of the Xsync solution is in the on state, the electronic device (such as the SF module in the electronic device) can determine whether the currently available Buffer uses the Xsync solution for fast synthesis processing according to the solution provided in Figure 10 or Figure 13 or Figure 14. Correspondingly, when the switch button of the Xsync solution is in the off state, the electronic device (such as the SF module in the electronic device) can trigger the dequeue synthesis of the existing Buffer based on the arrival of the next Vsync signal according to the native logic.

[0397] In some embodiments, the switch button of the Xsync solution can be turned on or off manually by the user. In other embodiments, the switch button of the Xsync solution can be turned on or off by the electronic device itself.

[0398] It is understandable that the electronic device provided in the embodiment of the present application includes a hardware structure and / or software module for performing each function in order to realize the above functions. Those skilled in the art should easily appreciate that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiment of the present application.

[0399] The embodiment of the present application can divide the functional modules of the above-mentioned electronic device according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0400] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of each functional module. Those skilled in the art should easily appreciate that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0401] The above-mentioned integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. In actual implementation, there may be other division methods.

[0402] For example, Figure 15 shows a schematic diagram of the components of an electronic device 1500. As shown in Figure 15, the electronic device 1500 may include: a processor 1501 and a memory 1502. In this example, the electronic device 1500 may also include a display screen 1503. The memory 1502 is used to store computer-executable instructions. For example, in some embodiments, when the processor 1501 executes the instructions stored in the memory 1502, the electronic device 1500 may execute any of the methods described in the above embodiments, thereby enabling the display screen 1503 to display relevant images more quickly.

[0403] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0404] Figure 16 shows a schematic diagram of the composition of a chip system 1600. The chip system 1600 may include: a processor 1601 and a communication interface 1602, which are used to support related devices to implement the functions involved in the above embodiments. In one possible design, the chip system also includes a memory for storing program instructions and data necessary for electronic devices. The chip system can be composed of chips, or it can include chips and other discrete devices. It should be noted that in some implementations of the present application, the communication interface 1602 may also be referred to as an interface circuit.

[0405] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0406] The functions, actions, operations, steps, etc. in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, they can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that can be integrated with the medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0407] Although the present application has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely illustrative of the present application as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, the present application is intended to include such modifications and variations as fall within the scope of the claims of the present application and their equivalents.

Claims

1. An image processing method, characterized in that: The method is applied to an electronic device, wherein the electronic device is provided with a display screen and a first application is also installed in the electronic device; The method comprises: After generating the first synchronous Vsync signal, receiving a first rendering instruction issued by the first application, wherein the first rendering instruction is used to instruct the electronic device to perform rendering processing on the Nth frame image; According to the first rendering instruction, obtaining a first rendering result; Perform synthesis processing according to the first rendering result to obtain a first image to be displayed; the first image to be displayed corresponds to the Nth frame image; After generating the second Vsync signal, controlling the display screen to display the Nth frame image according to the first display image; The second Vsync signal is generated after the first Vsync signal; after the first rendering result is obtained and before the synthesis processing is performed according to the first rendering result, no Vsync signal is generated.

2. The method according to claim 1, characterized in that Before generating the first Vsync signal, the method further includes: A first operation is received, where the first operation is an operation on the first interface, and the first interface is a display interface of the first application.

3. The method according to claim 2, characterized in that The method further comprises: Generate first operation information according to the first operation; the first operation information indicates the operation type of the first operation and the location information of the first operation; Sending the first operation information to the first application; The first operation information corresponds to the first rendering instruction.

4. The method according to any one of claims 1 to 3, characterized in that The electronic device is configured with a first buffer queue, and the first buffer queue corresponds to the first application; After obtaining the first rendering result according to the first rendering instruction, the method further includes: storing first information corresponding to the first rendering result in the first buffer queue; The first information includes any one of the following: The image information of the first rendering result; the identifier of the buffer storing the first rendering result; and the first file identifier indicating the storage location of the first rendering result in the memory of the electronic device.

5. The method according to claim 4, characterized in that After sending the first information corresponding to the first rendering result to the first buffer queue, the method further includes: The first identification field corresponding to the first buffer queue is configured as a second value; the first identification field being the second value indicates that new information is queued in the first buffer queue.

6. The method according to claim 5, characterized in that Before sending the first information corresponding to the first rendering result to the first buffer queue, the first identification field is configured as a first value; the first identification field being the first value indicates that the available information in the first buffer queue is empty.

7. The method according to claim 5 or 6, characterized in that: The performing synthesis processing according to the first rendering result to obtain a first display image includes: acquiring the first information from the first buffer queue, The first rendering result indicated by the first information is synthesized to obtain the first display image.

8. The method according to claim 7, characterized in that The electronic device is configured with an application whitelist, the application whitelist including at least one application information, and the application information of different applications is different; Before acquiring the first information from the first buffer queue, the method further includes: It is determined that application information corresponding to all buffer queues whose identification fields are the second value is included in the application whitelist.

9. The method according to claim 8, characterized in that The application information corresponding to the buffer queues whose identification fields are all determined to be the second value is included in the application whitelist, including: It is determined that application information corresponding to the first buffer queue is included in the application whitelist.

10. The method according to claim 9, characterized in that The application information includes the package name of the application; The application information corresponding to the first buffer queue includes: a package name of a first application corresponding to the first buffer queue; The determining that the information of the first buffer queue is included in the application whitelist includes: A package name of a first application corresponding to the first buffer queue is determined, and included in the application whitelist.

11. The method according to claim 10, characterized in that The method further comprises: According to the name of the first buffer queue, a package name of the first application corresponding to the first buffer queue is determined.

12. The method according to any one of claims 8 to 11, characterized in that The electronic device also has a second application installed therein; The method further comprises: After generating the third Vsync signal, receiving a second rendering instruction sent by the second application, wherein the second rendering instruction is used to instruct the electronic device to perform at least a portion of the Mth frame of image rendering processing; According to the second rendering instruction, obtaining a second rendering result; After generating the fourth Vsync signal, performing synthesis processing according to the second rendering result to obtain a second display image; the second display image corresponds to the Mth frame image; After generating the fifth Vsync signal, controlling the display screen to display the Mth frame image according to the second display image; The fourth Vsync signal is generated after acquiring the second rendering result.

13. The method according to claim 12, characterized in that After obtaining the second rendering result according to the second rendering instruction, the method further includes: It is determined that the application information of the second application is not included in the application whitelist.

14. The method according to claim 12 or 13, characterized in that After obtaining the second rendering result, the method further includes: The second information of the second rendering result is stored in a second buffer queue, where the second buffer queue is a buffer queue corresponding to the second application.

15. The method according to claim 14, characterized in that The method further comprises: After generating the third Vsync signal, receiving a third rendering instruction sent by the first application, wherein the second rendering instruction is used to instruct the electronic device to perform at least a portion of the Mth frame of image rendering processing; According to the third rendering instruction, obtaining a third rendering result; Storing third information of the third rendering result in the first buffer queue; The third rendering result is obtained before the fourth Vsync signal is generated.

16. The method according to claim 15, characterized in that After generating the fourth Vsync signal, performing synthesis processing according to the second rendering result to obtain a second display image includes: After generating the fourth Vsync signal, acquiring the third rendering result according to the third information of the first buffer queue; acquiring the second rendering result according to the second information of the second buffer queue; The second rendering result and the third rendering result are synthesized to obtain the second display image.

17. The method according to claim 15 or 16, characterized in that After storing the third information of the third rendering result in the first buffer queue, the method further includes: It is determined that application information corresponding to a buffer queue having at least one identification field with a second value is not included in the application whitelist.

18. The method according to claim 17, characterized in that The determining that application information corresponding to the buffer queue having at least one identification field with the second value is not included in the application whitelist includes: Determine that application information corresponding to the second buffer queue is not included in the application whitelist.

19. An electronic device, characterized in that: The electronic device comprises: a memory and one or more processors; the memory and the processor are coupled; The memory is used to store computer program codes, and the computer program codes include computer instructions. When the processor executes the computer instructions, the electronic device executes the method as claimed in any one of claims 1 to 18.

20. A chip system, characterized in that: The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive a signal from a memory of the electronic device and send the signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1-18.

Images