Image processing method and electronic equipment

Abstract

The embodiment of the invention provides an image processing method and electronic equipment, and relates to the technical field of image processing. According to the method, the display delay of the corresponding response content on the display screen after the user inputs the operation can be effectively reduced, and the chirality is improved. The method comprises the steps that after a first Vsync signal is generated, a first rendering instruction issued by a first application is received; and obtaining a first rendering result according to the first rendering instruction. And performing synthesis processing according to the first rendering result to obtain a first sending display image. The first sending and displaying image corresponds to the Nth frame image. And after the second Vsync signal is generated, controlling the display screen to display the Nth frame of image according to the first sending display image. Wherein the second Vsync signal is generated after the first Vsync signal. After the first rendering result is obtained, the generated Vsync signal does not exist before synthesis processing is carried out according to the first rendering result.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of image processing technology, and in particular, to an image processing method and an electronic device. Background Art

[0002] Currently, before an electronic device displays an image, it can perform processing such as rendering and synthesizing the image.

[0003] Exemplarily, after an application issues a rendering instruction, the electronic device can perform rendering processing according to the rendering instruction to obtain a corresponding rendering result. The electronic device can also perform synthesis processing on the rendering result when the next synchronization signal (such as a Vsync signal) arrives, so as to obtain a display image of this frame of image. The display screen of the electronic device can display according to the display image when the next Vsync signal arrives. Thus, the display of this frame of image on the display screen can be realized.

[0004] In this way, there is a delay of multiple Vsync signal cycles from when the application issues a rendering instruction to when the corresponding frame of image is displayed on the display screen. Some rendering instructions issued by applications can include responses to user input operations. This causes the user to wait a long time to see the response to the input operation on the display screen after the input operation. This can lead to a problem of poor followability. Summary of the Invention

[0005] The present application provides an image processing method and an electronic device, which can effectively reduce the delay in the display of the corresponding response content on the display screen after a user's input operation and improve the followability.

[0006] To achieve the above technical objectives, the present application adopts the following technical solutions:

[0007] In a first aspect, an image processing method is provided. This method is applied to an electronic device configured with a display screen and having a first application installed. The method includes: after generating a first synchronization Vsync signal, receiving a first rendering instruction issued by the first application, where the first rendering instruction is used to instruct the electronic device to perform rendering processing on the Nth frame of image. Obtaining a first rendering result according to the first rendering instruction. Performing synthesis processing according to the first rendering result to obtain a first display image. The first display image corresponds to the Nth frame of image. After generating a second Vsync signal, controlling the display screen to display the Nth frame of image according to the first display image. Wherein, the second Vsync signal is generated after the first Vsync signal. After obtaining the first rendering result and before performing synthesis processing according to the first rendering result, no generated Vsync signal exists.

[0008] Based on this, after obtaining the rendering result, the electronic device can directly perform the composition process according to the rendering result without waiting for the next Vsync signal to arrive. Thus, the time consumption of at least one Vsync cycle is saved. In this way, after receiving the rendering instruction sent by the first application, the electronic device can more quickly display the corresponding interface on the display screen, thereby significantly improving the followability.

[0009] Optionally, before generating the first synchronous Vsync signal, the method further includes: receiving a first operation, where the first operation is an operation on the first interface, and the first interface is the display interface of the first application. In this way, the content indicated by the first rendering instruction to be rendered may include the response content to the first operation.

[0010] Optionally, the method further includes: generating first operation information according to the first operation. The first operation information indicates the operation type of the first operation and the position information of the first operation. Sending the first operation information to the first application.

[0011] Optionally, a first buffer queue is configured in the electronic device, and the first buffer queue corresponds to the first application. After obtaining the first rendering result according to the first rendering instruction, the method further includes: sending the first information corresponding to the first rendering result to the first buffer queue. The first information includes any one of the following: the image information of the first rendering result. The identifier of the buffer (Buffer) storing the first rendering result. The first file identifier, where the first file identifier indicates the storage location of the first rendering result in the memory of the electronic device. Thus, a logical implementation for passing the rendering result backward is provided. For example, this logical implementation may be based on the buffer queue configured in the electronic device. After obtaining the rendering result, the corresponding Buffer can be enqueued to the buffer queue. Thus, when the composition process of the rendering result needs to be performed subsequently, the corresponding Buffer can be dequeued.

[0012] Optionally, after sending the first information corresponding to the first rendering result to the first buffer queue, the method further includes: configuring the first identification field corresponding to the first buffer queue to a second value. The first identification field being the second value indicates that there is new information enqueued in the first buffer queue.

[0013] Optionally, before sending the first information corresponding to the first rendering result to the first buffer queue, the first identification field is configured to a first value. The first identification field being the first value indicates that the available information in the first buffer queue is empty.

[0014] In this application, a solution implementation for managing a buffer queue is provided. For example, each buffer queue can be configured with a corresponding identification field. When the identification field has different values, it correspondingly indicates whether there is an available Buffer in the buffer queue, or represents the number of available Buffers in the buffer queue, etc. Thus, according to the change of different identification field values, the monitoring of new Buffer enqueue in each buffer queue can be achieved.

[0015] Optionally, performing a composition process based on the first rendering result to obtain a first image for display includes: obtaining the first information from the first buffer queue, and performing a composition process on the first rendering result indicated by the first information to obtain the first image for display.

[0016] Optionally, an application whitelist is configured in the electronic device. The application whitelist includes at least one application information, and the application information of different applications is different. Before obtaining the first information from the first buffer queue, the method further includes: determining that the application information corresponding to all buffer queues with the identification field being a second value is included in the application whitelist.

[0017] Optionally, determining that the application information corresponding to all buffer queues with the identification field being a second value is included in the application whitelist includes: determining that the application information corresponding to the first buffer queue is included in the application whitelist.

[0018] Optionally, the application information includes the package name of the application. The information of the first buffer queue includes: the package name of the first application corresponding to the first buffer queue. Determining that the information of the first buffer queue is included in the application whitelist includes: determining that the package name of the first application corresponding to the first buffer queue is included in the application whitelist.

[0019] Optionally, the method further includes: determining the package name of the first application corresponding to the first buffer queue according to the name of the first buffer queue.

[0020] Thus, a mechanism for triggering early composition processing is provided. In this example, an application whitelist of applications that need to trigger early composition processing can be pre-configured in the electronic device. In this way, when it is determined that there is a new Buffer enqueue in the buffer queue of an application in the whitelist, the Buffer dequeue can be composited and displayed in advance. Furthermore, the rendering instructions of the applications in the whitelist can be reflected on the display screen faster for display.

[0021] Optionally, a second application is also installed in the electronic device. The method further includes: after generating a third Vsync signal, receiving a second rendering instruction sent by the second application, where the second rendering instruction is used to instruct the electronic device to perform at least partial rendering processing on the M-th frame of image. According to the second rendering instruction, obtain a second rendering result. After generating a fourth Vsync signal, perform a synthesis process according to the second rendering result to obtain a second image for display. The second image for display corresponds to the M-th frame of image. After generating a fifth Vsync signal, control the display screen to display the M-th frame of image according to the second image for display. Wherein, the fourth Vsync signal is generated after obtaining the second rendering result.

[0022] Optionally, after obtaining the second rendering result according to the second rendering instruction, the method further includes: determining that the application information of the second application is not included in the application whitelist.

[0023] Thus, for an application not in the application whitelist, the electronic device may not trigger the XSync scheme for early synthesis. Furthermore, it is ensured that within the same Vsync cycle, the electronic device (such as the SF module of the electronic device) does not trigger multiple synthesis processes.

[0024] Optionally, after obtaining the second rendering result, the method further includes: storing second information of the second rendering result into a second buffer queue, where the second buffer queue is the buffer queue corresponding to the second application.

[0025] Optionally, the method further includes: after generating the third Vsync signal, receiving a third rendering instruction sent by the first application, where the second rendering instruction is used to instruct the electronic device to perform at least partial rendering processing on the M-th frame of image. According to the third rendering instruction, obtain a third rendering result. Store third information of the third rendering result into the first buffer queue. Wherein, the third rendering result is obtained before the fourth Vsync signal is generated.

[0026] Optionally, after generating the fourth Vsync signal, performing a synthesis process according to the second rendering result to obtain a second image for display includes: after generating the fourth Vsync signal, obtaining the third rendering result according to the third information in the first buffer queue. Obtaining the second rendering result according to the second information in the second buffer queue. Performing a synthesis process on the second rendering result and the third rendering result to obtain the second image for display.

[0027] Optionally, after storing the third information of the third rendering result into the first buffer queue, the method further includes: determining that there is at least one application information corresponding to a buffer queue with an identification field value of a second value and not included in the application whitelist.

[0028] Optionally, determining that the application information corresponding to at least one buffer queue with the identification field being the second value does not include in the application whitelist includes: determining that the application information corresponding to the second buffer queue does not include in the application whitelist.

[0029] In a second aspect, the present application further provides an electronic device, which includes: a memory and one or more processors. The memory and the processor are coupled. In some implementations, the electronic device may also be configured with a display screen. Wherein, the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the technical solutions provided in the first aspect and any possible implementation thereof. In this way, the display screen of the electronic device can perform image display faster.

[0030] In a third aspect, the present application further provides a chip system, which is applied to an electronic device; the chip system may include one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected through a line. The interface circuit is used to receive a signal from the memory of the electronic device and send the signal to the processor. The signal includes the computer instructions stored in the memory. When the processor executes the above computer instructions, the electronic device executes the technical solutions provided in the first aspect and any possible implementation thereof.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium, which includes computer instructions. When the computer instructions run on an electronic device, the electronic device executes the technical solutions provided in the first aspect and any possible implementation thereof.

[0032] In a fifth aspect, the present application further provides a computer program product. When the computer program product runs on a computer, the computer executes the technical solutions provided in the first aspect and any possible implementation thereof.

[0033] It can be understood that the solutions provided in the second aspect to the fifth aspect of the present application can respectively correspond to the first aspect and any possible design thereof. Therefore, the beneficial effects that can be achieved are similar and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a logical schematic diagram of an interface interaction and display;

[0035] Figure 2 It is a logical schematic diagram of the internal module interaction of an electronic device;

[0036] Figure 3 It is a logical schematic diagram of a multi-frame image processing process;

[0037] Figure 4 Schematic logical diagram of the multi-frame image processing process after the solution provided by the embodiment of the present application becomes effective;

[0038] Figure 5 Schematic diagram of the composition of an electronic device provided by the embodiment of the present application;

[0039] Figure 6 Schematic diagram of the composition of an electronic device provided by the embodiment of the present application;

[0040] Figure 7 Schematic diagram of the interaction between modules provided by the embodiment of the present application;

[0041] Figure 8 Schematic diagram of the interaction between modules provided by the embodiment of the present application;

[0042] Figure 9 Schematic diagram of the interaction between modules provided by the embodiment of the present application;

[0043] Figure 10 Schematic flowchart of the interaction between modules provided by the embodiment of the present application;

[0044] Figure 11 Schematic logical diagram of the multi-frame image processing process provided by the embodiment of the present application;

[0045] Figure 12 Schematic diagram of the display effect of different frame images provided by the embodiment of the present application;

[0046] Figure 13 Schematic flowchart of the interaction between modules provided by the embodiment of the present application;

[0047] Figure 14 Schematic flowchart of the interaction between modules provided by the embodiment of the present application;

[0048] Figure 15 Schematic diagram of the composition of an electronic device provided by the embodiment of the present application;

[0049] Figure 16 Schematic diagram of the composition of a chip system provided by the embodiment of the present application. Detailed implementation manners

[0050] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of this embodiment, unless otherwise stated, the meaning of "a plurality" is two or more.

[0051] Some application programs installed in an electronic device can display images through the display screen of the electronic device. Exemplarily, these application programs can include game applications, etc. Thus, when a game application is running, the game screen can be displayed through the display screen of the electronic device.

[0052] Exemplarily, taking the electronic device as a mobile phone, and there is an Application 1 installed in the electronic device, and this Application 1 is a game application as an example.

[0053] Reference Figure 1 , the electronic device can display the icon of Application 1 on the main interface. The user can input an operation 101 (such as a click operation) on the icon of Application 1 to instruct the electronic device to run this Application 1.

[0054] In response to the user's operation 101, the electronic device can run Application 1. Thereafter, Application 1 can issue a rendering instruction to instruct the electronic device to render and display the game screen.

[0055] In an example such as Figure 1 , after Application 1 runs, the electronic device can display an interface 102 on the display screen.

[0056] Taking the game screen of this Application 1 as an example of including continuously displayed image screens. Then, the first frame image to the fifth frame image and subsequent images as shown in Figure 1 can be sequentially displayed on the interface 102 of the electronic device. Thus, through the display of continuous images, the user can obtain the perception of continuous game screens.

[0057] It should be noted that in the embodiments of this application, the first frame to the fifth frame can be any continuous five frames during the game running process. This first frame does not necessarily refer to the first image displayed after the game runs.

[0058] Combined with Figure 2 , an example is given for the internal processing mechanism of each frame of image (such as the first frame image).

[0059] In an example such as Figure 2 , the electronic device can be configured with a drawing module 21, a buffer queue 22, a composition module 23, and a display screen 24.

[0060] Among them, the drawing module 21 is used to perform image rendering according to the rendering instruction issued by the application program (such as Application 1).

[0061] The buffer queue 22 can be used to temporarily store the rendering results obtained after image rendering.

[0062] The composition module 23 can be used to obtain the rendering result from the buffer queue 22 when the pre-configured timing arrives in the electronic device, and process the rendering result (such as composition processing). The composition module 23 can also be used to send the image data after composition processing to the display screen 24 so that the display screen 24 can display the image accordingly.

[0063] In Figure 2 the example, an interaction example among various modules during the processing and display of the first frame of image is provided.

[0064] As Figure 2 shown, the application 1 can send a rendering instruction 201 to the drawing module 21. For example, the rendering instruction 201 can be used to instruct the drawing module 21 to draw the first frame of image.

[0065] The drawing module 21 can call a component with image rendering capabilities in the electronic device (such as a graphics processing unit (GPU)) to perform image rendering according to the rendering instruction 201. Thus, the drawing module 21 can obtain the rendering result 202 of the first frame of image corresponding to the rendering instruction 201.

[0066] The drawing module 21 can store the rendering result 202 of the first frame of image into the buffer queue 22 to prepare for the subsequent display of the first frame of image.

[0067] The composition module 23 can obtain the stored image from the buffer queue 22 when the pre-configured timing arrives. Exemplarily, the arrival of the pre-configured timing can correspond to the arrival of a vertical synchronization (Vsync) signal generated by the electronic device. In this way, the composition module 23 can obtain the rendering result 202 from the buffer queue 22 according to the arrival of the Vsync signal.

[0068] After obtaining the rendering result 202, the composition module 23 can perform composition processing on the rendering result 202 to obtain the image 203 to be sent for display of the first frame of image.

[0069] Thus, the composition module 23 can send the image 203 to be sent for display to the display screen 24 to realize the display of the first frame of image.

[0070] It can be understood that for the processing and display processes of other frames of images (such as the second frame to the fifth frame, etc.), the processing mechanism of the first frame of image as Figure 2 shown can be referred to.

[0071] It should be noted that in some cases, the timing for the application program (such as application 1) to issue the rendering instruction 201 and the timing for the display screen 24 to display can also be controlled based on the Vsync signal. Taking the Vsync signal including signals V31 to V36 as an example.

[0072] Reference Figure 3 After the signal V31 arrives, Application 1 can issue a rendering instruction for the first frame of the image (such as rendering instruction 201). Correspondingly, after the signal V31 arrives, the rendering module 21 can perform the rendering process for the first frame of the image and complete the enqueue of the rendering result 202 of the first frame of the image into the buffer queue 22.

[0073] Next, when the signal V32 arrives, the synthesis module 23 can obtain the already enqueued rendering result 202 from the buffer queue 22. The synthesis module 23 can perform a synthesis process on the rendering result 202 and complete the synthesis process for the first frame of the image before the next Vsync signal (such as signal V33) arrives. Thus, before the signal V33 arrives, the display screen 24 can receive the display image 204 of the first frame of the image. In this application, the signal V32 can also be referred to as the first Vsync signal.

[0074] In this way, the display screen 24 can display the first frame of the image according to the display image 204 after the signal V33 arrives.

[0075] It can be understood that each module component in the electronic device can perform the processing of subsequent frames of the image according to a similar processing mechanism as described above.

[0076] Exemplarily, Figure 3 processing examples of the second frame of the image and the third frame of the image are also provided.

[0077] For the second frame of the image, Application 1 can issue a corresponding rendering instruction after the signal V32 arrives. Correspondingly, after the signal V32 arrives, the rendering module 21 can start the rendering process for the second frame of the image. In this example, the rendering duration of the second frame of the image can be greater than one Vsync cycle. Here, one Vsync cycle can be the time difference between the arrivals of two adjacent Vsync signals.

[0078] In this way, when the signal V33 arrives, the second frame of the image cannot be put into the buffer queue 22 in time because the rendering cannot be completed. Correspondingly, when the signal V33 arrives, the synthesis module 23 cannot obtain the rendering result of the second frame of the image from the buffer queue 22 for synthesis processing. Accordingly, the display screen will not obtain the display image of the second frame of the image before the next Vsync signal (such as signal V34) arrives. Thus, the display screen 24 cannot display the second frame of the image during the display cycle between this signal V34 and signal V35. For example, the display screen 24 can continue to display the first frame of the image during the display cycle between this signal V34 and signal V35.

[0079] In Figure 3In the example, the second frame image can be rendered and queued before the signal V34 arrives. In this way, after the signal V34 arrives, the synthesis module 23 can obtain the rendering result of the second frame from the buffer queue 22 for synthesis processing. Before the signal V35 arrives, the display screen 24 can receive the display image of the second frame. Thus, the display screen 24 can switch to display the second frame image when the signal V35 arrives.

[0080] Figure 3 The processing and display mechanism of the third frame image is also shown. In this Figure 3 example, the rendering instruction of the third frame image can be issued by the application 1 after the signal V34 arrives, and the rendering module 21 performs rendering processing to obtain the corresponding rendering result. This rendering result can be placed in the buffer queue 22 for consumption before the signal V35 arrives. In this way, the synthesis module 23 can obtain the rendering result of the third frame image from the buffer queue 22 after the signal V35 arrives for synthesis processing to obtain the corresponding display image. The display image of the third frame image can be transmitted to the display screen 24 before the signal V36 arrives. Thus, the display screen 24 can switch to display the third frame image when the signal V36 arrives.

[0081] It can be seen that in this Figure 3 example, since the rendering duration of the second frame image is greater than one Vsync period, the display screen 24 finally displays the first frame image continuously in two display cycles.

[0082] It should be noted that in some cases, the rendering instructions issued by the application 1 can be related to the operations already input by the user. For example, the user inputs an operation to release skill A on the game screen. Correspondingly, the rendering instructions issued by the application 1 can instruct the electronic device to render and display the corresponding screen effect after releasing skill A.

[0083] Taking the application 1 receiving the operation of releasing skill A input by the user between the signal V31 and the signal V32 as an example. In this way, in the second frame image, it can include the corresponding screen effect after releasing skill A. Obviously, the earlier the electronic device displays the second frame image, the earlier the user can see the corresponding screen effect after releasing skill A on the interface, and thus a better follow-up experience can be obtained.

[0084] However, combined with Figure 3In the example in [reference], since the rendering duration of the second frame image is longer than one Vsync period, the second frame image that should have been displayed during the display period from signal V34 to signal V35 is delayed to the next display period (such as from signal V35 to signal V36). In this way, after the user input operation, there is a delay of at least 3 Vsync periods (such as the Vsync periods between signal V32 and signal V35) from the start of rendering of the second frame image to its display on the display screen. Correspondingly, the user cannot timely see the visual effect of the operation. There are also relatively large delays of varying degrees in the display of other frame images. Thus, the problem of poor followability in the existing solution is manifested. In this application, the visual effect corresponding to the operation can also be referred to as the response content.

[0085] To solve the above problems, the technical solution provided in the embodiments of this application enables the synthesis module to obtain and synthesize the enqueued image data in advance without being limited to the arrival of the next Vsync signal. As a result, the display screen can obtain the synthesis result (such as the image to be sent for display) earlier for display.

[0086] Exemplarily, referring to Figure 4 , after the solution provided in the embodiments of this application becomes effective, Figure 3 the following is an example of the processing and display logic of each frame image in the shown scenario.

[0087] As Figure 4 shown, after the solution provided in the embodiments of this application becomes effective, for the first frame image, Application 1 can issue the corresponding rendering instruction after the arrival of signal V31. The rendering module can perform the rendering process of the first frame image to obtain the rendering result and enqueue it. This process is similar to the implementation of the current solution shown in Figure 3 .

[0088] Different from the implementation in the current solution, in the example such as Figure 4 , the synthesis module 23 does not need to wait for the arrival of the next Vsync signal to obtain data from the buffer queue 22 for subsequent processing. Correspondingly, the synthesis module 23 can directly obtain the enqueued data from the buffer queue 22 when there is new data available in the buffer queue 22. For example, the synthesis module 23 can obtain the rendering result of the newly enqueued first frame image from the buffer queue 22. Correspondingly, the synthesis module 23 can perform synthesis processing on the rendering result of the first frame image and send the display image of the first frame image obtained by the synthesis processing to the display screen 24 before the arrival of signal V32. Thus, the display screen 24 can display the received first frame image when the next Vsync signal (such as signal V32) arrives.

[0089] Thus, compared with Figure 3In the existing solution shown, in the solution provided by this application, the display timing of the first frame image is advanced from the display period between the signals V33 and V34 shown in Figure 3 to the display period between the signals V32 and V33. In this way, from the start of rendering the first frame image to its display on the display screen, there is only a delay of 1 Vsync period. This makes the display of the first frame image more timely.

[0090] Based on a similar processing mechanism for the first frame, the followability corresponding to other frame images can also be significantly improved.

[0091] Taking the second frame image as an example. As shown in Figure 4 , Application 1 can issue a rendering instruction for the second frame image after the signal V32 arrives. Correspondingly, similar to the scenario example in Figure 3 , the rendering duration of the second frame image can be greater than one Vsync period. For example, after the signal V33 arrives and before the signal V34 arrives, the rendering of the second frame image can be queued. In this application, the synthesis module 23 can directly obtain the rendering result of the second frame image from the buffer queue 22 for synthesis processing without waiting for the next Vsync signal (such as the signal V34) to arrive after the second frame image is queued for rendering. Thus, before the signal V34 arrives, the display screen 24 can obtain the image to be displayed for the second frame image. Correspondingly, after the signal V34 arrives, the display screen 24 can display the second frame image.

[0092] Thus, the display timing of the second frame image is advanced from the display period between the signals V35 and V36 shown in Figure 3 to the display period between the signals V34 and V35. In this way, even if the rendering duration of the second frame image is long, from the start of rendering the second frame image to its display on the display screen, there is only a delay of 2 Vsync periods. This makes the display of the second frame image more timely. Furthermore, the user can see the screen effect after releasing Skill A earlier, thereby significantly enhancing the followability experience during the operation of Application 1.

[0093] The solution provided by the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0094] It should be noted that the solution provided in the embodiments of the present application can be applied to an electronic device. The electronic device may include at least one of a mobile phone, a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, or a smart city device. The embodiments of the present application do not impose special restrictions on the specific type of the electronic device. In the present application, the electronic device may be configured with a display screen for image display.

[0095] In some embodiments, the electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) connector, a charging management module, a power management module, a battery, antenna 1, antenna 2, a mobile communication module, a wireless communication module, an audio module, a speaker, a receiver, a microphone, a headphone interface, a sensor module, a button, a motor, an indicator, a camera module, a display screen, and a subscriber identification module (SIM) card interface, etc. The sensor module may include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0096] The processor may include one or more processing units. For example, the processor may include an application processor (AP), a modem processor (Modem), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor (BP or BBP), and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0097] The processor may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0098] A memory may also be set in the processor for storing instructions and data. In some embodiments, the memory in the processor may be a cache memory. This memory can save the instructions or data that the processor has used or uses frequently. If the processor needs to use this instruction or data, it can directly call it from this memory. This avoids repeated accesses, reduces the waiting time of the processor, and thus improves the efficiency of the system.

[0099] It should be noted that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than the above examples, or combine certain components, or split certain components, or have different component arrangements. Each component may be implemented in hardware, software, or a combination of software and hardware.

[0100] In some embodiments, based on the above composition, the electronic device may implement the display function through a GPU, a display screen, an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor may include one or more GPUs, which execute program instructions to generate or change display information.

[0101] The display screen is used to display images, videos, etc. The display screen includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include one or more display screens.

[0102] The internal memory can be used to store computer-executable program code, and the executable program code includes instructions. The internal memory can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as game applications, etc.). The data storage area can store data created during the use of the electronic device (such as image data, audio data, phone book, etc.). In addition, the internal memory can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor executes various functional methods or data processing of the electronic device by running the instructions stored in the internal memory and / or the instructions stored in the memory provided in the processor.

[0103] The touch sensor, also known as the "touch control device". The touch sensor can be arranged on the display screen, and the touch screen, also known as the "touch control screen", is composed of the touch sensor and the display screen. The touch sensor is used to detect touch operations acting on it or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen. In some other embodiments, the touch sensor can also be arranged on the surface of the electronic device, in a different position from the display screen.

[0104] In some embodiments of the present application, the touch sensor can be implemented through a touch panel (TP). Taking the display screen as an LCD as an example. In some implementations, the TP can be integrated with the LCD to form a TP-LCD. In other implementations, the TP can be separated from the LCD. In this way, the user can, according to the picture displayed on the LCD (such as a game picture), achieve the input of operations by touching the corresponding position on the picture. Correspondingly, the TP can receive the operation and send operation information (such as operation type, position information, etc.) to the application for subsequent processing.

[0105] In the above example, the composition of an electronic device in the present application is provided. The embodiments of the present application also provide another composition of an electronic device. This composition can be used to describe the software composition logic of the electronic device.

[0106] Exemplarily, referring to Figure 5 , which is a schematic diagram of the composition of another electronic device provided by the embodiments of the present application.

[0107] In an example such as Figure 5 , the software system of the electronic device can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. The embodiments of the present application take the system of the layered architecture as an example to exemplarily illustrate the software structure of the electronic device.

[0108] As Figure 5 shown, the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom are the application layer, the application framework layer, Android runtime (ART) and native C / C++ libraries, the Hardware Abstract Layer (HAL), and the kernel layer.

[0109] The following will be described separately.

[0110] The application layer can include a series of application packages. The application layer can also be referred to as the application layer or the APP layer. As Figure 5 shown, the application packages can include applications such as games, calendars, maps, WLAN, music, text messages, calls, navigation, Bluetooth, and videos. For example, the game application package can correspond to the game application in the foregoing example, such as Application 1.

[0111] The application framework layer can also be referred to as the framework layer or the Framework layer. This framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0112] As Figure 5 shown, the application framework layer can include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, etc.

[0113] The window manager provides the Window Manager Service (WMS). The WMS can be used for window management, window animation management, surface management, and as a transfer station for the input system.

[0114] The content provider is used to store and obtain data, and enable these data to be accessed by applications. The data can include videos, images, audio, dialed and received calls, browsing history and bookmarks, phone books, etc.

[0115] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures. As an example, one or more Surface views can be configured in the view system. Each Surface view can be correspondingly configured with one or more textures. Each texture can be used to store part or all of the image content in a frame of an image. It should be noted that in the specific implementation process of this application, the image data in the Surface view and / or the texture can be stored in the memory of the electronic device. Correspondingly, the electronic device can manage each data, such as writing and reading, through the file identifier of each data in the memory.

[0116] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0117] The notification manager enables an application to display notification information in the status bar. It can be used to convey messages of the notification type, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that a download is complete, a message reminder, etc. The notification can also be in the form of a graph or a scrolling text bar in the system top status bar, such as the notification of a background-running application, or in the form of a dialog window on the screen. For example, it can prompt text information in the status bar, emit a prompt tone, vibrate the electronic device, blink the indicator light, etc.

[0118] The activity manager can provide the Activity Manager Service (AMS). AMS can be used for the startup, switching, scheduling of system components (such as activities, services, content providers, broadcast receivers), and the management and scheduling of application processes.

[0119] The input manager can provide the Input Manager Service (IMS). IMS can be used to manage system inputs, such as touch screen input, key input, sensor input, etc. IMS retrieves events from input device nodes and, through interaction with the WMS, distributes the events to appropriate windows. In some embodiments, IMS can include components such as IMSReader and IMSDispatcher (IMS scheduling module). As an example, after the TP of an electronic device receives a user operation, it can send the operation information to IMSReader. IMSReader can send the operation information to IMSDispatcher for centralized management and distribution. For example, IMSDispatcher can send the operation information to an application in the foreground display (such as a game application) so that the game application can respond to the user input operation. For example, the game application generates rendering instructions for the next frame of the image based on the user input operation.

[0120] The Android Runtime includes the core libraries and the Android Runtime. The Android Runtime is responsible for converting source code into machine code. The Android Runtime mainly includes the Ahead-of-Time (AOT) compilation technology and the Just-in-Time (JIT) compilation technology.

[0121] The core libraries are mainly used to provide the functions of basic Java class libraries, such as libraries for basic data structures, mathematics, IO, tools, databases, networks, etc. The core libraries provide APIs for users to develop Android applications.

[0122] The native C / C++ libraries can include multiple functional modules. For example: surface manager, Media Framework, libc, OpenGL ES, SQLite, Webkit, etc.

[0123] Among them, the surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The media framework supports the playback and recording of multiple common audio and video formats, as well as static image files, etc. The media library can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. OpenGL ES provides the drawing and operation of 2D and 3D graphics in applications. SQLite provides a lightweight relational database for the applications of electronic devices.

[0124] The hardware abstraction layer runs in the user space, encapsulates the kernel layer drivers, and provides call interfaces to the upper layer. Exemplarily, the hardware abstraction layer can include a display module, an audio module, a camera module, a Bluetooth module, etc. In some embodiments, the hardware abstraction layer can further include a touch module.

[0125] The kernel layer is the layer between the hardware and the software. The kernel layer at least includes display drivers (such as DRM Driver), camera drivers, audio drivers, and Bluetooth drivers. In some embodiments, the kernel layer can further include touch drivers.

[0126] It should be noted that, as Figure 5 shown, the software composition is only an exemplary illustration and does not constitute a limitation on the electronic devices involved in the embodiments of the present application. In some other embodiments, the electronic devices can also have other software compositions.

[0127] Exemplarily, referring to Figure 6 , it is a schematic diagram of the composition of another electronic device provided by the embodiments of the present application. In the example as Figure 6 shown, the upper-layer components composed of software modules and various possible configurations in the hardware components are shown at the same time.

[0128] As Figure 6 shown, in this example, the application layer can include multiple applications. For example, the multiple applications can include Application 1, Application 2, etc. Among them, Application 1 can be a game application. Application 2 can be a small-window application or a multi-window application. For example, during the running of Application 1, Application 2 can provide relevant controls (such as brightness adjustment, communication function configuration, etc.) during the running of the game application by displaying a window on the interface. In some embodiments, Application 2 can be a system-level application such as a game center installed in the electronic device.

[0129] The framework layer of the electronic device can be configured with a drawing module, an SF module, a view system, an IMS module, etc.

[0130] Among them, the drawing module is used to perform image rendering in response to a rendering instruction issued by an application (such as Application 1). This drawing module can correspond to the drawing module 21 in the foregoing example.

[0131] In some implementations, the drawing module can perform rendering processing according to the rendering instruction through the GPU configured in the hardware layer of the electronic device, so as to obtain the corresponding rendering result.

[0132] In the embodiments of the present application, the drawing module (or the application) can be used as a producer of a Buffer (buffer). Among them, the Buffer can be a storage space configured in the memory. Each Buffer can correspondingly store the data of a rendering result. In this way, after the drawing module completes the rendering processing of a frame image and obtains a rendering result, the rendering result can be stored in the corresponding Buffer, and the Buffer can be queued.

[0133] In some implementations, queuing the Buffer can correspond to storing the data of the rendering result (such as image information) stored in the Buffer into the corresponding buffer queue (BufferQueue).

[0134] In other implementations, queuing the Buffer can correspond to storing the identifier corresponding to the Buffer (such as Buffer ID) into the buffer queue. It can be understood that the Buffer ID can uniquely represent a Buffer. In this way, when the electronic device needs to use the Buffer, it can determine the address information of the Buffer corresponding in the memory by obtaining the Buffer ID, and read the rendering result stored in the Buffer through the address information.

[0135] In other implementations, queuing the Buffer can correspond to storing the file identifier of the rendering result stored in the Buffer into the buffer queue. Among them, the file identifier can include the address information indicating the data stored in the Buffer. In this way, when the electronic device needs to use the Buffer, it can determine the address information of the Buffer corresponding in the memory by the file identifier, and read the rendering result stored in the Buffer through the address information.

[0136] In the following example, the file identifiers of each Buffer are stored in the BufferQueue as an example.

[0137] The SF module, i.e., SurfaceFlinger, can be used to generate a Vsync signal based on information such as the currently displayed application and the refresh rate of the current display screen. In some implementations, the SF module can correspond to the composition module 23 as shown in Figure 4 below.

[0138] The SF module can be a consumer of the produced Buffers, obtaining the enqueued Buffers from the buffer queue for consumption and display. Exemplarily, the processing mechanism for consumption and display can correspond to the SF performing composition processing based on the rendering result corresponding to the Buffer obtained from the buffer queue, thereby obtaining the corresponding display image. The SF can directly or indirectly send the display image to the display screen of the electronic device for the display of the next frame of image.

[0139] In the embodiments of this application, the SF module can have the ability to monitor the buffer queue.

[0140] Exemplarily, the SF module can monitor whether there is a newly enqueued Buffer in the buffer queue. When a new Buffer is enqueued in the buffer queue, the SF can directly obtain the newly enqueued Buffer for consumption and display without waiting for the next Vsync signal to arrive.

[0141] In the framework layer, a view system can also be configured. The view system can include one or more surface views (Surface). When the drawing module performs the rendering process of a frame of image, it can be drawn based on one or more Surfaces. In this way, after the rendering operation of a frame of image is completed, the image obtained from the rendering process can be correspondingly stored in one or more Surfaces in the view system. In the embodiments of this application, one buffer can correspond to one or more Surfaces. When one Buffer corresponds to multiple Surfaces, then the multiple Surfaces can respectively store partial contents of the frame image corresponding to the Buffer. The collection of all the contents in the multiple Surfaces can constitute the content of a frame of image corresponding to the Buffer. In this way, after the drawing module completes the rendering process of a frame of image, it can achieve the enqueueing of one Buffer and simultaneously obtain one or more Surfaces storing the rendering result in the view system.

[0142] Correspondingly, when the SF module consumes a Buffer, by obtaining the file identifier of the Buffer from the BufferQueue, it can obtain the image data stored in the Surface corresponding to the Buffer from the address indicated by the file identifier for composition processing.

[0143] Such as Figure 6As shown, an IMS module can also be configured in the framework layer. Combining Figure 5 with the description in

[0144] this IMS module can include an IMSReader module and an IMS scheduling module. This IMS scheduling module is also the IMSDispatcher.

[0145] In some implementations of the present application, the IMS module can be used to implement the transfer of operation information input by the user from the lower layer to the upper layer. This process will be described in detail later.

[0146] In the following description, the example where the composition process is performed independently by the SF module is taken.

[0147] In some other embodiments of the present application, a hybrid renderer can also be configured in the abstraction layer to cooperate in implementing the composition process in different scenarios.

[0148] In the electronic device composition provided as Figure 6 described, a touch driver, a display driver, etc. can be configured in the kernel layer. Among them, the touch driver can be used to implement the control of the touch panel (TP) by the upper-layer software and the information transfer from the TP to the upper-layer software module. The display driver can be the (DRM Driver). The display driver can directly or indirectly receive the display image from the SF module and control the corresponding display screen to display according to the display image.

[0149] In the electronic device, each hardware component in the hardware layer as Figure 6 shown can be configured to support the above software modules to implement corresponding functions.

[0150] Exemplarily, a touch panel, a display screen, a GPU, a memory, etc. can be configured in the electronic device.

[0151] In some embodiments of the present application, when there are multiple application programs running in the electronic device, each application program can be correspondingly configured with a buffer queue in the memory.

[0152] For example, Application 1 and Application 2 are running simultaneously. Then a buffer queue BQ1 corresponding to Application 1 can be configured in the memory. This buffer queue BQ1 is used to store the Buffer obtained after the rendering module executes the rendering instructions issued by Application 1.

[0153] The memory may also be configured with a buffer queue BQ2 corresponding to Application 2. The buffer queue BQ2 is used to store the Buffers obtained after the rendering module executes the rendering instructions sent by Application 2.

[0154] In some implementations, these multiple buffer queues can be managed and maintained by a general buffer module (such as the BufferTX module). In different implementations, the BufferTX module can be configured in the framework layer, or the abstraction layer, or the kernel layer, or inside the memory of the electronic device. The embodiments of the present application do not limit the configuration selection of the BufferTX module.

[0155] In an example such as Figure 6 take the BufferTX module configured in the framework layer as an example.

[0156] In some embodiments, the BufferTX module can configure identification fields for each of the already configured buffer queues. The value of the identification field can be a first value, a second value, etc. When the identification field is the first value (such as the first value being 0), it means that there is no available Buffer in the corresponding buffer queue, that is, the buffer queue is empty. When the identification field is the second value (such as the second value being 1), it means that there is an available Buffer in the corresponding buffer queue. In some implementations, the second value can also be used to indicate the number of available Buffers in the current buffer queue. For example, if the second value is 1, it means that the number of available Buffers in the corresponding buffer queue is 1.

[0157] As an example, when a Buffer corresponding to Application 1 enters the buffer queue BQ1, the identification field corresponding to the buffer queue BQ1 in the BufferTX module can be configured to 1. This indicates that one Buffer has been enqueued in the buffer queue BQ1.

[0158] After the SF obtains the Buffer from the buffer queue BQ1 for consumption, the available Buffer in the buffer queue BQ1 becomes empty. The identification field corresponding to the buffer queue BQ1 in the corresponding BufferTX module is configured to 0. This indicates that the available Buffer in the buffer queue BQ1 is empty and no Buffer can be consumed.

[0159] Similarly, the BufferTX module can also implement the management and maintenance of the buffer queues corresponding to other applications (such as the buffer queue BQ2) based on a similar mechanism.

[0160] Thus, the SF module can determine whether there is a new Buffer entering the buffer queue through the respective identification fields configured in the BufferTX module.

[0161] In some other embodiments of the present application, a display queue may also be configured in the memory ( Figure 6 not shown in Figure 6 ). The display queue can be managed by a display driver. For example, when the display driver receives a display image each time, it can put the display image into the display queue. Thus, when the next Vsync signal arrives, the display driver can obtain the display image that first enters the display queue from the display queue and send it to the display screen for display.

[0162] The solutions provided in the embodiments of the present application can all be applied to the aforementioned Figure 5 or Figure 6 electronic devices provided.

[0163] In the following description, taking the application of this solution to the electronic device as shown in Figure 6 as an example, the implementation of the solution provided in the embodiments of the present application will be illustrated by way of example.

[0164] Exemplarily, referring to Figure 7 , it is an example of interaction between modules provided in the embodiments of the present application. Through the solution implemented as shown in Figure 7 , it can support the reporting of operations input by the user during the running of the application program to the application program. In the example as shown in Figure 7 , the currently running application program is taken as Application 1 for example.

[0165] As shown in Figure 7 , the solution may include:

[0166] S701. The touchpad of the electronic device receives an operation OP1 input by the user. The touchpad sends operation information 71 to the touch drive.

[0167] Exemplarily, when the display screen of the electronic device displays a certain interface during the running of Application 1, it can receive the operation OP1 input by the user. As a possible implementation, the operation OP1 may include an operation of clicking a control displayed in the current interface. Through the operation OP1, the user can instruct the game character currently controlled to perform corresponding actions, such as casting the skill A corresponding to the control, etc.

[0168] Correspondingly, the touchpad can generate operation information 71 according to the operation OP1. In some embodiments, the operation information 71 may include the operation type of the operation OP1 (such as a click operation or a swipe operation, etc.), the position information of the operation OP1 on the display screen, etc.

[0169] Thus, the touchpad can transmit the currently received operation information 71 to the upper-layer application module through the touch drive.

[0170] S702. The touch drive sends the operation information to the IMSReader module.

[0171] The IMSReader module S703 sends the operation information 71 to the IMS scheduling module.

[0172] In some embodiments, the IMSReader module can monitor and obtain the information reported by each driver. For example, the IMSReader can obtain the operation information 71 reported by the touch driver.

[0173] In this way, the IMSReader module can transmit the operation information 71 to the IMS scheduling module for centralized processing.

[0174] In some embodiments, the IMS scheduling module can determine that the operation information 71 is an operation input by the user to Application 1 based on the currently top-displayed interface being the interface corresponding to Application 1.

[0175] The IMS scheduling module S704 sends the operation information 71 to Application 1.

[0176] Thus, after the user inputs the operation OP1 during the use of Application 1, Application 1 can obtain the operation information 71 input by the user. Furthermore, Application 1 can execute subsequent operations based on the operation information 71.

[0177] Exemplarily, Application 1 can, according to the operation information 71, instruct each module in the electronic device to cooperate with each other to display an image corresponding to the operation information 71.

[0178] Combined Figure 4 Taking the example in which Application 1 receives the operation information 71 during the Vsync period between signal V31 and signal V32. Correspondingly, Application 1 can, according to the operation information 71, after signal V32 arrives, instruct the electronic device to perform processing such as rendering and sending display of the second frame of image. The second frame of image can include the screen effect corresponding to the operation OP1, such as the screen effect corresponding to releasing skill A.

[0179] As a possible implementation, referring to Figure 8 , another example of interaction between modules provided by the embodiments of the present application. Through the solution shown as Figure 8 , it can be made such that the image corresponding to the operation information 71 (such as the second frame of image) is produced and incorporated into the queue.

[0180] As Figure 8 shown, the solution can include:

[0181] Application 1 sends a rendering instruction 81 to the drawing module. In the present application, the rendering instruction 81 can also be referred to as the first rendering instruction.

[0182] Exemplarily, Application 1 can generate the rendering instruction 81 according to the operation information 71. Thus, through the rendering instruction 81, Application 1 can instruct the electronic device to perform a rendering process of a frame image (such as the second frame image) including the screen effect corresponding to the operation OP1. The second frame image can also be replaced by the Nth frame image.

[0183] S802. The drawing module calls the GPU to perform a rendering process according to the rendering instruction 81 to obtain a rendering result 82. In this application, the rendering result 82 can also be referred to as the first rendering result.

[0184] S803. The drawing module stores the rendering result 82 in the surface view S1.

[0185] Exemplarily, the drawing module can send the rendering instruction 81 to the GPU so that the GPU performs a rendering process of the second frame image according to the rendering instruction 81.

[0186] Through this rendering process, one or more texture maps corresponding to the second frame image can be obtained.

[0187] In this example, it is taken that the rendering result 82 corresponding to the rendering instruction 81 is stored in the surface view S1 as an example.

[0188] S804. The drawing module stores the file identifier 84 in the buffer queue BQ1. This buffer queue BQ1 can also be referred to as the first buffer queue.

[0189] Among them, the file identifier 84 can correspond to the rendering result 82.

[0190] In some embodiments, the file identifier 84 can indicate the storage location of the rendering result 82 in the electronic device. It can be understood that based on the foregoing description of the frame buffer production enqueue and consumption dequeue. The storage location of the rendering result 82 in the electronic device can also correspond to the position of the Buffer (such as Buffer1) configured for the current rendering instruction 81 in the memory.

[0191] In this way, in this example, by storing the file identifier 84 in the buffer queue BQ1, the production enqueue of Buffer1 corresponding to the rendering result 82 can be achieved.

[0192] Combined with the description of the BufferTX module in the foregoing example, in some embodiments of this application, an identification field 1 of the buffer queue BQ1 can be configured in the BufferTX module. After the Buffer1 corresponding to S804 is enqueued to the buffer queue BQ1, the identification field 1 of the buffer queue BQ1 in the BufferTX module can be configured to 1. Thus, it is indicated that there is a new Buffer enqueued in the current buffer queue BQ1 waiting to be consumed. This identification field 1 can also be referred to as the first identification field.

[0193] Combined with Figure 4 In the example of [[ID=]], taking the rendering instruction 81 used to indicate the rendering of the second frame image as an example.

[0194] In this way, after the signal V32 arrives, the application 1 can execute S801. Correspondingly, after the signal V33 arrives, the drawing module completes the enqueue of Buffer1 corresponding to S804.

[0195] In the embodiment of the present application, the SF module can be configured to, after a new Buffer is enqueued, without waiting for the next Vsync signal to arrive, directly obtain the Buffer from the buffer queue for consumption and display.

[0196] Exemplarily, referring to Figure 9 , which is another schematic diagram of the interaction between modules provided by the embodiment of the present application. Through the solution shown in Figure 9 , the electronic device can achieve the fast consumption and display of the newly enqueued Buffer.

[0197] As shown in Figure 9 , this solution may include:

[0198] S901. The SF module obtains the file identifier 84 from the buffer queue BQ1.

[0199] Exemplarily, after the file identifier 84 is enqueued, the SF module can sense that a new Buffer is enqueued, and then directly obtain the Buffer for consumption and display.

[0200] As a possible implementation, the SF module can determine whether there is a new Buffer enqueued currently by monitoring the identification field of the buffer queue configured for each currently running application in the BufferTX module.

[0201] For example, SF can determine that there is a new Buffer enqueued in the buffer queue BQ1 according to the identification field 1 being configured as 1.

[0202] Thus, the SF module can directly obtain the Buffer1 from the buffer queue BQ1 for subsequent processing.

[0203] In this example, taking the file identifier 84 being placed in the buffer queue as an example for the Buffer enqueue, correspondingly, the information of the newly enqueued Buffer obtained by the SF module from the buffer queue BQ1 can include the file identifier 84. This file identifier 84 can correspond to the storage address of Buffer1, or correspond to the storage address of the rendering result 82.

[0204] S902. The SF module obtains the rendering result 82 in the surface view S1 according to the file identifier 84.

[0205] Exemplarily, the SF module may obtain the rendering result 82 drawn in the view S1 according to the address indicated by the file identifier 84.

[0206] Thus, the SF module may perform a synthesis process based on the rendering result 82, and further obtain the complete content of the frame image indicated by the rendering instruction 81.

[0207] Exemplarily, the SF module may obtain the corresponding display image 91 through a synthesis process according to the rendering result 82. This display image 91 may also be referred to as the first display image.

[0208] It should be noted that in this Figures 8 to 9 description, it is taken as an example that the frame image indicated by the rendering instruction 81 is stored in the surface view S1 after rendering. In some other embodiments, the result after the drawing module renders according to the rendering instruction may be stored in multiple surface views. In this way, the SF module may obtain the rendering results in these multiple surface views according to the listed file identifiers, perform a synthesis process, and thus obtain the content of a complete frame image.

[0209] S903. The SF module transmits the display image 91 to the HWC.

[0210] S904. The HWC transmits the display image 91 to the display driver.

[0211] In this example, after the SF module completes the synthesis process, it may transmit the obtained display image 91 to the display driver through the HWC and wait for display.

[0212] S905. The display driver transmits the display image 91 to the display screen. The display screen performs display according to the display image 91.

[0213] Exemplarily, the display driver may store the display image 91 in the display queue after receiving the display image 91. So that after the next Vsync signal arrives, the earliest received first display image is obtained from the display queue for display. In this example, it is taken as an example that after the display image 91 enters the display queue, only the display image 91 is included in the display queue.

[0214] In this way, after the next Vsync signal arrives, the display driver may obtain the display image 91 from the display queue and transmit the display image 91 to the display screen for display.

[0215] Combined with Figure 4Example in. After the signal V33 arrives and before the signal V34 arrives, the rendering module can complete the rendering enqueue of the rendering result corresponding to the second frame of the image. Based on the Figure 9 scheme shown, the SF module (i.e., the synthesis module 23 shown in Figure 4 ) can, after the rendering of the second frame of the image is enqueued, directly consume and send the rendering result of the second frame of the image without waiting for the signal V34 to arrive. Thus, before the signal V34 arrives, the SF module can synthesize and obtain the corresponding display image (such as the display image 91) according to the rendering result of the second frame of the image, and transmit it to the display driver for waiting to be displayed. In this way, when the next Vsync signal (such as the signal V34) arrives, the display driver can control the display screen to display the second frame of the image according to the display image 91.

[0216] In order to more clearly illustrate the solution provided by the embodiments of the present application, the following uses the Figure 10 inter-module interaction process schematic diagram shown to illustrate the rendering and display process of the above-mentioned second frame of the image. In the Figure 10 process schematic diagram shown, the rendering result stored in the buffer queue is used for illustration. Combining the foregoing description of the Buffer production enqueue, in a specific implementation, the enqueue of the rendering result shown in Figure 10 to the buffer queue can also be implemented by the enqueue of the corresponding file identifier to the buffer queue. Correspondingly, when Buffer consumption and display are required, after obtaining the file identifier from the buffer queue, the rendering result can be obtained according to the storage address indicated by the file identifier.

[0217] As Figure 10 shown, the solution may include:

[0218] S1001. The touchpad receives the operation OP1. This operation OP1 can also be referred to as the first operation.

[0219] Exemplarily, before the touchpad receives the operation OP1, the first frame of the image may be displayed on the display screen.

[0220] In this way, the operation OP1 can be an operation input by the user on the first frame of the image. Thus, the touchpad can generate the operation information 71 according to the operation OP1 input by the user. In some embodiments, the operation information 71 may include the operation type, position information, etc. of the operation OP1. This operation information 71 can also be referred to as the first operation information.

[0221] S1002. The touchpad sends the operation information 71 to the application 1.

[0222] Exemplarily, combined with the Figure 7 description in, the touchpad can send the operation information 71 to the application 1 through the touch drive and the IMS module.

[0223] In this example, after application 1 receives the operation information 71, the next Vsync signal (such as signal V32) arrives.

[0224] S1003. Application 1 generates a rendering instruction 81.

[0225] Exemplarily, application 1 can generate a new rendering instruction according to the arrival of signal V32 and instruct other components of the electronic device to perform the rendering of new graphics.

[0226] In some embodiments, application 1 can generate a rendering instruction 81 according to the arrival of signal V32 and based on the received operation information 71.

[0227] S1004. Application 1 sends the rendering instruction 81 to the drawing module.

[0228] Thus, application 1 can, through the rendering instruction 81, instruct the drawing module to draw a new image. For example, the new image can be an image subsequent to the first frame image currently being displayed. Such as the second frame image.

[0229] S1005. The drawing module performs rendering processing to obtain a rendering result 82.

[0230] Exemplarily, the drawing module can perform rendering processing with the GPU configured in the electronic device according to the rendering instruction 81, so as to obtain the corresponding rendering result 82.

[0231] It should be noted that, combined with the example in Figure 4 In this example, the drawing module performs the rendering processing corresponding to the rendering instruction 81, which can span two Vsync cycles and complete the rendering processing before the end of the second Vsync cycle (such as the Vsync cycle between signal V33 and signal V34).

[0232] S1006. The drawing module enqueues the rendering result 82 into the buffer queue BQ1.

[0233] Among them, the buffer queue BQ1 can be a buffer queue configured corresponding to application 1. Thus, the production enqueue of the rendering result corresponding to the second frame image can be achieved.

[0234] S1007. The BufferTX module determines that there is a new Buffer enqueued.

[0235] Exemplarily, the BufferTX module can be used to monitor the enqueue situation of new Buffers in each buffer queue.

[0236] In this example, after the rendering result 82 is enqueued in the buffer queue BQ1, the BufferTX module can determine that there is a new buffer enqueued in the buffer queue BQ1 waiting to be consumed.

[0237] S1008. The BufferTX module configures the identification field 1 to 1.

[0238] Among them, the identification field 1 can be a field in the BufferTX module that identifies the available buffers in the buffer queue BQ1.

[0239] In this example, after a new buffer (such as the buffer corresponding to the rendering result 82) is enqueued in the buffer queue BQ1, the BufferTX module can configure the identification field 1 corresponding to the buffer queue BQ1 from 0 to 1. Thus, it can be indicated that the number of available buffers in the buffer queue BQ1 changes from empty to 1.

[0240] S1009. The SF module determines that the identification field 1 is configured to 1.

[0241] In this example, the SF module can monitor the changes of each identification field in the BufferTX module.

[0242] For example, based on the fact that the identification field 1 in the BufferTX module is configured to 1, the SF module can determine that there is 1 buffer to be consumed in the corresponding buffer queue BQ1.

[0243] S1010. The SF module obtains the rendering result 82 in the buffer queue BQ1 corresponding to the identification field 1.

[0244] Exemplarily, the corresponding relationship between each identification field and the corresponding buffer queue can be stored in the SF module. In this way, after determining that the identification field 1 in the BufferTX module changes from 0 to 1, the SF module can, according to this corresponding relationship, determine that there is a new buffer enqueued in the buffer queue BQ1. Thus, the SF module can directly obtain the enqueued buffer in the buffer queue BQ1 for consumption and display without waiting for the next Vsync signal to arrive.

[0245] As an implementation, the SF module can obtain the enqueued rendering result 82 from the buffer queue BQ1.

[0246] S1011. The SF module performs a composition process based on the rendering result 82 to obtain the display image 91.

[0247] Thus, the SF module can trigger the display and consumption of the second frame image without waiting for the next Vsync signal (such as signal V34) to arrive. For example, before the signal V34 arrives, the SF module can complete the synthesis process of the second frame image and obtain the corresponding display image 91.

[0248] S1012. The SF module transmits the display image 91 to the display module. In this application, the display module may include a display driver and a display screen.

[0249] S1013. The display module stores the display image 91 in the display queue.

[0250] For example, the SF module transmits the display image 91 to the display driver in the display module. In this way, the display driver can store the received display image 91 in the display queue and wait for display.

[0251] S1014. When the signal V34 arrives, the display module dequeues the display image 91 and displays the second frame image according to the display image 91.

[0252] Exemplarily, the display driver of the display module can, when the signal V34 arrives, obtain the stored display image 91 from the display queue and control the display screen of the display module to display according to the display image 91.

[0253] In this example, the display image 91 can correspond to the second frame image. In this way, the switching display from the lower first frame to the second frame image can be realized on the display screen.

[0254] It can be understood that through the solution shown as Figure 10 , even if the rendering process of the second frame image takes a long time, from the application 1 sending the rendering instruction 81 to the switching display of the corresponding second frame image on the display screen, there is only a delay of 2 Vsync cycles. Thus, in the case where the second frame image includes the response effect of the user input operation (such as operation OP1), the user can more quickly see the effect of the input operation on the display screen, thereby improving the display followability during the game operation.

[0255] The above Figures 8 to 10 has elaborated in detail on the rendering and display mechanism of one frame of image. It can be understood that for the rendering and display of other frame images, the various solutions provided in the embodiments of this application can also be referred to for implementation, thereby enabling the display screen to more quickly display the response effect corresponding to the user input operation and achieving the purpose of improving the followability.

[0256] When the solution provided in the embodiments of this application takes effect, that is, the corresponding effect as shown in Figure 4 is obtained. Compared with Figure 3The implementation effect of the existing solution shown can achieve significant benefits both in the synthesis process of the SF module and in the display process on the display screen.

[0257] Through experimental verification, Table 1 below provides the delay comparison of the synthesis process and the display process before and after the solution provided by the embodiments of the present application takes effect. Among them, taking a frame rate of 120Hz as an example.

[0258] Table 1

[0259] Synthetic delay Display delay Before the proposed solution of this application becomes effective 7 ms 8 ms After the proposed solution of this application becomes effective 0.8 ms 2 ms

[0260] In the example of Table 1, the synthesis delay can refer to the delay from the issuance of the rendering instruction for the current frame image to the start of the synthesis process of the current frame image. The display delay can refer to the delay from the issuance of the rendering instruction for the current frame image to the display of the current frame image on the display screen.

[0261] It can be seen that after the solution provided by the embodiments of the present application takes effect, it can significantly shorten the synthesis delay and the display delay, thereby achieving the effect of improving the followability.

[0262] It should be noted that in such Figure 10 example, the SF module can obtain a new Buffer enqueued according to the buffer queue corresponding to any application, that is, without waiting for the next Vsync signal to arrive, and consume the newly enqueued Buffer from this buffer queue. In some examples, this implementation can also be referred to as enabling the XSync solution, or enabling the XSync solution. That is, after the XSync solution is enabled, the SF module does not need to wait for the next Vsync signal to arrive to consume the newly enqueued Buffer.

[0263] In some other embodiments, after the SF module determines that there is a new Buffer enqueued (such as after executing S1009), it can determine whether to enable the XSync solution according to the relevant information of the buffer queue BQ1 corresponding to the identification field 1.

[0264] As a possible implementation, an application whitelist can be configured in an electronic device (such as the SF module of the electronic device). The application whitelist can include the package names of at least one application program. For example, the package names of the at least one application program can respectively correspond to the package name of a preset game application.

[0265] The SF module can determine that there is a new Buffer enqueued in the current buffer queue BQ1 according to the identification field 1 being configured as 1 after executing S1009 and before executing S1010.

[0266] The SF module can determine whether the application corresponding to the buffer queue BQ1 is included in the application whitelist according to the name of the buffer queue BQ1. For example, if the name of the buffer queue BQ1 includes the package name of Application 1 and Application 1 is included in the application whitelist, then the application corresponding to the buffer queue BQ1 is included in the application whitelist.

[0267] Thus, the SF module can determine to enable the XSync solution based on the fact that the application corresponding to the buffer queue with a new Buffer enqueued is included in the application whitelist. That is, before the signal V34 arrives, S1010 is executed to consume the newly enqueued Buffer in the buffer queue BQ1.

[0268] In some other embodiments, if the application corresponding to the buffer queue with a new Buffer enqueued is not included in the application whitelist, the SF module may not enable the XSync solution.

[0269] In this way, the SF module can wait for the next Vsync signal (such as signal V34) to arrive and then obtain the enqueued Buffer for consumption.

[0270] As an example, refer to Figure 11 , which provides another scenario schematic for the embodiments of this application.

[0271] Such as Figure 11 shown, the rendering instructions for the first frame image to the third frame image can all be issued by Application 1. Correspondingly, taking Application 1 being included in the application whitelist as an example. Combining the above description of enabling the XSync solution, for the synthesis processing of any frame image from the first frame image to the third frame image by the SF module, it can be executed without waiting for the next Vsync signal to arrive.

[0272] For the fourth frame image. Application 1 can issue a rendering instruction to render Image 4a when the signal V35 arrives. In addition, Application 2 can also issue a rendering instruction to render Image 4b when the signal V35 arrives.

[0273] Taking Application 1 being included in the application whitelist and Application 2 not being included in the application whitelist as an example.

[0274] In this way, for Image 4a, the SF module can enable the XSync solution for fast synthesis processing. For Image 4b, the SF module can not enable the XSync solution and, according to the native mechanism, wait for the next Vsync signal to arrive and then perform the corresponding synthesis processing on this Image 4b.

[0275] As an example, take the process of the SF module processing Image 4a as an example.

[0276] After the rendering of Image 4a is queued, the SF module can enable the XSync solution according to Application 1 being included in the application whitelist. Thus, the SF module can dequeue the rendering result of this Image 4a for compositing processing without waiting for signal V36 to arrive. In this way, Figure 11 In the example as shown, the compositing processing corresponding to Image 4a can be completed during the Vsync period between signal V36 and signal V37. In this way, the display screen can obtain the image to be displayed of Image 4a before signal V37 arrives, and then display Image 4a after signal V37 arrives. This Image 4a also corresponds to the 4th frame image displayed on the display screen.

[0277] Taking the process of the SF module processing Image 4b as an example.

[0278] After the rendering of Image 4b is queued, the SF module can disable the XSync solution according to Application 2 not being included in the application whitelist. For example, after the rendering of Image 4b is queued, the SF module can wait for the next Vsync signal (such as signal V36) to arrive. When signal V36 arrives, since the SF module is performing the compositing processing of Image 4a. The compositing processing for this Image 4b can continue to wait for the next Vsync signal to arrive. For example, the SF module can perform the compositing processing of the rendering result of this Image 4b after signal V37 arrives.

[0279] It should be noted that the above example is described taking the situation where neither Application 1 nor Application 2 is issuing a rendering instruction when signal V36 after signal V35 arrives. In this way, after signal V37 arrives, the rendering result of Image 4b that has not been consumed is still stored in the buffer queue BQ2 corresponding to Application 2. Therefore, after signal V37 arrives, the SF module can dequeue and consume the rendering result of this Image 4b from the buffer queue BQ2.

[0280] Generally, the application issuing the rendering instruction can be continuous.

[0281] Exemplarily, as Figure 11 shown in the scenario, when signal V35 arrives, Application 1 issues a rendering instruction for Image 4a, and Application 2 issues a rendering instruction for Image 4b. When signal V36 arrives, Application 1 can continue to issue a rendering instruction for Image 5a. Application 2 can continue to issue a rendering instruction for Image 5b.

[0282] In this way, in this example as Figure 11 shown, when signal V37 arrives, the buffer queue BQ2 may not include the rendering result of Image 4b.

[0283] As an example, Figure 11The corresponding situation of the rendering results in the buffer queues BQ1 and BQ2 at time T1 and time T2 is shown.

[0284] Among them, time T1 can be the time when image 5a is rendered and enqueued. Time T2 can be the time when image 5b is rendered and enqueued. Time T1 and time T2 can be two times within the Vsync period corresponding to signal V36 to signal V37.

[0285] As Figure 11 shown, at time T1, since image 5a has been rendered and enqueued, the rendering result of image 5a can be stored in buffer queue BQ1 waiting to be consumed. Correspondingly, since image 5b has not been completed rendering, the rendering result of image 4b can continue to be stored in buffer queue BQ2 waiting to be consumed.

[0286] And at time T2, since image 5b has been rendered and enqueued, buffer queue BQ2 can store the rendering result of image 5b waiting to be consumed.

[0287] Thus, based on the scenario of the fifth frame image as Figure 11 shown, and the implementation description of the XSync scheme in the foregoing description. In some embodiments of the present application, in this multi-application (or window) display scenario, the SF module determines whether to trigger the XSync scheme for a newly enqueued Buffer, which can be comprehensively determined based on the available Buffer situation in the buffer queues of multiple applications.

[0288] Exemplarily, taking the fourth frame image as an example.

[0289] After the rendering result of image 4a is enqueued, the rendering result of image 4b has not been completed and enqueued. In this way, the configured identification fields in the BufferTX module can include: the identification field 1 of buffer queue BQ1 is configured to 1, and the identification field 2 of buffer queue BQ2 is configured to 0. That is, the currently available Buffer is in buffer queue BQ1.

[0290] Thus, the SF module can trigger the XSync scheme for the newly enqueued Buffer (such as image 4a) in buffer queue BQ1 according to the name of the identification field 1 configured to 1, including a package name in the application whitelist.

[0291] Taking the fifth frame image as an example.

[0292] After the rendering result of Image 5a is queued, the rendering result of Image 5b has not been completed and queued. In this way, the configured identification fields in the BufferTX module may include: the identification field 1 of buffer queue BQ1 is configured to 1, and the identification field 2 of buffer queue BQ2 is configured to 1. Among them, the reason for configuring the identification field 2 of buffer queue BQ2 to 1 is that the rendering result of Image 4b has not been consumed.

[0293] Therefore, the SF module may not trigger the XSync scheme for the currently available Buffer according to the buffer queue where the currently available Buffer is located, including buffer queue BQ2 that is not in the application whitelist.

[0294] In this way, the SF module may wait for the next Vsync signal (such as signal V37) to arrive, obtain all currently available Buffers for unified composition processing, and obtain the display image of the corresponding fifth frame of image.

[0295] For example, the SF module may obtain the rendering result of Image 5a in buffer queue BQ1 and the rendering result of Image 5b in buffer queue BQ2 that are currently available at signal V37. According to the rendering results of Image 5a and Image 5b, perform composition processing to obtain the display image of the fifth frame and send it to the display driver.

[0296] The display driver may perform display according to the display image of the fifth frame when signal V38 arrives.

[0297] As an example, refer to Figure 12 . The rendering result of Image 4a is as shown in 1201. Correspondingly, when the electronic device displays the fourth frame of image on the display screen, it may perform display according to the display image after the composition processing of the rendering result of Image 4a.

[0298] In this example, Application 2 may realize the display of the small window 1202 by instructing the electronic device to perform the rendering and display of Image 4b and Image 5b.

[0299] In this way, based on the processing mechanism of the fourth frame and the fifth frame of images as in Figure 11 , the effect of the game screen corresponding to Application 1 included in the fourth frame of image as shown in Figure 12 can be obtained. And the display of the fifth frame may include the game screen instructed by Application 1 and the small window screen instructed by Application 2 at the same time.

[0300] To more clearly illustrate the processing mechanism in the multi-application (window) scenario corresponding to the fourth frame and the fifth frame of images, refer to Figure 13 and Figure 14, which is another schematic diagram of the interaction process between modules provided by the embodiments of the present application. Through this solution, the electronic device can accurately process the 4th frame image and the 5th frame image, and obtain display effects such as Figure 11 and Figure 12 .

[0301] Among them, Figure 13 provides the processing logic during the Vsync period from signal V35 to signal V36.

[0302] As Figure 13 shown, this solution may include:

[0303] S1301. The touchpad receives operation OP2.

[0304] Exemplarily, the operation OP2 may be an operation input by the user for Application 2.

[0305] For example, taking Application 2 as a small window application. The operation OP2 may correspond to operations such as dragging, sliding, and clicking on the control corresponding to Application 2 on the current display interface.

[0306] Through this operation OP2, the user can instruct the electronic device to display the small window of Application 2.

[0307] It can be understood that during the operation of Application 1, when the electronic device receives this operation OP2, the electronic device can, in subsequent image displays, simultaneously display the game screen of Application 1 and the small window interface corresponding to the operation OP2.

[0308] S1302. The touchpad sends operation information 72 to Application 2.

[0309] Exemplarily, after receiving the operation OP2, the touchpad can generate corresponding operation information 72. Similar to the process of sending operation information 71 to Application 1 after the input of operation OP1, in this example, the touchpad can send this operation information 72 to Application 2 through the touch drive and the IMS module.

[0310] Thus, after the next Vsync signal arrives, Application 1 can normally issue a rendering instruction to instruct the electronic device to perform the rendering of the game screen. Application 2 can also issue a rendering instruction to instruct the electronic device to perform the rendering and subsequent display of the small window.

[0311] Exemplarily, the electronic device can implement the rendering, synthesis, etc. of the current game screen (such as Image 4a) through the following processing mechanisms of S1303a - S1313a. The electronic device can also implement the corresponding processing of the small window (such as Image 4b) through the following processing mechanisms of S1303b - S1309b.

[0312] The following will be described separately.

[0313] S1303a. Application 1 generates rendering instruction 131. This rendering instruction 131 can also be referred to as the third rendering instruction.

[0314] Exemplarily, after signal V35 arrives, Application 1 can generate a rendering instruction 131 for Image 4a.

[0315] S1304a. Application 1 sends rendering instruction 131 to the drawing module.

[0316] S1305a. The drawing module performs rendering processing according to rendering instruction 131 and obtains rendering result 133. The rendering result 133 can also be referred to as the third rendering result.

[0317] S1306a. The drawing module sends rendering result 133 to buffer queue BQ1.

[0318] S1307a. The BufferTX module updates identification field 1.

[0319] S1308a. The SF module confirms the update of identification field 1 in the BufferTX module.

[0320] Exemplarily, the processing mechanism from S1304a to S1308a can refer to S1004 - S1009 in Figure 10 . The specific implementations can refer to each other and will not be elaborated here.

[0321] S1309a. The SF module determines that Application 1 corresponding to the available Buffer is in the application whitelist.

[0322] In this example, the SF module can determine the situation of all currently available Buffers according to each identification field in the BufferTX module.

[0323] For example, if the identification fields configured as 1 in the BufferTX module only include identification field 1, the SF module determines that the currently available Buffer only includes the buffer in buffer queue BQ1 corresponding to identification field 1.

[0324] In this way, the SF module can determine that the application corresponding to the currently available Buffer is in the application whitelist based on the name of buffer queue BQ1 including Application 1 and the package name of Application 1 being included in the application whitelist.

[0325] Thus, the SF module can enable the XSync scheme for the currently available Buffer. That is, the SF module can consume the existing Buffers in buffer queue BQ1 without waiting for the arrival of the next Vsync signal.

[0326] S1310a. The SF module obtains the rendering result 133 from the buffer queue BQ1.

[0327] S1311a. The SF module performs a synthesis process based on the rendering result 133.

[0328] S1312a. The SF module completes the synthesis process of the rendering result 133 and obtains the display image 92. In this example, combined with Figure 11 the description, in this example, the SF module can complete the synthesis process of the rendering result 133 after the signal V36 arrives.

[0329] S1313a. The SF module transmits the display image 92 to the display module.

[0330] In this way, after the next Vsync signal arrives, the display screen can display the display image 92.

[0331] Thus, the rapid display of the game screen corresponding to the image 4a can be achieved.

[0332] As Figure 13 shown, the electronic device can also implement the processing of the relevant instructions of the application 2 through S1303b - S1309b.

[0333] S1303b. The application 2 generates a rendering instruction 132. This rendering instruction 132 can also be referred to as the second rendering instruction.

[0334] Exemplarily, the application 2 can generate a rendering instruction 132 according to the operation information 72. This rendering instruction 132 can be used to instruct the electronic device to perform the rendering process of the small window interface. This small window interface can correspond to the image 4b in the foregoing example.

[0335] S1304b. The application 2 sends the rendering instruction 132 to the drawing module.

[0336] S1305b. The drawing module performs a rendering process according to the rendering instruction 132 and obtains the rendering result 134.

[0337] As Figure 13 shown, in this example, the application 1 and the application 2 can respectively generate corresponding rendering instructions for distribution according to the arrival of the signal V35. In this way, the drawing module can respectively perform the rendering processes corresponding to the application 1 and the application 2 through different rendering threads.

[0338] It can be understood that during different rendering processes, due to the different rendering contents indicated by the rendering instructions, the corresponding time consumptions are also different. For example, the timing for the drawing module to obtain the rendering result 133 can be earlier than the timing for obtaining the rendering result 134.

[0339] In this way, the rendering result 133 can enter the buffer queue earlier than the rendering result 134. This can trigger the above S1306a and subsequent processing procedures.

[0340] In contrast, after obtaining the rendering result 134 through the steps shown in S1305b, the drawing module can continue the processing according to the following steps.

[0341] S1306b: The drawing module sends the rendering result 134 to the buffer queue BQ2.

[0342] Among them, the buffer queue BQ2 can be the buffer queue configured for Application 2 in the memory of the electronic device. This can complete the production and enqueue of the new Buffer of the rendering result 134.

[0343] S1307b: The BufferTX module updates the identification field 2.

[0344] S1308b: The SF module confirms the update of the identification field 2 in the BufferTX module.

[0345] S1309b: The SF module determines that the application 2 corresponding to the available Buffer is not in the application whitelist.

[0346] It can be understood that in combination with the processing in S1311a. After the rendering result 133 corresponding to S1311a dequeues for composition processing, the available Buffer in the buffer queue BQ1 can be empty.

[0347] Correspondingly, after S1311a, the BufferTX module can adjust the identification field 1 corresponding to BQ1 to 0.

[0348] Thus, after the processing of S1307b, after the BufferTX module updates the identification field 2 (such as adjusting the identification field 2 to 1), the identification field 1 is 0 and the identification field 2 is 1. That is, the available Buffer indicated by the current BufferTX module is the Buffer corresponding to BQ2 corresponding to the identification field 2.

[0349] In this step S1309b, the SF module can determine that the application 2 included in the name of BQ2 corresponding to the available Buffer indicated by the identification field in the BufferTX module is not in the application whitelist according to the available Buffer corresponding to the identification field being the Buffer corresponding to BQ2. Thus, the SF module can disable the Xsync scheme for the current available Buffer. That is, the SF module can wait for the next Vsync signal to arrive and attempt to perform the composition processing of the current available Buffer.

[0350] In this way, during the Vsync period from signal V35 to signal V36, the processing of the fourth frame image can be completed.

[0351] Next, the electronic device can continue to execute the processing of the fourth frame image and the fifth frame image during the subsequent Vsync period as Figure 14 described.

[0352] As Figure 14 shown, this solution may include:

[0353] S1401a. Application 1 generates rendering instruction 141.

[0354] Combined with the example in Figure 11 , after the arrival of signal V36, Application 1 can also generate rendering instruction 141, thereby instructing the electronic device to perform processing such as rendering of Image 5a.

[0355] S1402a. Application 1 sends rendering instruction 141 to the drawing module.

[0356] S1403a. The drawing module performs rendering processing according to rendering instruction 141 and obtains rendering result 143.

[0357] S1404a. The drawing module sends rendering result 143 to buffer queue BQ1.

[0358] S1405a. The BufferTX module updates flag field 1.

[0359] S1406a. The SF module monitors the update of flag field 1.

[0360] S1407a. The SF module determines that the applications corresponding to the available Buffers include Application 2 which is not in the application whitelist.

[0361] It can be understood that, combined with the description in Figure 13 , before the rendering result 143 corresponding to S1404a is queued, in the BufferTX module, since the rendering result 133 has been dequeued, flag field 1 is 0. And the rendering result 134 is waiting for the arrival of the next Vsync signal and has not been dequeued yet, so flag field 2 is 1.

[0362] In this way, after the rendering result 143 corresponding to S1404a is queued, the BufferTX module may include: flag field 1 is 1 and flag field 2 is 1.

[0363] The SF module can determine that there is a new Buffer queued according to S1406a, and then determine whether to trigger the Xsync solution according to the steps corresponding to S1407a.

[0364] Exemplarily, the SF module can determine that the currently available Buffer can include 1 Buffer in buffer queue BQ1 and 1 Buffer in buffer queue BQ2 according to the identification field in the BufferTX module.

[0365] The SF module can determine that the Buffer corresponding to buffer queue BQ1 is the Buffer produced by the application in the application whitelist according to application 1 corresponding to the name of buffer queue BQ1.

[0366] The SF module can determine that the Buffer corresponding to buffer queue BQ2 is not the Buffer produced by the application in the application whitelist according to application 2 corresponding to the name of buffer queue BQ2.

[0367] In this way, since there is a Buffer in the currently available Buffer that is produced by an application not in the application whitelist, the SF module can not trigger the XSync scheme. Correspondingly, the SF module can wait for the next Vsync signal to arrive and then consume the two produced Buffers.

[0368] In addition, as Figure 13 shown, within the Vsync period from signal V35 to signal V36, the electronic device can also perform the following processing:

[0369] S1401b. Application 2 generates rendering instruction 142. The rendering instruction 142 can be used to indicate the rendering of image 5b.

[0370] S1402b. Application 2 sends the rendering instruction 142 to the drawing module.

[0371] S1403b. The drawing module performs rendering processing according to the rendering instruction 142 and obtains the rendering result 144.

[0372] S1404b. The drawing module sends the rendering result 144 to buffer queue BQ2.

[0373] S1405b. The BufferTX module updates identification field 2.

[0374] S1406b. The SF module monitors the update of identification field 2.

[0375] S1407b. The SF module determines that the applications corresponding to the available Buffer include application 2 not in the application whitelist.

[0376] Thus, through the implementation of the solution from S1401b to S1407b, the electronic device can, within this Vsync period, respond to the rendering instruction 142 of application 2 and perform corresponding processing. For example, the drawing module can complete the enqueue of the corresponding rendering result 144 into buffer queue BQ2.

[0377] In some embodiments of the present application, take an example that there is one valid Buffer configured in the buffer queue BQ2. In this way, at most one available Buffer can exist in the buffer queue BQ2 at the same time. Since there is an unconsumed rendering result 134 in the buffer queue BQ2. Therefore, after the rendering result 144 is enqueued, the rendering result 144 can overwrite the rendering result 134.

[0378] Correspondingly, after S1404b, the buffer queue BQ2 can include the rendering result 144.

[0379] And since there is a produced Buffer in the buffer queue BQ2, the identification field 2 in the BufferTX module can be 1.

[0380] In the present application, the SF module can execute a judgment such as S1407a or S1407b each time it detects a new update of the identification field to determine whether to trigger the XSync scheme.

[0381] In this S1407b, the SF module can determine that the currently available Buffer includes the Buffer produced by Application 2, and Application 2 is not included in the application whitelist. Therefore, the SF module does not trigger the XSync scheme.

[0382] Then, after the next Vsync signal (such as signal V37) arrives, the e-book can execute S1408 - S1412.

[0383] S1408, the display module displays according to the image 92 to be sent for display.

[0384] Thus, the fourth frame image corresponding to the image 92 to be sent for display can be displayed on the display screen of the electronic device. The fourth frame image can correspond to the rendering result of the image 4a as shown in Figure 11 the game screen, that is, the rendering result of the image 4a.

[0385] Combined with the description in S1407b, since all the currently produced Buffers need to wait for the next Vsync signal to arrive before they can be dequeued for consumption. Therefore, after the signal V37 arrives, this scheme can further include:

[0386] S1409, the SF module obtains the rendering result 143 from the buffer queue BQ1.

[0387] S1410, the SF module obtains the rendering result 144 from the buffer queue BQ2.

[0388] S1411, the SF module performs a synthesis process according to the rendering result 143 and the rendering result 144 to obtain the image 93 to be sent for display.

[0389] S1412. The SF module transmits the display image 93 to the display module.

[0390] In this way, the display image 93 can include the screen contents rendered by Application 1 and Application 2 after the signal V36 arrives. For example, the display image 93 can include Image 5a (such as a game screen) and Image 5b (such as a small window screen).

[0391] Thus, as shown in S1413, after the next Vsync signal (such as signal V38) arrives, the display module can perform display according to the display image 93. In this way, the complete fifth frame image can be displayed on the display screen.

[0392] It should be noted that in some other embodiments of the present application, a switch button enabling the Xsync scheme may also be configured in the electronic device. Through this switch button of the Xsync scheme, the effectiveness of the Xsync scheme can be overall controlled.

[0393] Exemplarily, when the switch button of the Xsync scheme is in the open state, the electronic device (such as the SF module in the electronic device) can, according to, Figure 10 or Figure 13 or Figure 14 the provided scheme, determine whether the currently available Buffer uses the Xsync scheme for fast synthesis processing. Correspondingly, when the switch button of the Xsync scheme is in the closed state, the electronic device (such as the SF module in the electronic device) can, according to the native logic, trigger the dequeue synthesis of the existing Buffer based on the arrival of the next Vsync signal.

[0394] In some embodiments, the switch button of the Xsync scheme can be manually opened or closed by the user. In some other embodiments, the switch button of the Xsync scheme can be opened or closed by the electronic device itself.

[0395] It can be understood that in order to implement the above functions, the electronic device provided in the embodiments of the present application includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving the hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0396] The embodiments of the present application can divide the above-mentioned electronic device into functional modules according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there may be other division methods in actual implementation.

[0397] The above mainly introduces the solutions provided by the embodiments of the present application from the perspective of each functional module. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described function for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0398] The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there may be other division methods in actual implementation.

[0399] Exemplarily, Figure 15 shows a schematic diagram of the composition of an electronic device 1500. As Figure 15 shown, the electronic device 1500 may include: a processor 1501 and a memory 1502. In this example, the electronic device 1500 may further include a display screen 1503. The memory 1502 is used to store computer execution instructions. Exemplarily, in some embodiments, when the processor 1501 executes the instructions stored in the memory 1502, the electronic device 1500 may be enabled to execute the method shown in any one of the above embodiments, so that relevant images can be displayed faster on the display screen 1503.

[0400] It should be noted that all relevant contents of each step involved in the above method embodiments can be cited in the function description of the corresponding functional module, and will not be repeated here.

[0401] Figure 16The schematic diagram of the composition of a chip system 1600 is shown. The chip system 1600 may include: a processor 1601 and a communication interface 1602, which are used to support related devices to implement the functions involved in the above embodiments. In a possible design, the chip system further includes a memory for storing necessary program instructions and data of the electronic device. The chip system may be composed of chips or may include chips and other discrete devices. It should be noted that, in some implementation manners of the present application, the communication interface 1602 may also be referred to as an interface circuit.

[0402] It should be noted that all relevant contents of each step involved in the above method embodiment can be cited in the function description of the corresponding functional module, and will not be elaborated here.

[0403] In the above embodiments, the functions, actions, operations, steps, etc. can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more integrated media. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0404] Although the present application has been described in connection with specific features and their embodiments, it will be apparent that various modifications and combinations can be made without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely exemplary illustrations of the present application as defined by the appended claims and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the present application. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these changes and modifications.

Claims

1. An image processing method, characterized in that, The method is applied to an electronic device, which is configured with a display screen, and a first application is also installed in the electronic device; The method includes: After generating a first synchronous Vsync signal, receiving a first rendering instruction sent by the first application, where the first rendering instruction is used to instruct the electronic device to perform rendering processing on the Nth frame of image; Obtaining a first rendering result according to the first rendering instruction; Performing a composition process according to the first rendering result to obtain a first image for display; the first image for display corresponds to the Nth frame of image; After generating a second Vsync signal, controlling the display screen to display the Nth frame of image according to the first image for display; Wherein, the second Vsync signal is generated after the first Vsync signal; and there is no generated Vsync signal before performing the composition process according to the first rendering result after obtaining the first rendering result.

2. The method according to claim 1, wherein Before generating the first Vsync signal, the method further includes: Receiving a first operation, where the first operation is an operation on the first interface, and the first interface is the display interface of the first application.

3. The method according to claim 2, wherein The method further includes: Generating first operation information according to the first operation; the first operation information indicates the operation type of the first operation and the position information of the first operation; Sending the first operation information to the first application; The first operation information corresponds to the first rendering instruction.

4. The method according to any one of claims 1-3, characterized in that A first buffer queue is configured in the electronic device, and the first buffer queue corresponds to the first application; After obtaining the first rendering result according to the first rendering instruction, the method further includes: Storing first information corresponding to the first rendering result in the first buffer queue; The first information includes any one of the following: Image information of the first rendering result; an identifier of a buffer (Buffer) storing the first rendering result; a first file identifier, where the first file identifier indicates the storage location of the first rendering result in the memory of the electronic device.

5. The method according to claim 4, wherein After sending the first information corresponding to the first rendering result to the first buffer queue, the method further includes: Configuring a first identification field corresponding to the first buffer queue to a second value; the first identification field being the second value indicates that there is new information queued in the first buffer queue.

6. The method according to claim 5, wherein Before sending the first information corresponding to the first rendering result to the first buffer queue, the first identification field is configured to a first value; the first identification field being the first value indicates that the available information in the first buffer queue is empty.

7. The method according to claim 5 or 6, characterized in that The performing a composition process according to the first rendering result to obtain a first image for display includes: Obtaining the first information from the first buffer queue, Performing a composition process on the first rendering result indicated by the first information to obtain the first image for display.

8. The method according to claim 7, wherein An application whitelist is configured in the electronic device, and the application whitelist includes at least one piece of application information, and the application information of different applications is different; Before obtaining the first information from the first buffer queue, the method further includes: Determining that the application information corresponding to all buffer queues with identification fields being the second value is included in the application whitelist.

9. The method according to claim 8, wherein The determining that the application information corresponding to all buffer queues with identification fields being the second value is included in the application whitelist includes: Determining that the application information corresponding to the first buffer queue is included in the application whitelist.

10. The method according to claim 9, wherein: The application information includes the package name of the application; The application information corresponding to the first buffer queue includes: the package name of the first application corresponding to the first buffer queue; The determining that the information of the first buffer queue is included in the application whitelist includes: Determining that the package name of the first application corresponding to the first buffer queue is included in the application whitelist.

11. The method according to claim 10, characterized in that, The method further includes: Determining the package name of the first application corresponding to the first buffer queue according to the name of the first buffer queue.

12. The method according to any one of claims 8-11, characterized in that, A second application is also installed in the electronic device; The method further includes: After generating a third Vsync signal, receiving a second rendering instruction sent by the second application, where the second rendering instruction is used to instruct the electronic device to perform at least partial rendering processing on the Mth frame of image; Obtaining a second rendering result according to the second rendering instruction; After generating a fourth Vsync signal, performing a synthesis process according to the second rendering result to obtain a second display image; the second display image corresponds to the Mth frame of image; After generating a fifth Vsync signal, controlling the display screen to display the Mth frame of image according to the second display image; Wherein, the fourth Vsync signal is generated after obtaining the second rendering result.

13. The method according to claim 12, wherein After the obtaining the second rendering result according to the second rendering instruction, the method further includes: Determining that the application information of the second application is not included in the application whitelist.

14. The method according to claim 12 or 13, characterized in that, After the obtaining the second rendering result, the method further includes: Storing the second information of the second rendering result into a second buffer queue, where the second buffer queue is the buffer queue corresponding to the second application.

15. The method according to claim 14, wherein The method further includes: After generating the third Vsync signal, receiving a third rendering instruction sent by the first application, where the second rendering instruction is used to instruct the electronic device to perform at least partial rendering processing on the Mth frame of image; Obtaining a third rendering result according to the third rendering instruction; Storing the third information of the third rendering result into the first buffer queue; Wherein, the third rendering result is obtained before the generation of the fourth Vsync signal.

16. The method according to claim 15, wherein The performing a synthesis process according to the second rendering result after generating the fourth Vsync signal to obtain a second display image includes: After generating the fourth Vsync signal, obtaining the third rendering result according to the third information of the first buffer queue; obtaining the second rendering result according to the second information of the second buffer queue; Perform a synthesis process on the second rendering result and the third rendering result to obtain the second image to be sent and displayed.

17. The method according to claim 15 or 16, characterized in that, After storing the third information of the third rendering result in the first buffer queue, the method further includes: Determine that there is at least one piece of application information corresponding to a buffer queue with an identification field value of the second value and not included in the application whitelist.

18. The method according to claim 17, wherein The determination that there is at least one piece of application information corresponding to a buffer queue with an identification field value of the second value and not included in the application whitelist includes: Determine that the application information corresponding to the second buffer queue is not included in the application whitelist.

19. An electronic device, characterized in that, The electronic device includes: a memory and one or more processors; the memory is coupled to the processor; Wherein, the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method described in any one of claims 1-18.

20. A chip system, characterized in that, The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected by lines; the interface circuits are used to receive signals from the memory of the electronic device and send the signals to the processor, and the signals include the computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the method described in any one of claims 1-18.

Images