Image processing method and electronic equipment

Abstract

The embodiment of the invention provides an image processing method and electronic equipment, and relates to the technical field of image processing. According to the method, the display delay of the corresponding response content on the display screen after the user inputs the operation can be effectively reduced, and the chirality is improved. The method comprises the steps that after a first synchronous Vsync signal is generated, a first rendering instruction issued by a first application is received; and obtaining a first rendering result according to the first rendering instruction. And generating first binding information. And performing synthesis processing according to the first rendering result to obtain a first sending display image. And after the second Vsync signal is generated, controlling the display screen to display the Nth frame of image according to the first sending display image. The second Vsync signal is generated after the first Vsync signal. After the first rendering result is obtained, the generated Vsync signal does not exist before synthesis processing is carried out according to the first rendering result. The duration between the first moment and the second moment is greater than or equal to the preset duration.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of image processing technologies, and in particular, to an image processing method and an electronic device. Background Art

[0002] Currently, before an electronic device displays an image, it can perform processing such as rendering and compositing of the image.

[0003] Exemplarily, after an application issues a rendering instruction, the electronic device can perform rendering processing according to the rendering instruction to obtain a corresponding rendering result. The electronic device can also perform compositing processing on the rendering result when the next synchronization signal (such as a Vsync signal) arrives, so as to obtain the display image of this frame of image. The display screen of the electronic device can display according to the display image when the next Vsync signal arrives. Thus, the display of this frame of image on the display screen can be realized.

[0004] In this way, from the time when the application issues a rendering instruction to the time when the corresponding frame of image is displayed on the display screen, there are delays of multiple Vsync signal cycles. Some rendering instructions issued by applications can include responses to user input operations. This causes the user to wait for a long time to see the response to the input operation on the display screen after the input operation. This will lead to a problem of poor touch responsiveness.

[0005] In some technical solutions, the electronic device can perform compositing processing in advance without being limited to the arrival of the Vsync signal. This can effectively advance the display of the image on the display screen. However, due to the advance of the compositing processing, it may cause the display images to be unable to be displayed in time and accumulate in the display queue.

[0006] Generally, the number of available buffers (Buffers) during the operation of an application is limited. Each display image also occupies one buffer. In this way, the accumulation of display images leads to a reduction in the number of available Buffers. This further causes the application to be unable to obtain an available buffer in time when performing subsequent image processing. This also causes the problem that the user needs to wait for a long time to see the corresponding response on the display screen after the input operation. Summary of the Invention

[0007] The present application provides an image processing method and an electronic device, which can effectively reduce the delay of the corresponding response content displayed on the display screen after the user's input operation, improve touch responsiveness; and at the same time avoid the accumulation of Buffers in the display queue.

[0008] To achieve the above technical objectives, the present application adopts the following technical solutions:

[0009] In a first aspect, a method for image processing is provided. The method is applied to an electronic device configured with a display screen and having a first application installed therein. The method includes: after generating a first Vsync signal, receiving a first rendering instruction issued by the first application, where the first rendering instruction is used to instruct the electronic device to perform rendering processing on the Nth frame of image. According to the first rendering instruction, obtaining a first rendering result. Generating first binding information. Performing a synthesis process based on the first rendering result to obtain a first image to be displayed. The first image to be displayed corresponds to the Nth frame of image. After generating a second Vsync signal, controlling the display screen to display the Nth frame of image according to the first image to be displayed. Wherein, the second Vsync signal is generated after the first Vsync signal. After obtaining the first rendering result and before performing the synthesis process based on the first rendering result, no generated Vsync signal exists. The duration between a first moment and a second moment is greater than or equal to a preset duration. The first moment is the moment when the first binding information is generated, the second moment is the moment when second binding information is generated, the second binding information is generated after obtaining a second rendering result, and the second rendering result is the rendering result of the (N - 1)th frame of image.

[0010] Thus, in the implementation of this solution, the electronic device can start performing the synthesis process of the existing rendering result without waiting for the arrival of the next Vsync signal (such as the second Vsync signal). Thereby shortening the duration from receiving a user operation to displaying the response content on the display screen.

[0011] In this example, when the electronic device performs the synthesis process, it can also be based on the generated first Binder information. By restricting the time difference between the generation time of the first binding information and the generation time of the binding information of the previous frame of image to be at least the preset duration, the control of the synthesis process frequency is achieved. In some embodiments, the preset duration can correspond to the frame interval corresponding to the frame rate of the first application. For example, if the first application is a game application and the frame rate of the first application is 120 Hz. The preset duration can be 8.3 ms corresponding to the frame interval of 120 Hz.

[0012] In this way, by controlling the frequency of the synthesis process, the acquisition frequency of the image to be displayed is controlled. Further avoiding the accumulation of the images to be displayed in the display queue.

[0013] Optionally, before generating the first binding information, the method further includes: determining a first duration between a third moment and the second moment. The third moment is the moment when the first rendering result is obtained. In the case where the first duration is less than the preset duration, the first moment is later than the third moment. The generating of the first binding information includes: generating the first binding information at the first moment.

[0014] Optionally, when the first duration is greater than or equal to the preset duration, the first moment is the same as the third moment. Generating the first binding information includes: generating the first binding information at the third moment.

[0015] Thus, in this application, the generation timing of the first binding information for the currently produced rendering result can be confirmed based on the time when the rendering result of the previous frame image is generated (or queued).

[0016] Optionally, the method further includes: determining the first duration.

[0017] Optionally, a first timer is configured in the electronic device, and the first timer is used to time a second duration. Before determining the first duration, it includes: determining the first duration when the timing of the first timer ends.

[0018] In this solution example, a loop judgment mechanism is provided. For example, the second duration is configured to be 30s. Thus, the electronic device can judge every 30s whether to postpone the generation of the binding information corresponding to the current frame image. Thus, it is not necessary to judge for each frame image, saving the corresponding overhead.

[0019] Optionally, a first buffer queue is configured in the electronic device, and the first buffer queue corresponds to the first application. After obtaining the first rendering result according to the first rendering instruction, the method further includes: sending the first information corresponding to the first rendering result to the first buffer queue. The first information includes any one of the following: the image information of the first rendering result. The identifier of the buffer Buffer storing the first rendering result. The first file identifier, which indicates the storage location of the first rendering result in the storage of the electronic device.

[0020] Thus, based on the production and consumption mechanism of the Buffer, the control of the generation of the rendering result and subsequent synthesis processing is realized.

[0021] Optionally, after sending the first information corresponding to the first rendering result to the first buffer queue, the method further includes: configuring the first identification field corresponding to the first buffer queue to a second value. The first identification field being the second value indicates that there is new information queued in the first buffer queue.

[0022] Optionally, before sending the first information corresponding to the first rendering result to the first buffer queue, the first identification field is configured to a first value. The first identification field being the first value indicates that the available information in the first buffer queue is empty.

[0023] Thus, through the configuration of the identification field, before the electronic device performs the synthesis process, it can determine whether there is a new Buffer queued currently or determine information such as all available Buffers currently based on the identification field. Furthermore, it realizes the control of existing Buffers by the electronic device through the identification field.

[0024] Optionally, performing a synthesis process based on the first rendering result to obtain a first image for display includes: obtaining the first information from the first buffer queue, and performing a synthesis process on the first rendering result indicated by the first information to obtain the first image for display.

[0025] Optionally, an application whitelist is configured in the electronic device, and the application whitelist includes at least one application information, and the application information of different applications is different. Before obtaining the first information from the first buffer queue, the method further includes: determining that the information of the first buffer queue is included in the application whitelist.

[0026] Optionally, the application information includes the package name of the application. The information of the first buffer queue includes: the package name of the first application corresponding to the first buffer queue. The determining that the information of the first buffer queue is included in the application whitelist includes: determining that the package name of the first application corresponding to the first buffer queue is included in the application whitelist.

[0027] Optionally, the method further includes: determining the package name of the first application corresponding to the first buffer queue according to the name of the first buffer queue.

[0028] Thus, through the setting of the application whitelist, the electronic device can selectively perform synthesis processing in advance for some pre-configured applications with strong requirements for followability. Furthermore, it obtains significantly improved effects for specific scenarios.

[0029] Optionally, a second application is also installed in the electronic device, and the second application corresponds to a second buffer queue. The second buffer queue includes second information, and the second information is information corresponding to a third rendering result, and the third rendering result is obtained by the electronic device through rendering under the instruction of the second application. Before obtaining the first information from the first buffer queue, the method further includes: determining that the information of the second buffer queue is included in the application whitelist.

[0030] Optionally, before performing a synthesis process based on the first rendering result to obtain a first image for display, the method further includes: obtaining the second information from the second buffer queue. Obtaining the third rendering result according to the second information. The performing a synthesis process based on the first rendering result to obtain a first image for display includes: performing a synthesis process based on the first rendering result and the third rendering result to obtain the first image for display.

[0031] Thus, when there are multiple available Buffers, if the applications corresponding to all available Buffers are in the application whitelist, the electronic device does not need to wait for the next Vsync signal to arrive, and can obtain all available Buffers for early synthesis.

[0032] Optionally, when the information of the second buffer queue is not included in the application whitelist, the method further includes: after generating the second Vsync signal, obtaining the first rendering result according to the first information, and obtaining the third rendering result according to the second information. Performing synthesis processing according to the first rendering result and the third rendering result to obtain a first display image. After generating the second Vsync signal, controlling the display screen to display the Nth frame image according to the first display image, including: after generating the third Vsync signal, controlling the display screen to display the Nth frame image according to the first display image, where the third Vsync signal is the Vsync signal after the second Vsync signal.

[0033] Thus, when there are multiple available Buffers, if there is at least one application corresponding to an available Buffer that is not in the application whitelist, then the XSync scheme is not triggered. For example, the electronic device can obtain all Buffers for synthesis after the next Vsync signal arrives. The corresponding display image can be displayed after the next Vsync signal arrives.

[0034] In a second aspect, the present application further provides an electronic device, which includes: a memory and one or more processors. The memory and the processor are coupled. In some implementations, the electronic device may also be configured with a display screen. Among them, the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the technical solutions provided in the first aspect and any possible implementation thereof. In this way, the display screen of the electronic device can display images faster.

[0035] In a third aspect, the present application further provides a chip system, which is applied to an electronic device; the chip system may include one or more interface circuits and one or more processors. The interface circuits and the processors are interconnected by lines. The interface circuits are used to receive signals from the memory of the electronic device and send the signals to the processors, and the signals include the computer instructions stored in the memory. When the processors execute the above computer instructions, the electronic device executes the technical solutions provided in the first aspect and any possible implementation thereof.

[0036] Fourthly, the present application also provides a computer-readable storage medium, including computer instructions, which, when running on an electronic device, enable the electronic device to execute the technical solutions provided in the first aspect and any possible implementation thereof as described above.

[0037] Fifthly, the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the technical solutions provided in the first aspect and any possible implementation thereof as described above.

[0038] It can be understood that the solutions provided in the second to fifth aspects of the present application can respectively correspond to the first aspect and any possible design thereof, and thus the beneficial effects that can be achieved are similar, which will not be elaborated here. Description of the Drawings

[0039] Figure 1 It is a logical schematic diagram of an interface interaction and display;

[0040] Figure 2 It is a logical schematic diagram of the interaction between internal modules of an electronic device;

[0041] Figure 3 It is a logical schematic diagram of a multi-frame image processing process;

[0042] Figure 4 It is a logical schematic diagram of a multi-frame image processing process after the solution provided in the embodiment of the present application comes into effect;

[0043] Figure 5 It is a logical schematic diagram of Buffer1 being occupied after the solution provided in the embodiment of the present application comes into effect;

[0044] Figure 6 It is a logical schematic diagram of a multi-frame image processing process after the solution provided in the embodiment of the present application comes into effect;

[0045] Figure 7 It is a schematic diagram of the composition of an electronic device provided in the embodiment of the present application;

[0046] Figure 8 It is a schematic diagram of the composition of an electronic device provided in the embodiment of the present application;

[0047] Figure 9 It is a schematic diagram of the interaction between modules provided in the embodiment of the present application;

[0048] Figure 10 It is a schematic diagram of the interaction between modules provided in the embodiment of the present application;

[0049] Figure 11 It is a schematic diagram of the interaction between modules provided in the embodiment of the present application;

[0050] Figure 12 A schematic flow diagram of interaction between modules provided by an embodiment of the present application;

[0051] Figure 13 A schematic flow diagram of interaction between modules provided by an embodiment of the present application;

[0052] Figure 14 A schematic diagram of the composition of an electronic device provided by an embodiment of the present application;

[0053] Figure 15 A schematic diagram of the composition of a chip system provided by an embodiment of the present application. Detailed implementation manners

[0054] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of this embodiment, unless otherwise stated, the meaning of "a plurality" is two or more.

[0055] When the application programs installed in the electronic device are running, images can be displayed through the display screen of the electronic device. Exemplarily, the application programs may include game applications and the like. In this way, when the game application is running, the game screen can be displayed through the display screen of the electronic device.

[0056] Exemplarily, taking the electronic device as a mobile phone and there is an application 1 installed in the electronic device, and the application 1 is a game application as an example.

[0057] Refer to Figure 1 , the electronic device can display the icon of the application 1 on the main interface. The user can indicate the electronic device to run the application 1 by inputting an operation 101 (such as a click operation) on the icon of the application 1.

[0058] In response to the user's operation 101, the electronic device can run the application 1. Thereafter, the application 1 can instruct the electronic device to perform rendering and display of the game screen by issuing a rendering instruction.

[0059] In an example such as Figure 1 , after the application 1 runs, the electronic device can display an interface 102 on the display screen.

[0060] Taking the game screen of the application 1 including continuously displayed image screens as an example. Then, the first frame image to the fifth frame image and subsequent images as shown in Figure 1 can be sequentially displayed on the interface 102 of the electronic device. Thus, by displaying continuous images, the user can obtain the visual experience of continuous game screens.

[0061] It should be noted that in the embodiments of the present application, the first frame to the fifth frame can be any consecutive five frames during the game operation. The first frame does not necessarily refer to the first image displayed after the game starts.

[0062] Combined with Figure 2 , an example of the internal processing mechanism of a frame of image (such as the first frame image) is given.

[0063] In an example such as Figure 2 , the electronic device can be configured with a rendering module 21, a buffer queue 22, a composition module 23, and a display screen 24.

[0064] Among them, the rendering module 21 is used to perform image rendering according to the rendering instructions issued by the application program (such as Application 1).

[0065] The buffer queue 22 can be used to temporarily store the rendering results obtained after image rendering.

[0066] The composition module 23 can be used to obtain the rendering results from the buffer queue 22 when the pre-configured timing arrives in the electronic device, and process the rendering results (such as composition processing). The composition module 23 can also be used to send the image data after the composition processing to the display screen 24, so that the display screen 24 can display the image accordingly.

[0067] In Figure 2 , an example of the interaction between modules during the processing and display of the first frame image is provided.

[0068] Such as Figure 2 shown, Application 1 can send a rendering instruction 201 to the rendering module 21. For example, the rendering instruction 201 can be used to instruct the rendering module 21 to draw the first frame image.

[0069] The rendering module 21 can call a component with image rendering capabilities in the electronic device (such as a graphics processing unit (GPU)) to perform image rendering according to the rendering instruction 201. Thus, the rendering module 21 can obtain the rendering result 202 of the first frame image corresponding to the rendering instruction 201.

[0070] The rendering module 21 can store the rendering result 202 of the first frame image in the buffer queue 22 to prepare for the subsequent display of the first frame image.

[0071] The composition module 23 can obtain the stored image from the buffer queue 22 when the pre-configured timing arrives. Exemplarily, the arrival of the pre-configured timing can correspond to the arrival of the Vsync signal generated by the electronic device. In this way, the composition module 23 can obtain the rendering result 202 from the buffer queue 22 according to the arrival of the Vsync signal.

[0072] After the synthesis module 23 obtains the rendering result 202, it can perform synthesis processing on the rendering result 202 to obtain the display image 203 of the first frame image.

[0073] Thus, the synthesis module 23 can send the display image 203 to the display screen 24 to realize the display of the first frame image.

[0074] It can be understood that for the processing and display processes of other frame images (such as the second frame to the fifth frame, etc.), the processing mechanism of the first frame image as shown can be referred to. Figure 2 shown.

[0075] It should be noted that in some cases, the timing of the application program (such as Application 1) issuing the rendering instruction 201 and the timing of the display screen 24 for display can also be controlled based on the Vsync signal. Taking the Vsync signal including signals V31 to V36 as an example.

[0076] Refer to Figure 3 . After the signal V31 arrives, Application 1 can issue a rendering instruction for the first frame image (such as the rendering instruction 201). Correspondingly, the drawing module 21 can perform rendering processing on the first frame image after the signal V31 arrives and complete the enqueue of the rendering result 202 of the first frame image into the buffer queue 22.

[0077] Next, when the signal V32 arrives, the synthesis module 23 can obtain the already enqueued rendering result 202 from the buffer queue 22. The synthesis module 23 can perform synthesis processing on the rendering result 202 and complete the synthesis processing of the first frame image before the next Vsync signal (such as signal V33) arrives. Thus, before the signal V33 arrives, the display screen 24 can receive the display image 204 of the first frame image.

[0078] In this way, the display screen 24 can display the first frame image according to the display image 204 after the signal V33 arrives.

[0079] It can be understood that each module component in the electronic device can perform subsequent frame image processing according to a similar processing mechanism as above.

[0080] Exemplarily, as Figure 3 shown, after the signal V32 arrives, Application 1 can issue a rendering instruction to instruct the electronic device to perform rendering processing on the second frame image. It can be understood that for different rendering instructions, due to different rendering contents and other situations, the rendering processing times of different frame images are different.

[0081] In this example, the rendering duration of the second frame image is greater than one Vsync period, and the rendering is completed after the signal V33 arrives. Correspondingly, the second frame image obtained after the rendering is completed can be transmitted to the buffer queue 22. The composition module 23 can obtain the rendering result of the second frame image from the buffer queue 22 for composition processing after the next Vsync signal (such as signal V34) arrives. Thus, within the Vsync period between signal V34 and signal V35, the composition module 23 can send the display image obtained through the composition processing to the display screen 24 for waiting to be displayed. The display screen 24 can display the second frame image after the next Vsync signal (signal V35) arrives.

[0082] Similarly, for the third frame image, Application 1 can issue a rendering instruction after the signal V34 arrives. Correspondingly, the rendering module 21 can perform rendering processing and obtain the rendering result of the third frame image and queue it into the buffer queue 22 before the signal V35 arrives. The composition module can obtain the rendering result of the third frame image for composition processing after the next Vsync signal (such as signal V35) arrives. Before the signal V36 arrives, the composition module transmits the display image of the third frame image to the display screen 24 for waiting to be displayed. The display screen can display the third frame image after the signal V36 arrives.

[0083] For the fourth frame image, Application 1 can issue a rendering instruction after the signal V35 arrives. Correspondingly, the rendering module 21 can perform rendering processing and obtain the rendering result of the fourth frame image and queue it into the buffer queue 22 before the signal V36 arrives. The composition module can obtain the rendering result of the fourth frame image for composition processing after the next Vsync signal (such as signal V36) arrives. Before the next Vsync signal arrives, the display image of the fourth frame image is transmitted to the display screen 24 for waiting to be displayed. The display screen can display the fourth frame image after the next Vsync signal arrives.

[0084] And so on, to implement processes such as Figure 1 the rendering processing, composition processing, and display of multiple frame images as shown.

[0085] It can be understood that during the process of the display screen 24 displaying according to the display image, it can be implemented in combination with the configured display queue.

[0086] Exemplarily, after obtaining the display image, the composition module 23 can transmit the display image to the display queue so that the display image can be temporarily stored in the display queue. After the next Vsync signal arrives, the display image can dequeue from the display queue and be transmitted to the display screen 24 for display.

[0087] Such as Figure 3As shown, examples of data stored in the display queue during each Vsync period in the above processing mechanism are also provided.

[0088] As Figure 3 shown, during the Vsync period between signal V32 and signal V33, the first frame of the image is completed for synthesis processing. Correspondingly, the display image of the first frame of the image enters the display queue. Thus, after signal V33 arrives, the display image of the first frame of the image dequeues and is displayed on the display screen 24. It can be understood that after the first frame of the image dequeues, the display queue is empty.

[0089] During the Vsync period between signal V33 and signal V34, no new display image enqueues. In this way, the display screen 24 can continue to display the first frame of the image.

[0090] During the Vsync period between signal V34 and signal V35, the second frame of the image is completed for synthesis processing. Correspondingly, the display image of the second frame of the image enters the display queue. Thus, after signal V35 arrives, the display image of the second frame of the image dequeues and is displayed on the display screen 24.

[0091] During the Vsync period between signal V35 and signal V36, the third frame of the image is completed for synthesis processing. Correspondingly, the display image of the third frame of the image enters the display queue. Thus, after signal V36 arrives, the display image of the third frame of the image dequeues and is displayed on the display screen 24.

[0092] During the Vsync period between signal V36 and subsequent Vsync signals (such as signal V37), the fourth frame of the image is completed for synthesis processing. Correspondingly, the display image of the fourth frame of the image enters the display queue. Thus, after the next Vsync signal (such as signal V37) arrives, the display image of the fourth frame of the image dequeues and is displayed on the display screen 24.

[0093] It can be understood that as Figure 3In the scenario shown, for the second frame image, Application 1 issues a rendering instruction when signal V32 arrives. And this second frame image can be displayed only after signal V35 arrives. Taking the response content of the user input operation included in this second frame image as an example. Then, after the user performs an input operation (such as an input operation before signal V32 arrives), it takes 3 Vsync cycles to see the response content on the display screen. In addition, due to the delay of this second frame image, similar problems will also occur in the display of subsequent frame images (such as the third frame image, the fourth frame image, etc.). For the first frame image earlier than the second frame image, although the rendering duration does not exceed one Vsync cycle, since both the composition process and the display need to wait for the Vsync signal to arrive, there is also a delay of 1 Vsync cycle from the issuance of the rendering instruction to the display of the first frame image.

[0094] In this way, in some scenarios with high requirements for followability and real-time image display (such as game scenarios, etc.), the delay of this display relative to the operation will cause relatively obvious problems.

[0095] Based on this, in the technical solution provided by the embodiments of the present application, the composition module 23 can be configured to: for the rendering result indicated by a preset application for rendering, without waiting for the next Vsync signal to arrive, after the rendering result is queued (such as entering the buffer queue), directly perform the dequeue composition of the rendering result. Thereby, the image can be composited earlier, and thus the image can be displayed earlier.

[0096] Among them, the preset application can correspond to: the application package name is included in the configured application whitelist. It can be understood that the buffer queues corresponding to each different application can be different. The name of the buffer queue can include a field indicating the application package name. Therefore, the composition module 23 can also determine whether the application corresponding to the currently queued rendering result is in the application whitelist through the name of the buffer queue.

[0097] In the embodiments of the present application, the composition module 23 determines to perform the composition process of the queued rendering result without waiting for the next Vsync signal, which can also be referred to as triggering, enabling, or activating the XSync scheme. This XSync scheme can also be referred to as XSync capability.

[0098] Exemplarily, taking Application 1 as included in the application whitelist and the composition module 23 triggering the XSync scheme for all rendering results of Application 1 as an example.

[0099] Taking Figure 3 the scenario shown as an example. Refer to Figure 4 , which is an example of the image processing logic after the solution provided by the embodiments of the present application takes effect.

[0100] Such asFigure 4 As shown, for the first frame image, after the signal V31 arrives, application 1 can issue a rendering instruction to start the rendering process. After the rendering process is completed, the composition module 23 can directly dequeue (such as dequeue from the buffer queue 22) the rendering result of the first frame image for composition processing without waiting for the next Vsync signal to arrive. Correspondingly, before the signal V32 arrives, the composition module 23 can transmit the display image of the first frame image to the display screen 24 for waiting to be displayed. Thus, after the signal V32 arrives, the display screen 24 can display the first frame image.

[0101] For subsequent frame images, it is similar, which can enable the composition processing of each frame image to be advanced.

[0102] Such as Figure 4 It also provides a logical example of enqueueing / dequeueing the display images in the display queue after the XSync scheme is enabled in this application.

[0103] Such as Figure 4 As shown, within the Vsync period from signal V31 to signal V32, the display image of the first frame image is enqueued. Compared with the existing scheme as Figure 3 shown, the enqueue time of the first frame image is advanced by 1 Vsync period. Correspondingly, after the signal V32 arrives, the display screen 24 can execute the display of the first frame image. Thus, compared with the existing scheme as Figure 3 shown, the display time of the first frame image is advanced by 1 Vsync period.

[0104] After that, within the Vsync period from signal V32 to signal V33, no new frame images are enqueued. The display screen 24 continues to display the first frame image.

[0105] Within the Vsync period from signal V33 to signal V34, the display image of the second frame image is enqueued. Correspondingly, after the signal V34 arrives, the display screen 24 can execute the display of the second frame image.

[0106] Within the Vsync period from signal V33 to signal V34, the display image of the second frame image is enqueued. Correspondingly, after the signal V34 arrives, the display screen 24 can execute the display of the second frame image.

[0107] Thus, compared with the existing scheme as Figure 3 shown, the display time of the second frame image is also advanced by 1 Vsync period.

[0108] In Figure 4 the example, it also provides a processing example of subsequent frame images.

[0109] Taking the third frame image as an example. After the signal V34 arrives, Application 1 can instruct the electronic device (such as the rendering module 21 of the electronic device) to perform the rendering process of the third frame image. Correspondingly, within the Vsync period from signal V34 to signal V35, the third frame image is rendered and queued. Then, the composition module 23 can directly dequeue and compose the rendering result of the third frame image without waiting for the next Vsync signal to arrive. The composition module 23 can obtain the composition result of the third frame image and complete enqueueing to the display queue after the signal V35 arrives.

[0110] After that, the rendering process and composition process of the fourth frame image can both be completed before the signal V36 arrives.

[0111] Corresponding to the display queue, between signal V34 and signal V35, the third frame image has not completed enqueueing. Therefore, after the signal V35 arrives, the display screen 24 can continue to display the second frame image. Within the period from signal V35 to signal V36, the third frame image and the fourth frame image are queued in sequence. Thus, after the signal V36 arrives, the display screen 24 can preferentially display the third frame image that was queued earlier.

[0112] After the signal V36 arrives, Application 1 can also continue the rendering of subsequent frame images. For example, Application 1 can instruct the rendering module 21 to perform the rendering process of the fifth frame image after the signal V36 arrives. In this example, the rendering process and composition process of the fifth frame image can be completed within one Vsync period.

[0113] In this way, corresponding to the display queue, after the signal V37 arrives, although the display image of the fifth frame has been queued, since the display image of the fourth frame was queued earlier and has not been dequeued. Therefore, after the signal V37 arrives, the display screen 24 can display the fourth frame image. After that, after the next Vsync signal arrives, the display screen 24 can continue to display the fifth frame image.

[0114] It should be noted that in the actual implementation process, the transmission of the above data such as rendering results and display images can all be implemented based on a buffer (Buffer).

[0115] Among them, one Buffer can correspond to a part of the storage space in the memory of the electronic device. The Buffer can be used to allocate to the corresponding process (such as the process of Application 1) / thread (such as the rendering thread of Application 1, etc.) before starting image rendering, and is used for storing intermediate data during image rendering, storing rendering results, etc.

[0116] Generally, in order to meet the storage space requirements of different applications, an electronic device can configure different buffer queues for different applications. The number of available Buffers corresponding to different buffer queues can be pre-configured. For example, the available Buffers for each buffer queue can be configured to be 3.

[0117] In this way, during the process of an application instructing the electronic device to perform image processing, the electronic device can rotate based on the 3 Buffers to achieve data storage and transmission.

[0118] As an example, take Application 1 which is configured with up to 3 available Buffers.

[0119] Take the processing of the first frame of image as an example. Refer to Figure 5 .

[0120] Before Application 1 issues a rendering instruction for the first frame of image, it can apply to the electronic device for a new Buffer. For example, the electronic device can configure Buffer1 for Application 1. In this way, Application 1 can carry the information of Buffer1 in the issued rendering instruction 201. Thereby instructing the drawing module to process the first frame of image on Buffer1. In some embodiments, the application for a new Buffer can be implemented by calling the dequeueBuffer() function.

[0121] Thus, the drawing module can perform rendering processing of the first frame of image on Buffer1. After the rendering processing is completed, the rendering result 202 of the first frame of image can be included on Buffer1.

[0122] Then, Buffer1 including the rendering result 202 can enter the buffer queue to realize the production of the Buffer and wait for consumption.

[0123] It should be noted that in different embodiments of the present application, the enqueueing of the Buffer into the buffer queue can be specifically implemented in any of the following forms:

[0124] The enqueueing of the Buffer ID into the buffer queue; the enqueueing of the image data of the rendering result stored in the Buffer into the buffer queue; the enqueueing of the file identifier corresponding to the Buffer into the buffer queue.

[0125] Among them, take the enqueueing of the Buffer ID into the buffer queue as an example. Different Buffers can be configured with different IDs. In this way, the storage address of the data stored in the Buffer in the memory can be uniquely determined through the Buffer ID. Furthermore, when the Buffer dequeues, the corresponding data (such as the rendering result) can be obtained through the obtained Buffer ID for subsequent processing (such as synthesis processing).

[0126] Take, for example, the enqueueing of the image data of the rendering result stored in the Buffer into the buffer queue. In this way, when processing the Buffer, the composition module can directly obtain the rendering result stored in the Buffer.

[0127] Take, for example, the enqueueing of the file identifier corresponding to the Buffer into the buffer queue. The file identifier can uniquely represent the storage address configured for the Buffer in the memory. When the rendering result is stored in the Buffer, the storage address represented by the file identifier is also the one where the corresponding rendering result is stored. In this way, when the Buffer is dequeued, the composition module can obtain the rendering result stored in the Buffer from the corresponding address through the file identifier for subsequent processing (such as composition processing).

[0128] The composition module can obtain the Buffer ID (such as 1) of Buffer1 from the buffer queue (such as buffer queue 22) for composition processing. Thus, the corresponding display image 203 can be obtained. For example, the display image 203 can still be stored in Buffer1. In some other embodiments, the display image 203 can be stored in another buffer with an ID different from Buffer1. It can be understood that in different embodiments, this process will occupy an available Buffer of an application 1.

[0129] After the composition processing is completed, Buffer1 including the display image 203 can be placed in the display queue to wait for display.

[0130] After the display image 203 is displayed, Buffer1 is released.

[0131] In this way, after application 1 applies for Buffer1, until the display image 203 in Buffer1 is displayed, since Buffer1 is being used, the number of available Buffers of application 1 is 2.

[0132] Combined with Figure 3 or Figure 4 's example, based on the Buffer usage mechanism as Figure 5 shown, the storage and transfer of data in the existing solution or the XSync solution provided by this application can be realized.

[0133] It can be understood that as Figure 4 shown, during the continuous processing of multiple frames of images, there may be a situation where multiple Buffers are being used simultaneously.

[0134] For example, as Figure 4As shown, during the Vsync period from signal V35 to signal V36, neither the third frame image nor the fourth frame image is displayed. Therefore, in the display queue, two Buffers are occupied. Thus, during this Vsync period, the number of available Buffers for Application 1 is 1.

[0135] With the continuous processing of multiple frame images, if the synthesis module completes the synthesis process quickly for multiple consecutive frames and sends the corresponding Buffers to the display queue. There will be a backlog of Buffers in the display queue. In this way, when all available Buffers are backlogged in the display queue, it will cause the application program (such as Application 1) to wait for the Buffers in the display queue to be released when it needs to apply for new Buffers for subsequent processing. This will also cause the user to wait longer to see the response content corresponding to the operation on the display screen after the user input operation.

[0136] To solve the above problems, the embodiment of this application, based on the XSync scheme as Figure 4 shown, also provides a scheme to avoid the too-fast speed of the synthesized image output by controlling the frequency at which the synthesis module starts the synthesis process. Furthermore, it avoids the backlog of Buffers in the display queue. Thus, it can be ensured that the number of available Buffers is greater than or equal to 1 when the application program needs to apply for Buffers. In this way, the application program can obtain the applied Buffers without waiting, and then quickly issue a rendering instruction to instruct the electronic device to perform subsequent operations. In the following description, this scheme is simply referred to as the Binder control scheme.

[0137] Combined with Figure 2 the inter-module process schematic in. In this application. After the synthesis module finishes the synthesis process of the existing Buffers, it needs to wait to receive the binding (Binder) information sent by the drawing module. Generally, the drawing module can send this Binder information to the synthesis module after completing the enqueue of the new Buffer into the buffer queue. In the embodiment of this application, the drawing module can control the frequency of the dequeue synthesis of the new Buffer by the synthesis module by controlling the sending timing of the Binder information.

[0138] Exemplarily, referring to Figure 6 , taking the scenario as Figure 4 shown as an example. In the case where the XSync scheme and the Binder control scheme are in effect, taking the Vsync period from signal V34 to signal V35 as an example.

[0139] After signal V34 arrives, combined with Figure 4According to the description in , Application 1 can issue a rendering instruction for the third frame of the image. Correspondingly, the rendering module can perform the rendering process of the third frame of the image and complete the rendering result of the third frame of the image before the signal V35 arrives.

[0140] Taking the example that the rendering result of the third frame of the image is stored in Buffer1. In this way, before the signal V35 arrives, this Buffer1 can complete the enqueue operation for the buffer queue of Application 1 (such as buffer queue 22).

[0141] It can be understood that, combined with Figure 4 the description of the XSync scheme in , for the composition module, it can determine to perform the dequeue composition of this Buffer1 without waiting for the next Vsync signal based on the enqueue of Buffer1 and the fact that Application 1 is included in the application whitelist.

[0142] For example, after the rendering module enqueues Buffer1, it can send Binder information 1 to the composition module to indicate that a new Buffer has been enqueued. Correspondingly, the composition module can determine that Application 1 corresponding to the new Buffer is included in the application whitelist, and then trigger the XSync scheme for the newly enqueued Buffer1. The composition module can also perform the dequeue composition of this Buffer1 according to the received Binder information 1.

[0143] In this example, before sending Binder information 1, the rendering module can obtain the duration between the enqueue time of the current frame of the image and the enqueue time of the previous frame of the image. For example, the rendering module can obtain the duration T0 between the enqueue time of the rendering result of the third frame of the image (or described as the Buffer of the third frame of the image) and the enqueue time of the rendering result of the second frame of the image (or described as the Buffer of the second frame of the image). In the case where the duration T0 is greater than or equal to the preset duration T2, the rendering module can directly send this Binder information 1 to the composition module after the Buffer of the third frame of the image is enqueued. In some implementations, the preset duration T2 can be the frame interval (such as 8.33 ms) corresponding to the frame rate of the current Application 1 (such as 120 Hz, etc.).

[0144] Thereafter, the composition process and the display process for the third frame of the image can refer to Figure 4 the description in . Details are not described again.

[0145] Next, for the fourth frame of the image. Application 1 can, after the signal V35 arrives, instruct the rendering module to perform the rendering process of the fourth frame of the image. Correspondingly, before the signal V36 arrives, the enqueue of the Buffer of this fourth frame of the image is completed.

[0146] Similar to the above-mentioned Binder information sending mechanism, before sending the Binder information 2 corresponding to the 4th frame of the image, the rendering module can determine the duration T1 between the time when the buffer of the 4th frame of the image is queued and the time when the buffer of the 3rd frame of the image is queued. If the duration T1 is less than the duration T2, it indicates that if the XSync scheme is immediately triggered to perform the synthesis process of the buffer of the 4th frame of the image, it may cause the buffer to accumulate in the display queue. Therefore, the rendering module can delay sending the Binder information 2.

[0147] For example, after the buffer of the 3rd frame of the image is queued, after a duration T2 (corresponding to after the signal V36 arrives), the rendering module sends the Binder information 2 to the synthesis module.

[0148] Therefore, the synthesis module can start the synthesis process of the buffer of the 4th frame at least after a duration T2 after starting the synthesis process of the buffer of the 3rd frame.

[0149] As Figure 6 shown, since the time when the synthesis module receives the Binder information 2 is postponed, compared to the example as Figure 4 shown, the display image of the 4th frame of the image can be synthesized and sent to the display queue within the Vsync period between the signal V36 and the signal V37. Therefore, although the display effect is the same as that after the XSync scheme as Figure 4 shown is enabled, within the Vsync period from the signal V35 to the signal V36, only one buffer (such as the buffer storing the display image of the 3rd frame of the image) in the display queue is occupied. In addition, within the Vsync period from the signal V36 to the signal V37, only one buffer (such as the buffer storing the display image of the 4th frame of the image) in the display queue is occupied.

[0150] In this way, through the cooperation of the XSync scheme and the Binder control scheme, compared to the existing scheme as Figure 3 shown, the image can be displayed on the display screen earlier, and the accumulation of the buffer in the display queue can also be avoided.

[0151] The XSync scheme and the Binder control scheme provided by the embodiments of the present application will be described in detail below.

[0152] It should be noted that the solution provided in the embodiments of this application can be applied to an electronic device. The electronic device may include at least one of a mobile phone, a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, or a smart city device. The embodiments of this application do not impose any special restrictions on the specific type of the electronic device. In this application, the electronic device may be configured with a display screen for image display.

[0153] In some embodiments, the electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) connector, a charging management module, a power management module, a battery, antenna 1, antenna 2, a mobile communication module, a wireless communication module, an audio module, a speaker, a receiver, a microphone, a headphone interface, a sensor module, a key, a motor, an indicator, a camera module, a display screen, and a subscriber identification module (SIM) card interface, etc. The sensor module may include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0154] The processor may include one or more processing units. For example, the processor may include an application processor (AP), a modem processor (Modem), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor (BP or BBP), and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0155] The processor may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0156] A memory may also be provided in the processor for storing instructions and data. In some embodiments, the memory in the processor may be a cache memory. This memory can save the instructions or data that the processor has used or uses frequently. If the processor needs to use this instruction or data, it can be directly called from this memory. This avoids repeated accesses, reduces the waiting time of the processor, and thus improves the efficiency of the system.

[0157] It should be noted that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than the above examples, or combine certain components, or split certain components, or have different component arrangements. Each component may be implemented in hardware, software, or a combination of software and hardware.

[0158] In some embodiments, based on the above composition, the electronic device may implement the display function through the GPU, the display screen, and the application processor, etc. The GPU is a microprocessor for image processing, connecting the display screen and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor may include one or more GPUs, which execute program instructions to generate or change the display information.

[0159] The display screen is used to display images, videos, etc. The display screen includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include one or more display screens.

[0160] The internal memory can be used to store computer-executable program code, and the executable program code includes instructions. The internal memory can include a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as game applications, etc.). The data storage area can store the data created during the use of the electronic device (such as image data, audio data, phone book, etc.). In addition, the internal memory can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor executes various functional methods or data processing of the electronic device by running the instructions stored in the internal memory and / or the instructions stored in the memory provided in the processor.

[0161] The touch sensor, also known as the "touch control device". The touch sensor can be disposed on the display screen, and the touch screen, also known as the "touch screen", is composed of the touch sensor and the display screen. The touch sensor is used to detect touch operations acting on it or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen. In some other embodiments, the touch sensor can also be disposed on the surface of the electronic device, at a different position from the display screen.

[0162] In some embodiments of the present application, the touch sensor may be implemented by a touch panel (TP). Taking the display screen as an LCD as an example. In some implementations, the TP may be integrated with the LCD to form a TP-LCD. In other implementations, the TP may be separated from the LCD. In this way, the user can, according to the picture displayed on the LCD (such as a game picture), input an operation by touching the corresponding position on the picture. Correspondingly, the TP can receive the operation and send operation information (such as operation type, position information, etc.) to the application for subsequent processing.

[0163] In the above example, the composition of an electronic device in the present application is provided. Embodiments of the present application also provide another composition of an electronic device. Exemplarily, referring to Figure 7 , which is a schematic diagram of the composition of another electronic device provided by an embodiment of the present application.

[0164] In an example such as Figure 7 , the software system of the electronic device may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. Embodiments of the present application take the system of the layered architecture as an example to exemplarily illustrate the software structure of the electronic device.

[0165] As Figure 7 shown, the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom are the application layer, the application framework layer, Android runtime (ART) and native C / C++ libraries, the Hardware Abstract Layer (HAL), and the kernel layer.

[0166] The following will be described separately.

[0167] The application layer may include a series of application packages. The application layer may also be referred to as the application layer or the APP layer. As Figure 7 shown, the application packages may include applications such as games, calendars, maps, WLAN, music, text messages, calls, navigation, Bluetooth, and videos. For example, the game application package may correspond to the game application in the foregoing example, such as Application 1.

[0168] The application framework layer may also be referred to as the framework layer or the Framework layer. This framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0169] As shown Figure 7 in the figure, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, etc.

[0170] The window manager provides a window management service (Window Manager Service, WMS), and the WMS can be used for window management, window animation management, surface management, and as a transfer station for the input system.

[0171] The content provider is used to store and obtain data, and make this data accessible to applications. This data can include videos, images, audio, incoming and outgoing calls, browsing history and bookmarks, phone books, etc.

[0172] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon can include a view for displaying text and a view for displaying pictures. As an example, one or more surface views (Surface) can be configured in the view system. Each surface view can be correspondingly configured with one or more texture maps. Each texture map can be used to store part or all of the image content in a frame of an image. It should be noted that in the specific implementation process of this application, the image data in the surface view and / or the texture map can be stored in the memory of the electronic device. Correspondingly, the electronic device can manage each data, such as writing and reading, through the file identifier of each data in the memory.

[0173] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0174] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify that the download is complete, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or a scroll bar text, such as a notification of a background-running application, or a notification that appears in the form of a dialogue window on the screen. For example, prompting text information in the status bar, emitting a prompt tone, vibrating the electronic device, flashing the indicator light, etc.

[0175] The activity manager can provide an activity management service (Activity Manager Service, AMS), and the AMS can be used for the startup, switching, scheduling of system components (such as activities, services, content providers, broadcast receivers), and the management and scheduling of application processes.

[0176] The input manager can provide an Input Manager Service (IMS), and the IMS can be used to manage the input of the system, such as touch screen input, key input, sensor input, etc. The IMS retrieves events from the input device nodes and, through interaction with the WMS, distributes the events to the appropriate windows. In some embodiments, the IMS may include components such as an IMSReader and an IMSDispatcher (IMS scheduling module). As an example, after the TP of the electronic device receives the user's operation, it can send the operation information to the IMSReader. The IMSReader can send the operation information to the IMSDispatcher for centralized management and distribution. For example, the IMSDispatcher can send the operation information to the application displayed in the foreground (such as a game application). So that the game application can respond to the operations input by the user. For example, the game application generates rendering instructions for the next frame of the image based on the operations input by the user, etc.

[0177] The Android runtime includes a core library and the Android runtime. The Android runtime is responsible for converting the source code into machine code. The Android runtime mainly includes the ahead of time (AOT) compilation technology and the just in time (JIT) compilation technology.

[0178] The core library is mainly used to provide the functions of the basic Java class library, such as libraries for basic data structures, mathematics, IO, tools, databases, networks, etc. The core library provides APIs for users to develop Android applications.

[0179] The native C / C++ libraries can include multiple functional modules. For example: surface manager, Media Framework, libc, OpenGL ES, SQLite, Webkit, etc.

[0180] Among them, the surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The Media Framework supports the playback and recording of multiple common audio and video formats, as well as static image files, etc. The media library can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. OpenGL ES provides the drawing and operation of 2D and 3D graphics in the application. SQLite provides a lightweight relational database for the applications of the electronic device.

[0181] The Hardware Abstraction Layer runs in the user space, encapsulates the kernel layer drivers, and provides call interfaces to the upper layer. Exemplarily, the Hardware Abstraction Layer may include a display module, an audio module, a camera module, a Bluetooth module, etc. In some embodiments, the Hardware Abstraction Layer may further include a touch module.

[0182] The kernel layer is the layer between hardware and software. The kernel layer at least includes a display driver (such as DRM Driver), a camera driver, an audio driver, and a Bluetooth driver. In some embodiments, the kernel layer may further include a touch driver.

[0183] It should be noted that the software composition shown as Figure 7 is only an exemplary illustration and does not constitute a limitation on the electronic device involved in the embodiments of the present application. In some other embodiments, the electronic device may also have other software compositions.

[0184] Exemplarily, referring to Figure 8 , it is a schematic diagram of the composition of another electronic device provided by the embodiments of the present application. In the example as Figure 8 , the upper-layer components composed of software modules and various possible configurations in the hardware components are shown at the same time.

[0185] As Figure 8 shown, in this example, the application layer may include multiple application programs. For example, the multiple application programs may include Application 1, Application 2, etc. Among them, Application 1 may be a game application. Application 2 may be a floating window application or a multi-window application. For example, during the running of Application 1, Application 2 may provide relevant controls (such as brightness adjustment, communication function configuration, etc.) during the running of the game application by displaying a window on the interface. In some embodiments, Application 2 may be a system-level application such as a game center installed in the electronic device.

[0186] The framework layer of the electronic device may be configured with a drawing module, an SF module, a view system, an IMS module, etc.

[0187] Among them, the drawing module is used to perform image rendering in response to a rendering instruction issued by an application program (such as Application 1). The drawing module may correspond to the drawing module 21 in the foregoing example.

[0188] In some implementations, the drawing module may perform rendering processing according to the rendering instruction through the GPU configured in the hardware layer of the electronic device, so as to obtain the corresponding rendering result.

[0189] In the present application, the drawing module may also be used to send Binder information to the SF module after a new Buffer is queued. The Binder information may be used to indicate that a new Buffer is queued.

[0190] In combination with the foregoing description of the Binder control scheme, before sending Binder information, the drawing module can determine the time difference between the current enqueue of the Buffer and the enqueue of the Buffer in the previous frame of the image, and directly send the Binder information when the time difference is greater than a preset duration (such as duration T2). Alternatively, when the time difference is less than the preset duration, wait until the preset duration ends and then send the Binder information.

[0191] Thus, by controlling the sending of Binder information, the synthesis frequency of the SF module is controlled. Furthermore, the accumulation of Buffers in the display queue caused by a relatively high synthesis frequency is avoided.

[0192] In the embodiments of the present application, the SF module is namely SurfaceFlinger. In some implementations, the SF module can correspond to the synthesis module 23 as shown in Figure 4 Figure 23.

[0193] The SF module can be used to generate a Vsync signal according to information such as the currently displayed application and the refresh rate of the current display screen.

[0194] The SF module can be a consumer of the produced Buffers, and obtain the enqueued Buffers from the buffer queue (BufferQueue) for consumption and display. Exemplarily, the processing mechanism for consumption and display can correspond to that the SF performs synthesis processing according to the rendering result corresponding to the Buffer obtained from the buffer queue, so as to obtain the corresponding display image. The SF can directly or indirectly send the display image to the display screen of the electronic device for display.

[0195] In the embodiments of the present application, the SF module can have the ability to confirm that a new Buffer has been enqueued.

[0196] Exemplarily, in some embodiments, the SF module can confirm that a new Buffer has been enqueued according to the received Binder information.

[0197] In other embodiments, the SF module can confirm that a new Buffer has been enqueued through the identification field in the BufferTX module configured in the framework layer.

[0198] Wherein, the BufferTX module can be configured with identification fields corresponding one-to-one to the buffer queues. For any one of the identification fields, when it is configured as the first value (such as 0), it indicates that there is no available Buffer in the corresponding buffer queue. Correspondingly, when the identification field is configured as the second value (such as 1), it indicates that there is 1 available Buffer in the corresponding buffer queue. The available Buffer can be produced by the drawing module indicated by the application corresponding to the buffer queue.

[0199] Thus, the SF module can determine that a new Buffer is queued according to the identification field in the BufferTX module being configured as the second value.

[0200] The SF module can also determine whether to trigger the XSync scheme for the newly queued Buffer when a new Buffer is queued.

[0201] Exemplarily, in some embodiments, take the currently running application including only Application 1 as an example. The corresponding buffer queue is Buffer Queue BQ1. The SF module can determine that a new Buffer enters Buffer Queue BQ1 according to the identification field 1 of this Buffer Queue BQ1 in the BufferTX module being configured as the second value. Then, the SF module can confirm to trigger the XSync scheme for the Buffer newly entering Buffer Queue BQ1 according to Application 1 being included in the preset application whitelist. For example, the SF module can perform the synthesis process on the Buffer newly entering Buffer Queue BQ1 without waiting for the next Vsync signal to arrive.

[0202] In some other embodiments, take the currently running application including multiple applications as an example. Such as Application 1 and Application 2. Application 1 corresponds to Buffer Queue BQ1, and the identification field in the BufferTX module is Identification Field 1. Application 2 corresponds to Buffer Queue BQ2, and the identification field in the BufferTX module is Identification Field 2.

[0203] The SF module can determine that a new Buffer enters Buffer Queue BQ1 according to the identification field 1 of Buffer Queue BQ1 in the BufferTX module being configured as the second value.

[0204] The SF module can also determine whether there are other available Buffers. For example, there is also an available Buffer in the current Buffer Queue BQ2. Thus, the SF module can determine whether the applications corresponding to all buffer queues including available Buffers are all included in the application whitelist. For example, the SF module can determine whether Application 1 and Application 2 are both included in the application whitelist.

[0205] Take Application 1 and Application 2 both being included in the application whitelist as an example. The SF module can trigger the XSync scheme for the Buffer newly entering Buffer Queue BQ1. For example, the SF module can obtain all currently available Buffers for synthesis processing without waiting for the next Vsync signal to arrive.

[0206] Take the example that Application 2 is not included in the application whitelist. The SF module may not trigger the XSync scheme for the Buffer newly entered into the buffer queue BQ1. For example, the SF module may wait for the next Vsync signal to arrive and then obtain all available Buffers currently for synthesis processing.

[0207] In the above example, after determining that it is necessary to trigger the XSync scheme for the newly queued Buffer, the SF module may perform the synthesis processing on the newly queued Buffer without waiting for the next Vsync signal to arrive.

[0208] In the embodiment of the present application, the SF module may wait to receive the Binder information from the drawing module before specifically performing the synthesis processing on the newly queued Buffer.

[0209] Exemplarily, take the example that the SF module determines to trigger the XSync scheme for Buffer1 newly entered into the buffer queue BQ1. The SF module may also wait for the arrival of the Binder information corresponding to the enqueue of Buffer1, and thus perform the dequeue and synthesis processing on Buffer1.

[0210] In such as Figure 8 In the example, a view system may also be configured in the framework layer. The view system may include one or more Surface views. When the drawing module performs the rendering processing of a frame of image, it may be drawn based on one or more Surfaces. In this way, after completing the rendering operation of a frame of image, the image obtained by the rendering processing may be correspondingly stored in one or more Surfaces in the view system.

[0211] In the embodiment of the present application, one Buffer may correspond to one or more Surfaces. In the case where one Buffer corresponds to multiple Surfaces, then the multiple Surfaces may respectively store partial contents of the frame image corresponding to the Buffer. The set of all contents in the multiple Surfaces may constitute the content of a frame of image corresponding to the Buffer. In this way, after the drawing module completes the rendering processing of a frame of image, it may realize the enqueue of one Buffer, and at the same time obtain one or more Surfaces storing the rendering result in the view system.

[0212] Exemplarily, when the SF module consumes a Buffer, by obtaining the file identifier of the Buffer from the BufferQueue, it may obtain the image data stored in the Surface corresponding to the Buffer from the address indicated by the file identifier for synthesis processing.

[0213] Such as Figure 8As shown, an IMS module can also be configured in the framework layer. Combining with Figure 7 the description in

[0214] this IMS module can include an IMSReader module and an IMS scheduling module. This IMS scheduling module is also the IMS Dispatcher.

[0215] In some implementations of this application, the IMS module can be used to implement the transfer of operation information input by the user from the lower layer to the upper layer. This process will be described in detail later.

[0216] In the following description, it is taken as an example that the composition processing is performed independently by the SF module.

[0217] In some other embodiments of this application, a hybrid renderer can also be configured in the abstract layer to cooperate in implementing the composition processing in different scenarios.

[0218] In the electronic device composition such as Figure 8 provided, a touch drive, a display drive, etc. can be configured in the kernel layer. Among them, the touch drive can be used to implement the control of the touch panel (TP) by the upper-layer software and the information transfer from the TP to the upper-layer software module. The display drive can be the (DRM Driver). The display drive can directly or indirectly receive the display image sent from the SF module and control the corresponding display screen to display according to the display image.

[0219] In the electronic device, various hardware components in the hardware layer as Figure 8 shown can be configured to support the above software modules to implement corresponding functions.

[0220] Exemplarily, a touch panel, a display screen, a GPU, a memory, etc. can be configured in the electronic device.

[0221] In the embodiments of this application, a display queue can also be configured in the memory. This display queue can be managed by the display drive. For example, the display drive can put the display image into the display queue every time it receives a display image. Thus, when the next Vsync signal arrives, the display drive obtains the display image that first enters the display queue from the display queue and sends it to the display screen for display.

[0222] The solutions provided in the embodiments of this application can all be applied to the aforementioned Figure 7 or Figure 8 the electronic devices provided.

[0223] In the following description, taking the application of this solution to the electronic device as shown in Figure 8 as an example, the implementation of the solution provided in the embodiments of this application will be illustrated.

[0224] Exemplarily, referring to Figure 9 , it is an example of the interaction between modules provided in the embodiments of this application. Through the implementation of the solution as shown in Figure 9 , it can support the reporting of the operations input by the user during the running of the application to the application. In the example as shown in Figure 9 , the currently running application is Application 1 as an example.

[0225] As shown in Figure 9 , this solution may include:

[0226] S901. The touchpad of the electronic device receives the operation OP1 input by the user. The touchpad sends the operation information 91 to the touch drive.

[0227] Exemplarily, when the display screen of the electronic device displays a certain interface during the running of Application 1, it can receive the operation OP1 input by the user. As a possible implementation, the operation OP1 may include an operation of clicking on a control displayed in the current interface. Through this operation OP1, the user can instruct the game character currently controlled to perform corresponding actions, such as casting the skill A corresponding to the control, etc.

[0228] Correspondingly, the touchpad can generate the operation information 91 according to the operation OP1. In some embodiments, the operation information 91 may include the operation type of the operation OP1 (such as a click operation or a swipe operation, etc.), the position information of the operation OP1 on the display screen, etc.

[0229] Thus, the touchpad can, through the touch drive, realize the transfer of the currently received operation information 91 to the upper-layer application module.

[0230] S902. The touch drive sends the operation information to the IMSReader module.

[0231] S903. The IMSReader module sends the operation information 91 to the IMS scheduling module.

[0232] In some embodiments, the IMSReader module can monitor and obtain the information reported by each drive. For example, the IMSReader module can obtain the operation information 91 reported by the touch drive.

[0233] In this way, the IMSReader module can transmit the operation information 91 to the IMS scheduling module for centralized processing.

[0234] In some embodiments, the IMS scheduling module can determine that the operation information 91 corresponds to an operation input by the user to Application 1 based on the currently top-displayed interface being the interface corresponding to Application 1.

[0235] S904. The IMS scheduling module sends the operation information 91 to Application 1.

[0236] Thus, after the user inputs the operation OP1 during the use of Application 1, Application 1 can obtain the operation information 91 input by the user. Furthermore, Application 1 can perform subsequent operations based on the operation information 91.

[0237] Exemplarily, Application 1 can, according to the operation information 91, instruct each module in the electronic device to cooperate with each other to display an image corresponding to the operation information 91.

[0238] Combined Figure 6 with the example in. Taking Application 1 receiving the operation information 91 during the Vsync period between signal V33 and signal V34 as an example. Correspondingly, Application 1 can, according to the operation information 91, after the arrival of signal V34, instruct the electronic device to perform processing such as rendering and displaying the third frame of the image. The third frame of the image can include the screen effect corresponding to the operation OP1, such as the screen effect corresponding to releasing skill A. It can be understood that only the third frame of the image is used as an example in this example. In the actual implementation process, the third frame of the image can also be replaced with any other frame of the image, such as the Nth frame of the image.

[0239] As a possible implementation, referring to Figure 10 , another example of interaction between modules provided by the embodiments of the present application. By implementing the solution as shown in Figure 10 , it can make the production and enqueueing of the Buffer of the image corresponding to the operation information 81 (such as the third frame of the image).

[0240] As Figure 10 shown, this solution can include:

[0241] S1001. Application 1 sends a rendering instruction 101 to the drawing module.

[0242] Exemplarily, Application 1 can generate the rendering instruction 101 according to the operation information 81. Thus, through the rendering instruction 101, Application 1 can instruct the electronic device to perform rendering processing on a frame image (such as the third frame of the image) including the screen effect corresponding to the operation OP1.

[0243] S1002. The rendering module calls the GPU to perform rendering processing according to the rendering instruction 101 to obtain the rendering result 102.

[0244] S1003. The rendering module stores the rendering result 102 in the surface view S1.

[0245] Exemplarily, the rendering module may send the rendering instruction 101 to the GPU so that the GPU performs the rendering processing of the third frame image according to the rendering instruction 101.

[0246] Through this rendering processing, one or more texture maps corresponding to the third frame image can be obtained.

[0247] In this example, it is taken that the rendering result 102 corresponding to the rendering instruction 101 is stored in the surface view S1.

[0248] S1004. The rendering module stores the Buffer of the third frame image in the buffer queue BQ1.

[0249] Combined with the above description of Buffer enqueueing. In this example, the rendering module can enqueue the Buffer of the third frame image into the buffer queue BQ1 by storing the Buffer ID of the third frame image, or the rendering result of the third frame image, or the file identifier corresponding to the third frame image in the buffer queue BQ1.

[0250] In this way, the rendering module can complete the production enqueueing of the Buffer of the third frame image.

[0251] Combined with the description of the BufferTX module in the foregoing example, in some embodiments of the present application, an identification field 1 of the buffer queue BQ1 may be configured in the BufferTX module. After the Buffer1 corresponding to S1004 is enqueued into the buffer queue BQ1, the identification field 1 of the buffer queue BQ1 in the BufferTX module can be configured to 1. Thus, it is indicated that there is a new Buffer enqueued in the current buffer queue BQ1 waiting to be consumed.

[0252] S1005. The rendering module sends the Binder information 3 to the SF module. The Binder information 3 is used to indicate the enqueueing of the Buffer of the third frame image by the SF module.

[0253] In the present application, after the Buffer of the third frame image is enqueued, the rendering module can also determine whether to immediately send the Binder information 3 corresponding to the third frame image to the SF module in combination with the enqueueing time of the Buffer of the previous frame image.

[0254] Exemplarily, take the buffer of the second frame image enqueued at time A2 and the buffer of the third frame image enqueued at time A3 as an example.

[0255] In this way, the rendering module can determine the magnitude relationship between the duration T3 between time A3 and time A2 and the preset duration T2, and thereby determine the timing of sending the Binder information 3.

[0256] Taking the duration T3 being greater than the duration T2 as an example. After the buffer of the third frame image is enqueued, the rendering module can immediately send the Binder information 3 to the SF module. That is, the rendering module can send the Binder information 3 to the SF module at time A3.

[0257] For example Figure 10 The above example is described taking the duration T3 being greater than the duration T2 as an example. In some other embodiments, take the duration T3 being less than the duration T2 as an example. Correspondingly, the rendering module may not send the Binder information 3 to the SF module at time A3. Correspondingly, the rendering module can send the Binder information 3 to the SF module at time A3' after time A3. Among them, the duration between time A3' and time A2 is the duration T2. That is, when the duration between the enqueue time of the current frame image and the enqueue time of the previous frame image is less than the duration T2, the rendering module waits until the previous frame image has been enqueued for a duration of T2, and then sends the Binder information 3 to the SF module.

[0258] Refer to Figure 11 , which is another schematic diagram of the interaction between modules provided by the embodiments of the present application. Through the solution shown above, the electronic device can achieve the fast consumption and display of the newly enqueued buffer. Figure 11 As shown above, the solution may include:

[0259] For example Figure 11 As shown, the solution may include:

[0260] S1101. The SF module obtains the buffer of the third frame from the buffer queue BQ1.

[0261] Exemplarily, after the buffer of the third frame is enqueued, the SF module can sense that a new buffer has been enqueued. For example, the SF module can determine that a channel buffer has entered the buffer queue BQ1 according to the identification field 1 corresponding to the buffer queue BQ1 in the BufferTX module being configured as a second value.

[0262] The SF module can determine that the application 1 corresponding to the buffer queue BQ1 is in the application whitelist according to the name of the buffer queue BQ1 where the newly enqueued buffer is located. That is, the newly enqueued buffer of the third frame is indicated for rendering by application 1. In this example, it is assumed that the other buffer queues are all empty.

[0263] In this way, the SF module can determine an XSync scheme for triggering the Buffer of the third-frame image.

[0264] Combined with Figure 10 the description in, taking the case where the SF module has received Binder information 3 as an example.

[0265] Thus, the SF module can obtain the newly enqueued Buffer from the buffer queue BQ1 without waiting for the next Vsync signal to arrive. For example, the SF module can obtain the file identifier corresponding to the Buffer of the third frame from the buffer queue BQ1. This file identifier can correspond to the storage address of the Buffer of the third frame in the memory, or correspond to the storage address of the rendering result 102 in the memory.

[0266] S1102. The SF module obtains the rendering result 102 in the surface view S1.

[0267] Thus, the SF module can perform a composition process based on the rendering result 102, and further obtain the display image 111 of the frame image indicated by the rendering instruction 101.

[0268] It should be noted that in the description such as Figures 10 to 11 taking the case where the frame image indicated by the rendering instruction 101 is stored in the surface view S1 after rendering as an example. In some other embodiments, the result after rendering by the drawing module can be stored in multiple surface views. In this way, the SF module can obtain the rendering results in the multiple surface views according to the dequeued Buffer for composition processing, thereby obtaining the content of a complete frame image.

[0269] S1103. The SF module transmits the display image 111 to the HWC.

[0270] S1104. The HWC transmits the display image 111 to the display driver.

[0271] In this example, after the SF module completes the composition process, it can transmit the obtained display image 111 to the display driver through the HWC and wait for display.

[0272] S1105. The display driver transmits the display image 111 to the display screen. The display screen displays according to the display image 111.

[0273] Exemplarily, after receiving the image to be displayed 111, the display driver may store the image to be displayed 111 in the queue for the image to be displayed. So that after the next Vsync signal arrives, the earliest received first image to be displayed can be obtained from the queue for the image to be displayed for display. In this example, it is taken that after the image to be displayed 111 enters the queue for the image to be displayed, only the image to be displayed 111 is included in the queue for the image to be displayed.

[0274] In this way, after the next Vsync signal arrives, the display driver can obtain the image to be displayed 111 from the queue for the image to be displayed, and transmit the image to be displayed 111 to the display screen for display. Thus, the display of the third frame image on the display screen can be realized.

[0275] The above Figures 9 to 11 Taking the processing of the third frame image as an example, the rendering, composition, and display processes of this frame of image are illustrated. Similarly, for other frame images, the electronic device can also adopt a similar solution to implement, in combination with the XSync solution and the Binder control solution, to realize the fast composition processing of the Buffer, and at the same time avoid the accumulation of the Buffer in the display queue due to the too high frequency of compositing and sending the display.

[0276] In order to be able to more clearly illustrate the solution provided by the embodiments of the present application, the following will be described through Figure 12 and Figure 13 The schematic diagram of the interaction process between modules shown, to illustrate the rendering and sending display processes of the above-mentioned third frame image and the subsequent fourth frame image.

[0277] As Figure 12 shown, a schematic diagram of the interaction process between modules of the third frame image is provided. As Figure 12 shown, this solution may include:

[0278] S1201. The touchpad receives the operation OP1.

[0279] Exemplarily, before the touchpad receives the operation OP1, the second frame image may be displayed on the display screen.

[0280] In this way, the operation OP1 may be an operation input by the user on the second frame image. Thus, the touchpad may generate operation information 91 according to the operation OP1 input by the user. In some embodiments, the operation information 91 may include the operation type, position information, etc. of the operation OP1.

[0281] S1202. The touchpad sends the operation information 91 to the application 1.

[0282] Exemplarily, in combination with Figure 9 the description in, the touchpad may send the operation information 91 to the application 1 through the touch drive and the IMS module.

[0283] In this example, after Application 1 receives operation information 91, the next Vsync signal (such as signal V34) arrives.

[0284] S1203. Application 1 generates rendering instruction 101.

[0285] Exemplarily, Application 1 can generate a new rendering instruction based on the arrival of signal V34 and instruct other components of the electronic device to perform the rendering of a new image.

[0286] In some embodiments, Application 1 can generate rendering instruction 101 according to the arrival of signal V34 and based on the received operation information 91.

[0287] S1204. Application 1 sends rendering instruction 101 to the drawing module.

[0288] Thus, Application 1 can instruct the drawing module to draw a new image through rendering instruction 101. For example, the new image can be an image subsequent to the currently displayed second frame image. Such as the third frame image.

[0289] S1205. The drawing module performs rendering processing to obtain rendering result 102.

[0290] Exemplarily, the drawing module can perform rendering processing with the GPU configured in the electronic device according to rendering instruction 101, so as to obtain the corresponding rendering result 102.

[0291] It should be noted that, combined with Figure 6 the example in, in this example, the drawing module performs the rendering processing corresponding to rendering instruction 101, which can span two Vsync cycles and complete the rendering processing before the end of the second Vsync cycle (such as the Vsync cycle between signal V35 and signal V36).

[0292] S1206a. The drawing module enqueues rendering result 102 into buffer queue BQ1. In some embodiments, the drawing module can store the Buffer storing rendering result 102 into buffer queue BQ1.

[0293] Among them, buffer queue BQ1 can be a buffer queue configured corresponding to Application 1. Thus, the production enqueue of the Buffer of the third frame image can be realized.

[0294] S1206b. The drawing module sends Binder information 3 to the SF module. This Binder information 3 is used to indicate that a new Buffer has entered buffer queue BQ1.

[0295] In this example, the execution of S1206a (i.e., enqueueing the rendering result 102) is performed after a time period T2 or longer after the rendering result of the previous frame image is enqueued.

[0296] In this way, the drawing module can execute S1206a to enqueue a new Buffer, and can also send Binder information 3 to the SF module through S1206b.

[0297] S1207. The BufferTX module determines that a new Buffer has been enqueued.

[0298] S1208. The BufferTX module configures the identification field 1 to 1.

[0299] Among them, the identification field 1 can be a field in the BufferTX module that identifies the Buffer in the buffer queue BQ1.

[0300] In this example, after a new Buffer (such as the Buffer corresponding to the rendering result 102) is enqueued in the buffer queue BQ1, the BufferTX module can configure the identification field 1 corresponding to the buffer queue BQ1 from 0 to 1. Thus, it can be indicated that the available Buffer in the buffer queue BQ1 changes from empty to 1.

[0301] S1209. The SF module determines that the identification field 1 is configured to 1.

[0302] In this example, the SF module can monitor the changes of each identification field in the BufferTX module.

[0303] For example, the SF module can determine that there is 1 Buffer to be consumed in the corresponding buffer queue BQ1 according to the identification field 1 in the BufferTX module being configured to 1.

[0304] S1210. The SF module determines that all applications corresponding to the available Buffers are in the application whitelist.

[0305] Exemplarily, taking the case where the only buffer queue with available Buffers currently is this BQ1 as an example.

[0306] In this way, the SF module can determine that the currently enqueued Buffer is produced by Application 1 according to the package name of Application 1 included in the name of the buffer queue BQ1.

[0307] Combined with the foregoing description of the Xsync scheme, the SF module can also determine to trigger the Xsync scheme for the newly enqueued Buffer (such as the Buffer corresponding to the rendering result 102) according to Application 1 being in the application whitelist.

[0308] S1211. The SF module obtains the rendering result 102 in the buffer queue BQ1 corresponding to the identification field 1.

[0309] It can be understood that, based on the description in S1206b, when the SF module determines to execute the XSync scheme on the newly queued Buffer, it has already received the Binder information 3 sent by the rendering module. Thus, the SF module can immediately perform the dequeueing and composition processing on the newly queued Buffer.

[0310] In addition, after the SF module dequeues the rendering result 102, the available Buffer in the buffer queue BQ1 is empty. Correspondingly, the identification field 1 in the BufferTX module is configured to 0.

[0311] S1212. The SF module performs composition processing based on the rendering result 102 to obtain the display image 111.

[0312] Thus, the SF module can trigger the display consumption of the third frame image without waiting for the next Vsync signal (such as signal V35) to arrive. For example, before the signal V35 arrives, the SF module can complete the composition processing of the third frame image to obtain the corresponding display image 111.

[0313] S1213. The SF module transmits the display image 111 to the display module. Among them, the display module may include a display queue, a display screen, etc.

[0314] Combined with Figure 6 , the SF module can transmit the display image 111 corresponding to the third frame image to the display module within the Vsync period from signal V35 to signal V36.

[0315] S1214. The display module stores the display image 111 in the display queue.

[0316] For example, the SF module transmits the display image 111 to the display driver in the display module. In this way, the display driver can store the received display image 111 in the display queue and wait for display.

[0317] It can be understood that, combined with Figure 6 the example in, based on the implementation of S1214, since the time difference between the enqueue time of the rendering result of the third frame image and the enqueue time of the rendering result of the second frame image is greater than or equal to the duration T2, therefore, when the signal V36 arrives, only one Buffer in the display queue is occupied (such as the display image 111 of the third frame image).

[0318] S1215. The display module performs dequeueing of the display image 111 according to the arrival of the signal V36, and displays the third frame image according to the display image 111.

[0319] Exemplarily, when the display drive of the display module receives the signal V36, it can obtain the stored display image 111 from the display queue and control the display screen of the display module to display according to the display image 111. After the display is completed, the occupied Buffer in the display queue becomes empty.

[0320] In this example, the display image 111 can correspond to the third frame image. In this way, the switching display from the second frame to the third frame image can be realized on the display screen.

[0321] It can be understood that through the solution shown in Figure 12 , by means of the Xsync solution, the third frame image can be quickly synthesized and processed. At the same time, since the time duration between the enqueue time of the rendering result of the third frame image and the enqueue time of the rendering result of the second frame image is greater than or equal to the time duration T2, there will be no accumulation of Buffers in the display queue due to too fast synthesis frequency.

[0322] It can be understood that based on this solution, since there is no accumulation of Buffers in the display queue, when Application 1 needs to apply for a Buffer for rendering other frame images subsequently, it does not need to wait for the Buffer to be released from the display queue. Thus, the efficiency of Application 1 issuing rendering instructions for other frame images is improved.

[0323] The above Figure 12 provides an example of the rendering, synthesis, and display process of the third frame image. The following takes Figure 13 as an example to illustrate the processing process of the fourth frame image. Among them, it is assumed that the time duration between the enqueue time of the rendering of the fourth frame image and the enqueue time of the rendering of the third frame image is less than the time duration T2. As shown in Figure 13 , this solution may include:

[0324] S1301. Application 1 generates a rendering instruction 131.

[0325] Exemplarily, Application 1 can generate a rendering instruction 131 after the signal V35 arrives to instruct the electronic device to render and perform subsequent processing on the fourth frame image.

[0326] In some embodiments, as shown in Figure 13 , the touchpad can receive the user input operation OP2 before the signal V35 arrives. Correspondingly, the touchpad can generate operation information 92 corresponding to the operation OP2. This process can refer to S1201 - S1202 in Figure 12 . In this way, the rendering instruction 131 generated by Application 1 can include a rendering instruction for instructing to perform the content corresponding to the operation information 92.

[0327] S1302. Application 1 sends rendering instruction 131 to the rendering module.

[0328] S1303. The rendering module performs rendering processing and obtains rendering result 132.

[0329] In this example, the rendering module can complete this rendering processing and obtain rendering result 132 within the Vsync period from signal V35 to signal V36.

[0330] S1304a. The rendering module enqueues rendering result 132 into buffer queue BQ1.

[0331] Thus, the production enqueue of the Buffer corresponding to rendering result 132 into buffer queue BQ1 can be achieved.

[0332] Combined with Figure 12 the description in, when the rendering module enqueues a new Buffer, it can determine whether to immediately send Binder information to the SF module. Taking the Binder information for the 4th frame image as Binder information 4 as an example.

[0333] In this example, the rendering module can determine that the time difference between the enqueue time of the rendering result 132 corresponding to the 4th frame image and the enqueue time of the rendering result 102 corresponding to the 3rd frame image is less than duration T2. This correspondingly indicates that if the synthesis of the 4th frame image and the enqueue into the display queue are immediately executed, it may cause the accumulation of Buffers in the display queue.

[0334] Thus, in this example, the rendering module can wait for a period of time and then send Binder information 4 to the SF module.

[0335] For example, the rendering module can wait until the moment when the duration T2 has passed after the enqueue time of rendering result 102, and then send Binder information 4. As Figure 13 shown, in this example, the rendering module can send Binder information 4 after a period of time after signal V37 arrives.

[0336] In addition, for components such as the SF module, since a new Buffer has been enqueued, it is necessary to start determining whether to execute the Xsync scheme for this Buffer.

[0337] S1305. The BufferTX module determines that a new Buffer has been enqueued.

[0338] S1306. The BufferTX module configures identification field 1 to 1.

[0339] S1307. The SF module determines that identification field 1 is configured to 1.

[0340] S1308. The SF module determines that all applications corresponding to the available Buffers are in the application whitelist.

[0341] The execution of S1305 - S1308 can refer to S1207 - S1210 as in Figure 12 .

[0342] In this way, the SF module can determine the Xsync scheme for the newly queued Buffer (such as the Buffer storing the rendering result 132 of the 4th frame image).

[0343] Since the Binder information 4 has not arrived yet, the SF module can wait to receive the Binder information 4 and then start the dequeueing and composition processing of the newly queued Buffer.

[0344] S1304b. The rendering module sends the Binder information 4 to the SF module.

[0345] Exemplarily, the rendering module waits until the time when the duration T2 has passed after the enqueueing time of the rendering result 102, and then sends the Binder information 4.

[0346] S1309. The SF module obtains the rendering result 132 in the buffer queue BQ1 corresponding to the identification field 1.

[0347] S1310. The SF module performs composition processing based on the rendering result 132 to obtain the display image 133.

[0348] S1311. The SF module transmits the display image 133 to the display module.

[0349] S1312. The display module stores the display image 133 in the display queue.

[0350] The execution of S1309 - S1312 can refer to S1211 - S1214 as shown in Figure 12 .

[0351] It can be understood that after S1304b, the SF module can already determine to trigger the Xsync scheme and receive the Binder information 4. In this way, the SF module can start the dequeueing and composition processing corresponding to the 4th frame image.

[0352] In this example, the electronic device can complete the enqueueing of the display image 133 of the 4th frame image into the display queue within the Vsync period from signal V36 to signal V37.

[0353] Combined with Figure 12According to the description in , within the Vsync period from signal V36 to signal V37, the display module can complete the dequeue display of the display image 111 that has already been in the display queue. Thus, when signal V37 arrives, there will only be one occupied Buffer in the display queue, and the 4th frame image corresponding to the display image 133 can be stored in this Buffer.

[0354] S1313. The display module executes the dequeue of the display image 133 according to the arrival of signal V37, and displays the 3rd frame image according to the display image 133.

[0355] Thus, the electronic device can complete the logic of rendering, synthesizing, and displaying the 4th frame image.

[0356] It can be understood that in the processing logic of the 4th frame image, since the time difference between the rendering enqueue time of the 4th frame image and the rendering enqueue time of the 3rd frame image is relatively short, the drawing module can control the sending timing of the Binder information (such as delaying based on duration T2), so that the 4th frame image does not enter the display queue too early. Thus, while not affecting the display of the 4th frame image when signal V37 arrives, it is ensured that there is only one Buffer waiting for display in the display queue. This can also reserve sufficient available Buffers for subsequent Buffer applications of Application 1.

[0357] Thus, through the implementation of the above solution, when the Binder control solution takes effect, it can be ensured that the time difference between the generation and sending moment (such as the first moment) of the Binder information of the 4th frame image and the generation and sending moment (such as the second moment) of the Binder information of the 3rd frame image is at least the preset duration (such as duration T2). For example, the rendering result of the 4th frame image is obtained at the third moment. Then, if the first duration between the third moment and the second moment is greater than T2, the third moment is the first moment. The drawing module can generate and send Binder information 3 at the third moment (i.e., the moment when the rendering result of the 4th frame image is obtained). In contrast, when the first duration is less than T2, the drawing module can wait until the first moment arrives, that is, after a duration of T2 after the second moment, and then generate and send Binder information 3. This Binder information 3 can also be called the first binding information. The 4th frame image can also be called the Nth frame image. Correspondingly, the 3rd frame image can also be called the (N - 1)th frame image.

[0358] In other embodiments of the present application, the 4th frame image can also be any other frame image indicated by Application 1 to be processed, such as the Mth frame image.

[0359] In the above embodiments, the drawing module may perform corresponding judgments before sending the Binder information of each frame of image. For example, the drawing module may determine whether the time when the rendering result of the current frame of image is queued is greater than a preset duration T2 compared to the time when the rendering result of the previous frame of image is queued. In some other embodiments of the present application, the drawing module may be configured to trigger the above judgment mechanism at regular intervals. Exemplarily, the periodic judgment mechanism may be implemented based on a first timer configured in the electronic device that can perform cyclic timing. Taking the timing duration (such as the second duration) of the first timer as 30s as an example. In this way, the first timer can perform 30s of timing in a loop.

[0360] The drawing module may be configured to trigger a judgment each time the first timer finishes timing, that is, every 30 seconds. In the case where the judgment result is that the time when the rendering result of the current frame of image is queued is greater than or equal to the preset duration T2 compared to the time when the rendering result of the previous frame of image is queued, the Binder information is immediately sent according to the scheme as Figure 12 shown. Correspondingly, in the case where the judgment result is that the time when the rendering result of the current frame of image is queued is less than the preset duration T2 compared to the time when the rendering result of the previous frame of image is queued, the Binder information is delayed and sent according to the scheme as Figure 13 shown, waiting according to the duration T2.

[0361] In addition, in the above Figure 12 and Figure 13 examples, the Xsync scheme for triggering the new Buffer produced by the corresponding application 1 is taken as an example for illustration. In some other embodiments, before the new Buffer produced by this application 1 enters the buffer queue BQ1, there are already available Buffers in the buffer queues of other applications.

[0362] Exemplarily, taking the case where before Buffer1 produced by application 1 enters the buffer queue BQ1, there is an available Buffer2 in the buffer queue BQ2 corresponding to application 2 as an example.

[0363] The SF module may determine whether all applications corresponding to the available Buffers are in the application whitelist based on Buffer1 entering the buffer queue BQ1.

[0364] For example, the SF module may determine that application 1 corresponding to the buffer queue BQ1 is in the application whitelist. The SF module may determine that application 2 corresponding to the buffer queue BQ2 is not in the application whitelist.

[0365] In this way, the SF module may not trigger the Xsync scheme for Buffer1. Wait for the next Vsync signal to arrive and obtain all available Buffers for synthesis processing.

[0366] Of course, if Application 1 is not included in the application whitelist, then as Figure 12 or Figure 13 shown in the solution, the SF module can wait for the next Vsync signal to arrive before performing the synthesis process on the newly queued Buffer.

[0367] Thus, in each of the above embodiments, through the cooperation of the Xsync solution and the Binder control solution, the qualified Buffers can be quickly synthesized and processed and displayed on the display screen, while avoiding the accumulation of a large number of Buffers in the display queue. This can improve the efficiency of the application program's application for Buffers, and thus have a significant positive effect on the rendering delay of the frame image.

[0368] Through experimental verification, Table 1 below provides the comparison of the rendering delays before and after the Binder control solution provided in the embodiments of the present application takes effect. Among them, taking the frame rate of Application 1 as 120Hz as an example.

[0369] Table 1

[0370] Rendering delay Before the proposed solution of this application becomes effective 7ms After the proposed solution of this application becomes effective 4.8ms

[0371] As shown in Table 1, during the processing of multiple consecutive frames of images. Before the Binder control solution takes effect, due to the accumulation of a large number of available Buffers in the display queue, the application program cannot apply for available Buffers in a timely manner. This will affect the rendering process of subsequent frame images. For example, the rendering delay is as long as 7ms. Correspondingly, after the Binder control solution takes effect, due to the reasonable control of the synthesis frequency of the previous frame images, the large accumulation of Buffers in the display queue is avoided. This significantly improves the efficiency of the application program's application for and acquisition of Buffers, and thus can optimize the rendering delay of the subsequent images. For example, after the Binder control solution takes effect, the rendering delay can be shortened to 4.8ms.

[0372] It can be understood that in order to implement the above functions, the electronic device provided in the embodiments of the present application includes the corresponding hardware structures and / or software modules for performing each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed in this article, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0373] The above mainly introduces the solution provided by the embodiments of the present application from the perspective of each functional module. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described function for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0374] Exemplarily, Figure 14 shows a schematic diagram of the composition of an electronic device 1400. As Figure 14 shown, the electronic device 1400 may include: a processor 1401 and a memory 1402. In this example, the electronic device 1400 may further include a display screen 1403. The memory 1402 is used to store computer execution instructions. Exemplarily, in some embodiments, when the processor 1401 executes the instructions stored in the memory 1402, the electronic device 1400 may be caused to execute the method shown in any one of the above embodiments, whereby relevant images can be displayed faster on the display screen 1403.

[0375] It should be noted that all relevant contents of the steps involved in the above method embodiments can be cited in the function descriptions of the corresponding functional modules, and will not be repeated here.

[0376] Figure 15 shows a schematic diagram of the composition of a chip system 1500. The chip system 1500 may include: a processor 1501 and a communication interface 1502, which are used to support relevant devices to implement the functions involved in the above embodiments. In a possible design, the chip system further includes a memory for storing necessary program instructions and data of the electronic device. The chip system may be composed of chips or may include chips and other discrete devices. It should be noted that in some implementation manners of the present application, the communication interface 1502 may also be referred to as an interface circuit.

[0377] It should be noted that all relevant contents of the steps involved in the above method embodiments can be cited in the function descriptions of the corresponding functional modules, and will not be repeated here.

[0378] The functions, actions, operations, steps, etc. in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more integrated media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0379] Although the present application has been described in connection with specific features and their embodiments, it is obvious that various modifications and combinations can be made without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely exemplary illustrations of the present application defined by the appended claims and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the present application. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these changes and modifications.

Claims

1. An image processing method, characterized in that, The method is applied to an electronic device, which is configured with a display screen and also installs a first application; The method includes: After generating a first synchronous Vsync signal, receiving a first rendering instruction sent by the first application, where the first rendering instruction is used to instruct the electronic device to perform rendering processing on the Nth frame of image; Obtaining a first rendering result according to the first rendering instruction; Generating first binding information; Performing synthesis processing according to the first rendering result to obtain a first image for display; the first image for display corresponds to the Nth frame of image; After generating a second Vsync signal, controlling the display screen to display the Nth frame of image according to the first image for display; Wherein, the second Vsync signal is generated after the first Vsync signal; and there is no generated Vsync signal after obtaining the first rendering result and before performing synthesis processing according to the first rendering result; The time duration between a first moment and a second moment is greater than or equal to a preset time duration; the first moment is the moment when the first binding information is generated, the second moment is the moment when second binding information is generated, the second binding information is generated after obtaining a second rendering result, and the second rendering result is the rendering result of the (N - 1)th frame of image.

2. The method according to claim 1, wherein Before generating the first binding information, the method further includes: Determining a first time duration between a third moment and the second moment; the third moment is the moment when the first rendering result is obtained; When the first time duration is less than the preset time duration, the first moment is later than the third moment; The generating of the first binding information includes: At the first moment, generating the first binding information.

3. The method according to claim 2, wherein When the first time duration is greater than or equal to the preset time duration, the first moment is the same as the third moment; The generating of the first binding information includes: At the third moment, generating the first binding information.

4. The method according to claim 2 or 3, characterized in that, The method further includes: Determining the first time duration.

5. The method according to claim 4, wherein The electronic device is configured with a first timer, and the first timer is used for timing a second time duration; Before determining the first time duration, it includes: At the end of the timing of the first timer, determining the first time duration.

6. The method according to any one of claims 1-5, characterized in that, The electronic device is configured with a first buffer queue, and the first buffer queue corresponds to the first application; After obtaining the first rendering result according to the first rendering instruction, the method further includes: Sending first information corresponding to the first rendering result to the first buffer queue; The first information includes any one of the following: Image information of the first rendering result; an identifier of a buffer Buffer storing the first rendering result; a first file identifier, where the first file identifier indicates a storage location of the first rendering result in the storage of the electronic device.

7. The method according to claim 6, wherein After sending the first information corresponding to the first rendering result to the first buffer queue, the method further includes: Configure the first identification field corresponding to the first buffer queue to a second value; the first identification field being the second value indicates that new information has been queued in the first buffer queue.

8. The method according to claim 7, wherein Before sending the first information corresponding to the first rendering result to the first buffer queue, configure the first identification field to a first value; the first identification field being the first value indicates that the available information in the first buffer queue is empty.

9. The method according to claim 5 or 6, characterized in that, The synthesizing process according to the first rendering result to obtain a first display image includes: Obtain the first information from the first buffer queue, Perform a synthesizing process on the first rendering result indicated by the first information to obtain the first display image.

10. The method according to claim 9, characterized in that, The electronic device is configured with an application whitelist, and the application whitelist includes at least one application information, and the application information of different applications is different; Before obtaining the first information from the first buffer queue, the method further includes: Determine that the information of the first buffer queue is included in the application whitelist.

11. The method according to claim 10, wherein, The application information includes the package name of the application; The information of the first buffer queue includes: the package name of the first application corresponding to the first buffer queue; The determining that the information of the first buffer queue is included in the application whitelist includes: Determine that the package name of the first application corresponding to the first buffer queue is included in the application whitelist.

12. The method according to claim 11, wherein The method further includes: Determine the package name of the first application corresponding to the first buffer queue according to the name of the first buffer queue.

13. The method according to any one of claims 10 - 11, characterized in that, A second application is further installed in the electronic device, and the second application corresponds to a second buffer queue; the second buffer queue includes second information, and the second information is information corresponding to a third rendering result, and the third rendering result is obtained by the electronic device under the instruction of the second application; Before obtaining the first information from the first buffer queue, the method further includes: Determine that the information of the second buffer queue is included in the application whitelist.

14. The method according to claim 13, wherein Before the synthesizing process according to the first rendering result to obtain a first display image, the method further includes: Obtain the second information from the second buffer queue; Obtain the third rendering result according to the second information; The synthesizing process according to the first rendering result to obtain a first display image includes: Perform a synthesizing process according to the first rendering result and the third rendering result to obtain the first display image.

15. The method according to claim 13 or 14, characterized in that, In the case where the information of the second buffer queue is not included in the application whitelist, the method further includes: After generating the second Vsync signal, obtain the first rendering result according to the first information and obtain the third rendering result according to the second information; Perform a synthesizing process according to the first rendering result and the third rendering result to obtain a first display image; After generating the second Vsync signal, control the display screen to display the Nth frame image according to the first display image, including: After generating the third Vsync signal, control the display screen to display the Nth frame image according to the first image to be sent for display, where the third Vsync signal is the Vsync signal after the second Vsync signal.

16. An electronic device, characterized in that, The electronic device includes: a memory and one or more processors; the memory is coupled to the processor; wherein, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor executes the computer instructions, the electronic device executes the method described in any one of claims 1-15.

17. A chip system, characterized in that, The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected by lines; the interface circuits are used to receive signals from the memory of the electronic device and send the signals to the processors, and the signals include the computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the method described in any one of claims 1-15.

18. A computer-readable storage medium, characterized in that, Includes computer instructions, and when the computer instructions run on an electronic device, the electronic device is caused to execute the method described in any one of claims 1-15.

Images