Frequency modulation method of processor, electronic equipment, chip system and storage medium

Abstract

The invention discloses a frequency modulation method of a processor, electronic equipment, a chip system and a storage medium, and belongs to the technical field of terminals. The method comprises the following steps: in response to a first operation, starting a first application and displaying an application picture of the first application; in the running process of the first application, the residence time length of the latest synthesized image frame of the first application is obtained, and the residence time length refers to the time length from the time when the image frame is stored in the graphic buffer area to the time when the image frame is taken out of the graphic buffer area to be synthesized after rendering is completed; and according to a comparison result of the residence duration and the target duration, adjusting the frequency of a target processor running the rendering thread of the first application. According to the invention, frequency modulation is carried out on the processor based on the frame buffer residence duration, so that the timeliness of frequency modulation can be improved, and a random frame loss phenomenon is reduced or avoided.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a frequency modulation method of a processor, an electronic device, a chip system and a storage medium. Background Art

[0002] Playing games is an important scenario in the use of mobile phones, tablets and other terminals, and the gaming experience is also an important indicator for evaluating the quality of terminals. The frame rate and jitter rate of the game are important indicators that affect the user's gaming experience. Poor frame rate or high jitter rate will lead to a poor gaming experience. At this time, the terminal equipment manufacturers need to do platform optimization.

[0003] The direct cause of poor frame rate and high jitter rate is random frame loss. At present, terminals generally use load-based resource scheduling algorithms to improve the phenomenon of random frame loss. Specifically, the load information in the rendering process of the game screen is collected in real time, and the load changes in the rendering process are statistically analyzed based on the load information of multiple time windows, and reasonable processor resources are allocated to the rendering process based on the load changes. For example, if the load change indicates that the load in multiple consecutive time windows is greater than the load threshold, the processor where the rendering thread is located is frequency-increased so that the processor uses a higher frequency to run the rendering thread of the game screen, thereby reducing the probability of subsequent frame loss.

[0004] However, the load-based resource scheduling algorithm needs to accumulate statistics on load changes over a certain period of time before it can perform frequency modulation. Therefore, it may not be able to perceive load changes in a timely manner and accurately predict the timing of frame drops, and thus cannot allocate the necessary resources in a timely manner to cooperate with the rendering process, resulting in the processor frequency not being able to be increased in time when frames are about to be dropped, and the improvement effect of random frame drops is not good. Summary of the invention

[0005] The present application provides a frequency modulation method of a processor, an electronic device, a chip system and a storage medium, which can improve the timeliness of frequency modulation and reduce or avoid random frame loss. The technical solution is as follows:

[0006] In a first aspect, a processor frequency modulation method is provided, which is applied to an electronic device, and the method includes: in response to a first operation, starting a first application and displaying an application screen of the first application; during the running of the first application, obtaining the residence time of the latest synthesized image frame of the first application; and adjusting the frequency of a target processor of a rendering thread running the first application based on a comparison result of the residence time and a target time.

[0007] Among them, the residence time refers to the cache time of the image frame in the graphics buffer, that is, the time from the image frame being stored in the graphics buffer to being taken out of the graphics buffer for synthesis after rendering is completed. The target duration refers to the stable duration corresponding to the residence time, that is, when the frame rate is stable, the residence time of each frame is generally maintained at around the target duration. Among them, the state of stable frame rate refers to the state where the application has no frame loss and the jitter rate of the frame rate is less than the preset jitter rate. The preset jitter rate can be set as needed, for example, it can be the maximum jitter rate that does not affect the display effect.

[0008] When the frame rate is stable, the dwell time of each image frame should be maintained at around the target time. By comparing the dwell time with the target time, the changes in the workload of the image producer and the changes in the production speed can be perceived in a timely manner, so that possible frame losses can be predicted in a timely and accurate manner. Therefore, based on the comparison results of the dwell time and the target time, the processor of the rendering thread running the application is frequency-adjusted, and frequency adjustment can be performed in a timely manner when frame losses may occur, thereby reducing or avoiding the random frame losses of the application, thereby reducing or avoiding the freeze phenomenon of the application screen, and improving the user's application experience.

[0009] In one embodiment, the dwell time of multiple historical frames rendered by the application under a stable frame rate can be obtained, and the target duration can be determined according to the dwell time of the multiple historical frames. For example, the average dwell time of the multiple historical frames is determined as the target duration. In this way, a more accurate target duration can be obtained, which improves the accuracy of subsequent frequency modulation based on the dwell time.

[0010] In one embodiment, the target duration T matches the frame rate of the application. For example, in order to ensure the accuracy of frequency modulation, different target durations T can be set in different frame rate scenarios. For example, when the frame rate of the application is 60fps, the target duration T = 8.3ms. When the frame rate of the application is 90fps, the target duration T = 5.5ms.

[0011] In a possible implementation, a correspondence between a frame rate and a target duration may be preset, and the correspondence may include multiple frame rates and target durations corresponding to different frame rates. When setting the target duration, the target duration corresponding to the frame rate of the current application may be determined from the correspondence. The target duration corresponding to each frame rate may be determined based on the dwell time of multiple historical frames rendered by the application at the corresponding frame rate and in a stable frame rate state, for example, based on the average dwell time of multiple historical frames.

[0012] In one embodiment, when the residence time of the latest synthesized image frame is less than the target time, the frequency of the processor where the rendering thread of the first application is located is increased, that is, the frequency of the processor where the rendering thread of the first application is located is increased, so that the processing performance of the processor can be timely improved, thereby increasing the production speed of the image producer and preventing frame loss. Conversely, when the residence time of the latest synthesized image frame is greater than the target time, the frequency of the processor where the rendering thread of the first application is reduced, that is, the frequency of the processor where the rendering thread is located is reduced, so as to reduce the production speed of the image producer and save power consumption.

[0013] In one embodiment, in order to avoid frequent frequency modulation of the processor and save power consumption, a duration difference tolerance can be pre-set. Before frequency modulation, the duration difference between the residence time and the target time is compared with the duration difference tolerance. If the duration difference between the two is less than or equal to the duration difference tolerance, the frequency of the processor where the rendering thread of the first application is located is not modulated. If the duration difference between the two is greater than the duration difference tolerance, the residence time and the target time are compared. If the residence time of the latest synthesized image frame is less than the target time, the frequency of the processor where the rendering thread of the first application is located is increased; if the residence time of the latest synthesized image frame is greater than the target time, the frequency of the processor where the rendering thread of the first application is located is reduced.

[0014] In one embodiment, considering the high overhead caused by frequent frequency modulation, in order to save power consumption and avoid severe frequency jitter (sudden rise and fall), the frequency of the processor where the rendering thread of the first application is located can be gradually adjusted when the frequency of the processor where the rendering thread of the first application is located is modulated. In this way, the problem that the frequency point is raised to the highest frequency point when the frequency is increased in the load-based resource scheduling algorithm, which is not friendly to power consumption, can be improved.

[0015] As an example, when increasing the frequency, the frequency value of the target processor can be first increased by a first frequency value, and then the residence time of the next image frame synthesized by the first application can be obtained. If the residence time of the next image frame is less than the target time, the frequency value of the target processor can be increased by the first frequency value until the residence time of the latest image frame synthesized by the first application is greater than or equal to the target time.

[0016] As an example, when frequency reduction is performed, the frequency value of the target processor can be first reduced by a second frequency value, and then the residence time of the next image frame synthesized by the first application can be obtained. If the residence time of the next image frame is greater than the target time, the frequency value of the target processor can be further reduced by the second frequency value until the residence time of the latest image frame synthesized by the first application is less than or equal to the target time.

[0017] In one embodiment, the rendering process of the rendering thread of the first application can be monitored to obtain the time when the rendering thread stores the image frame in the graphics buffer after rendering the image frame, and obtain the buffer storage time of the image frame; the synthesis process of the image consumer corresponding to the rendering process can be monitored to obtain the time when the image consumer takes out the image frame from the graphics buffer for synthesis, and obtain the buffer retrieval time of the image frame; according to the buffer storage time and buffer retrieval time of the image frame, the residence time of the image frame is determined.

[0018] By monitoring the rendering process of the rendering thread of the first application and the synthesis process of the image consumer (such as SurfaceFlinger), the buffer storage time and buffer retrieval time of the latest synthesized image frame can be monitored in time, and the residence time of the image frame can be determined in time according to the buffer storage time and buffer retrieval time of the image frame.

[0019] In one embodiment, the first application may be a game application, a video application, a video call application, etc., which is not limited in this embodiment of the present application.

[0020] In a second aspect, a frequency modulation device of a processor is provided, wherein the frequency modulation device of the processor has the function of implementing the frequency modulation method of the processor in the first aspect. The frequency modulation device of the processor includes at least one module, and the at least one module is used to implement the frequency modulation method of the processor provided in the first aspect.

[0021] In a third aspect, a frequency modulation device for a processor is provided, wherein the structure of the frequency modulation device for the processor includes a processor and a memory, wherein the memory is used to store a program that supports the frequency modulation device for the processor to execute the frequency modulation method for the processor provided in the first aspect, and to store data involved in implementing the frequency modulation method for the processor described in the first aspect. The processor is configured to execute the program stored in the memory. The frequency modulation device for the processor may also include a communication bus, wherein the communication bus is used to establish a connection between the processor and the memory.

[0022] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the frequency modulation method of the processor described in the first aspect.

[0023] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the processor frequency modulation method described in the first aspect.

[0024] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a system schematic diagram of an image rendering system provided by an embodiment of the present application;

[0026] Figure 2 It is a producer-consumer framework diagram of a buffer queue;

[0027] Figure 3 is a schematic diagram of a multi-core processor architecture of an electronic device provided in an embodiment of the present application;

[0028] Figure 4 is a schematic diagram of a game screen of a game application provided in an embodiment of the present application;

[0029] Figure 5 It is a trace diagram of a rendering process of an application provided in an embodiment of the present application;

[0030] Figure 6 is a schematic diagram of the dwell time of an image frame provided in an embodiment of the present application;

[0031] Figure 7 Another trace diagram of the synthesis process of SurfaceFlinger provided in an embodiment of the present application;

[0032] Figure 8 It is a flow chart of a method for frequency modulation of a processor based on frame dwell time provided in an embodiment of the present application;

[0033] Fig. 9 is a block diagram of a software system of an electronic device provided in an embodiment of the present application;

[0034] Fig.10 is a schematic flow chart of a frequency modulation method of a processor provided in an embodiment of the present application;

[0035] Fig.11 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In the following description, specific details such as specific system structures and technologies are provided for illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0037] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0038] It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more than two; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0039] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", "fourth", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0040] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0041] To facilitate understanding, the nouns involved in the embodiments of the present application are first explained.

[0042] Frame rate: refers to the number of frames generated per second, that is, the frequency at which bitmap images in frames appear continuously on the display. The unit of frame rate is fps (frame per second). Frame rate represents the smoothness of the picture. The higher the frame rate, the better the smoothness of the picture.

[0043] Due to the persistence of vision of the human eye, when the frame rate is high enough (usually around 50fps to 60fps), the picture will look continuous and smooth. At present, in order to improve the gaming experience, the frame rate of game applications is generally 60fps, 90fps or 120fps.

[0044] Jitter rate: The rate of change of the frame rate, generally calculated based on the frame rate difference between two adjacent frames. The higher the jitter rate, the greater the fluctuation of the frame rate, the worse the frame rate stability, and the worse the picture smoothness.

[0045] Frame loss: Also known as frame drop, it usually refers to the situation where some frames are not displayed or processed correctly in any application that requires a continuous frame rate, such as video playback, animation rendering, or games. Frame loss will cause the picture to be jittery and incoherent.

[0046] Graphics buffer: It is a container for storing rendered frame data to be synthesized.

[0047] Dwell time: The length of time (dwell time) that the rendered frame data is cached in the graphics buffer. That is, the length of time from when the image producer completes the production of a frame of data to when the consumer takes the frame of data out of the graphics buffer for consumption.

[0048] To facilitate the understanding of terms such as frame buffer and dwell time, the overall architecture of the image rendering system is explained below.

[0049] Please refer to Figure 1 , Figure 1 : is a system diagram of an image rendering system provided by an embodiment of the present application. The image rendering system mainly adopts a producer-consumer model, and the core of screen rendering is the production and consumption of image data. Figure 1 As shown in the figure, the image rendering system mainly includes: image producer, graphics buffer and image consumer. The image data generated by the image producer is first stored in the graphics buffer, and then taken out and consumed by the image consumer. This is a typical producer-consumer model.

[0050] Image producer: used to generate image data. Image producers mainly include MediaPlayer, CameraPreview, open graphics library for embedded systems (OpenGL ES) or native developer kits (NDK). Among them, MediaPlayer and Camera Preview generate image data by directly reading the image source. NDK (such as Skia) and OpenGL generate image data through their own drawing capabilities. In addition, for applications (APP), applications (hereinafter referred to as applications) can generate image data through rendering threads calling OpenGL, etc., so applications or rendering threads of applications can also be called image producers.

[0051] Graphics buffer: used to store image data generated by image producers, mainly to assist in the transmission of image data between image producers and image consumers. That is, after the image producer generates image data, the image data is first stored in the graphics buffer, and then the image consumer takes it out of the graphics buffer and consumes it.

[0052] Image consumer: used to fetch image data from the graphics buffer and display the image data to display the image data on the screen. Among them, the most common image consumer is SurfaceFlinger. After SurfaceFlinger fetches image data from the graphics buffer, it can use OpenGL and / or hardware composer (HWC) to synthesize the fetched image data, and then send the synthesized image data for display. That is, the consumption process of SurfaceFlinger includes synthesis and display.

[0053] The graphics buffer is managed by the buffer queue, and the core logic of the buffer queue is the producer-consumer logic. Figure 2 , Figure 2 It is a producer-consumer framework diagram of a buffer queue. Figure 2 As shown, the working process of the buffer queue includes:

[0054] 1) The image producer requests a piece of graphics buffer content, that is, a graphic buffer (also referred to as buffer for short), from the buffer queue by calling the dequeue method.

[0055] 2) The image producer writes the generated image data into the graphic buffer and submits it to the buffer queue through the queue method.

[0056] 3) The buffer queue receives the enqueue message from the graphic buffer and notifies the image consumer to call the acquire method to obtain the graphic buffer filled with data for processing or display.

[0057] 4) After the image consumer has finished processing, it calls the release method to return the graphic buffer to the bufferqueue, and this graphic buffer can be reused later.

[0058] 5) The image producer, buffer queue, and image consumer work in coordination, and the graphic buffer coordinates the flow between the three, so that the image is displayed or processed smoothly.

[0059] In the rendering process of the APP, the rendering thread of the APP is the image producer and SurfaceFlinger is the image consumer. In one embodiment, when the rendering thread of the APP needs a graphic buffer, it requests a graphic buffer from the buffer queue by calling the dequeue buffer method and specifying the width, height, pixel format and usage flag of the buffer. The rendering thread of the APP performs rendering, writes the generated image data to the graphic buffer after the rendering is completed, and calls the queue buffer method to return the graphic buffer to the buffer queue. After receiving the Vsync signal, SurfaceFlinge starts to prepare for synthesis, calls the acquire buffer method to obtain the graphic buffer from the buffer queue, and synthesizes the image data in the graphic buffer. After the synthesis is completed, SurfaceFlinger returns the graphic buffer to the buffer queue by calling the release buffer method.

[0060] It should be understood that the buffer queue can adopt a dual-buffer architecture or a 3-buffer architecture. Of course, with the advancement of technology, a 4-buffer architecture or a 5-buffer architecture can also be adopted. The embodiment of the present application does not limit the buffer architecture adopted by the buffer queue. With a 3-buffer architecture, the 3-buffer architecture of the buffer queue includes three graphics buffers, namely a foreground buffer, a background buffer, and an idle buffer. The foreground buffer is the buffer currently being consumed by the image consumer; the background buffer is the buffer that the image producer is filling with the next frame of content; when the image producer is preparing content in the background buffer, the idle buffer is idle. Once the background buffer is consumed by the image consumer, the idle buffer will become a new background buffer, waiting for the image producer to fill the next frame of content. Through this 3-buffer architecture, the buffer queue can ensure that there is always a buffer ready during the production and consumption process, thereby avoiding display delays caused by waiting for new content. At the same time, it can also effectively utilize system resources and improve overall performance.

[0061] Frequency modulation: Frequency modulation refers to adjusting the frequency of the central processing unit (CPU), such as increasing the frequency of the CPU (increasing the frequency) or decreasing the frequency of the CPU (decreasing the frequency). In the embodiment of the present application, the central processing unit (hereinafter referred to as the processor) where the rendering thread of the application is located can be frequency modulated to reasonably use processor resources, improve rendering performance, and reduce or avoid frame loss.

[0062] Frequency point: A frequency point is a number given to a fixed frequency. Each frequency point can correspond to a fixed frequency band. The larger the number, the larger the upper limit of the corresponding frequency band. For example, the frequency interval between adjacent frequency points is 200KHz.

[0063] The following is an example of the processor architecture involved in the frequency modulation scenario.

[0064] The processor is the computing and control core of electronic devices, mainly used to realize information processing and program running. In order to improve the processing speed and performance of the processor, multi-core processors are generally used in electronic devices. Processors can be divided into large cores, medium cores and small cores. Compared with the medium core, the large core has higher computing power, but also consumes more power. Similarly, the medium core has higher computing power than the small core, but also consumes more power. For example, a multi-core processor includes a small core, a medium core and a large core. The frequency of the medium core is greater than that of the small core, and the frequency of the large core is greater than that of the medium core.

[0065] As an example, see Figure 3 , Figure 3 Schematic diagram of a multi-core processor architecture of an electronic device provided in an embodiment of the present application. Figure 3 As shown, the electronic device has 12 processors (CPU0-CPU11) with different architectures. These 12 processors include 4 large cores (CPU0-3), 4 medium cores (CPU4-7) and 4 small cores (CPU8-11). The frequencies of large cores, medium cores and small cores increase in sequence. For example, the frequency band of the small core is 600M-1.5G, that is, the minimum frequency is 600M and the maximum frequency is 1.5G; the frequency band of the medium core is 1.5G-3.0G; the frequency band of the large core is 3.0G-5.0G.

[0066] For a multi-core processor, the rendering thread of the application can be run by one or several processors in the multi-core processor, that is, the rendering thread of the application runs on all or part of the processors of the multi-core processor. However, the frequency of the processor where the rendering thread is located may not match the rendering thread, resulting in frame loss. In an embodiment of the present application, the processor where the rendering thread of the application is located can be frequency-adjusted at an appropriate time to reasonably use processor resources, improve rendering performance, and reduce or avoid frame loss.

[0067] The frequency modulation method of the processor provided in the embodiment of the present application can be applied to electronic devices. Electronic devices include electronic devices, which can also be called terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. The electronic device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the electronic device.

[0068] Next, the application scenarios involved in the embodiments of the present application are described.

[0069] Playing games is an important scenario in the use of mobile phones, tablets and other terminals, and the gaming experience is also an important indicator for evaluating the quality of terminals. The frame rate and jitter rate of the game are basic and important evaluation items. With the continuous development of smart terminal devices, users' demand for gaming experience is getting higher and higher. With the continuous development of terminal technology, users' demand for gaming experience is getting higher and higher. At present, there are a variety of game applications on the application market. Some of them are due to complex game scenes, random changes in load, and some are due to the uneven level of developers, resulting in very unsatisfactory performance in frame rate and jitter rate, such as low frame rate and / or high jitter rate, which leads to poor gaming experience. At this time, the terminal equipment manufacturers need to make platform optimizations.

[0070] For example, due to the complexity of the game scene, the amount of content in different game screens varies greatly, resulting in large differences in the load required to render different game screens, which in turn may cause frame drops due to rendering timeouts when rendering game screens with large amounts of content. For example, when the game scene enters a new map, due to the complexity of the scene and the large amount of data in the new map, the load required to render the new map screen is large and the time required is long. Frame drops may occur due to the failure to render the new map in time, resulting in the game interface remaining on the current game screen after the user switches to the new map, and failing to switch to the new map screen in time, causing the screen to freeze.

[0071] Please refer to Figure 4 , Figure 4 Schematic diagram of a game interface of a game application provided in an embodiment of the present application. Figure 4 As shown, the game interface of the game application first displays a game screen 310, which includes a small Fig.10 , the game object operated by the user is currently located at point A in the small map. If the user wants to switch to another location in the small map (i.e. switch to a new map), he can click point C in the small map on the game screen 310. In response to the user's operation, the new map corresponding to point C should be entered immediately, that is, the game screen 310 should be switched to the new map screen corresponding to point C. However, due to the complex scene, more new content and larger data volume of the new map, the load required to render the new map screen is large and the required time is long, resulting in the new map screen not being rendered in time when the screen refresh signal arrives. Therefore, the current game screen (game screen 311) is still displayed when the screen is refreshed, and the rendered new map screen (game screen 312) is displayed only when the next screen refresh signal arrives. For the user, after executing the operation of switching to a new map, the user will see that the game screen is stuck, and it will switch to the new map after the stuck. The user feels that the operation is not smooth and the game experience is not good.

[0072] In order to improve the user's gaming experience, terminal equipment manufacturers need to make platform optimizations. The direct cause of poor frame rate and high jitter rate is random frame loss. At present, terminals generally use load-based resource scheduling algorithms to improve the phenomenon of random frame loss. However, the load-based resource scheduling algorithm is not accurate enough in perceiving and predicting the load changes in the rendering process of the game screen, and thus cannot allocate the necessary resources in time to cooperate with the rendering process, resulting in the processor frequency point not being able to be timely increased when the frame is about to be lost, and the improvement effect of the random frame loss phenomenon is not good.

[0073] Please refer to Figure 5 , Figure 5 is a behavior (trace) diagram of a rendering process of an application provided in an embodiment of the present application, which shows the logs of various key points in the rendering process of application A. Figure 5As shown in the figure, the key tasks ①②③ of the rendering thread of application A run on CPU4, CPU5 and CPU6 respectively, and the maximum frequency of CPU4, CPU5 and CPU6 is 2.5GHz. Assuming that the frame rate of application A is 60fps, the average rendering time interval between two frames should be 16.6ms. Figure 5 As shown in the figure, the rendering completion time of the i-th frame and the i+1-th frame differ by 16.6ms, and after the rendering of the i+1-th frame is completed, the frequencies of CPU4, CPU5 and CPU6 are reduced to 600MHz. In the case of a stable frame rate, the rendering of the i+2-th frame should start at the time point (shown at ④) which is also 16.6ms different from the rendering completion time of the i+1-th frame. However, since the data volume of the i+2-th frame is large and the required load is relatively high, the frequency point at ⑤ when the load starts to run is too low, so the rendering of the i+3-th frame cannot be started in time at ④, resulting in the failure to render the i+3-th frame in time, resulting in the loss of the i+3-th frame. In order to reduce frame loss, a load-based resource scheduling algorithm is generally used at present. Based on this algorithm, it is necessary to continuously collect load information for a period of time (such as the time period from ⑤ to ⑥ in the figure), and to perform statistics on the load changes during this period. If the statistical results indicate that the load during this period continues to increase and is greater than the load threshold, the frequency of CPU4, CPU5 and CPU6 is increased at time point ⑥ to increase the frequency to 2.5GHz. However, this scheduling algorithm does not increase the frequency until after point ④, and does not raise the frequency to 2.5 GHz until point ⑥. However, frame loss has already occurred at this time. Therefore, this scheduling algorithm cannot predict frame loss in time and cannot increase the frequency in time.

[0074] Therefore, the load-based resource scheduling algorithm has the following problems: 1. The frequency is increased too late, and the probability of saving the current frame is low, resulting in poor effect in improving the frame loss phenomenon. 2. When increasing the frequency, the frequency point is generally increased to the highest frequency point, which is not friendly to power consumption.

[0075] In response to the above-mentioned problem 1, that is, the frequency increase timing of the load-based resource scheduling algorithm is too late, and the probability of saving the current frame is not high, resulting in poor effect in improving the frame loss phenomenon, in an embodiment of the present application, the residence time of the latest synthesized image frame (also referred to as frame) of the application can be obtained during the running of the application, and the processor of the rendering thread running the application can be frequency-modulated according to the residence time and the target time. Among them, the frame data of the image frame is generally cached in the graphics buffer after rendering is completed, and then taken out of the graphics buffer for synthesis and display. The residence time refers to the cache time of the frame data of the image frame in the graphics buffer after rendering is completed. The target time is the overall residence time of the image frame when the frame rate of the application is stable, that is, the stable value of the residence time.

[0076] When the frame rate is stable, the dwell time of each image frame should be the target time, or not much different from the target time. By comparing the dwell time with the target time, the changes in the workload of the image producer and the changes in the production speed can be perceived in a timely manner, so that frame loss can be predicted in a timely and accurate manner. Therefore, according to the comparison results of the dwell time and the target time, the processor of the rendering thread running the application is frequency-adjusted, and frequency adjustment can be performed in time when frame loss may occur, thereby increasing the probability of reducing or avoiding frame loss, improving the effect of improving random frame loss, and then reducing or avoiding the freeze phenomenon of the application screen, improving the user's application experience.

[0077] As an example, if the residence time of the latest synthesized image frame is less than the target duration, the frequency of the processor where the rendering thread of the application is located is increased, that is, the frequency (also called the working frequency) of the processor where the rendering thread is located is increased, so that the processing performance of the processor can be improved in time, thereby increasing the production speed of the image producer and preventing frame loss. Conversely, if the residence time of the latest synthesized image frame is greater than the target duration, the frequency of the processor where the rendering thread of the application is located is reduced, that is, the frequency of the processor where the rendering thread is located is reduced to reduce the production speed of the image producer and save power consumption.

[0078] Furthermore, in order to avoid frequent frequency modulation of the processor and save power consumption, a duration difference tolerance can be pre-set. Before frequency modulation, the duration difference between the residence time and the target duration is compared with the duration difference tolerance. If the duration difference between the two is less than or equal to the duration difference tolerance, the frequency of the processor where the rendering thread of the application is located is not modulated. If the duration difference between the two is greater than the duration difference tolerance, the residence time and the target time are compared. If the residence time of the latest synthesized image frame is less than the target time, the frequency of the processor where the rendering thread of the application is located is increased; if the residence time of the latest synthesized image frame is greater than the target time, the frequency of the processor where the rendering thread of the application is located is reduced.

[0079] When the frame rate is stable, the image producer (rendering thread) and the image consumer (SurfaceFlinger) work at a relatively consistent pace. After a relatively stable period of time has passed since the production of a buffer, the buffer is consumed, that is, the residence time of each frame produced by the image producer is also relatively stable. If the residence time of a frame produced by the image producer becomes shorter or longer, it means that the working rhythm of the image producer (rendering thread) and the image consumer (SurfaceFlinger) has changed. Considering that the image consumer (SurfaceFlinger) has a low workload and works under a precise vsync clock, it generally does not affect the changes in the above supply and demand relationship. Therefore, it can be inferred that the main factor causing the above changes is that the workload of the image producer (rendering thread) has changed, resulting in a change in the production speed. For example, if the residence time of the frames produced by the image producer in the graphics buffer becomes shorter, it means that the workload of the image producer has increased and the production speed has slowed down, which has led to the shorter residence time of the produced frames in the graphics buffer. At this time, by increasing the frequency of the processor where the rendering thread of the application is located, the processing performance of the processor can be improved in time, thereby increasing the production speed of the image producer and preventing frame loss. For another example, if the residence time of the frames produced by the image producer in the graphics buffer becomes longer, it means that the workload of the image producer has decreased and the production speed has increased, which has led to the shorter residence time of the produced frames in the graphics buffer. At this time, by reducing the frequency of the processor where the rendering thread of the application is located, the production speed of the image producer can be reduced to save power consumption.

[0080] It should be understood that the processor where the rendering thread of the application is located may include one or more processors, that is, the rendering thread of the game application may be run by one or more processors. Figure 5 As shown, the processors where the rendering thread of the game application is located may include CPU4, CPU5 and CPU6.

[0081] As an example, frequency modulation of a processor running a rendering thread of an application includes: frequency modulation of a processor running a key task of the rendering thread of the application, that is, frequency modulation of a processor where the key task of the rendering thread of the application is located.

[0082] Please refer to Figure 6 , Figure 6 Schematic diagram of the dwell time of an image frame provided by an embodiment of the present application. Figure 6 As shown in the figure, the black rectangle represents the state of a frame of data cached in the graphic buffer after rendering is completed, and its length represents the length of time the frame data stays in the graphic buffer.

[0083] Please refer to Figure 7 , Figure 7 Another trace diagram of the synthesis process of SurfaceFlinger provided in an embodiment of the present application shows the logs of various key points in the image consumption process of SurfaceFlinger. SurfaceFlinger can obtain cached image data from the graphics buffer and consume (synthesize and send to display) the obtained image data. Figure 7 As shown in the figure, the black rectangle represents the state of a frame of data cached in the graphic buffer, and its length represents the length of time the frame data resides in the graphic buffer. The vertically superimposed blocks behind the black rectangle represent the methods called by SurfaceFlinger to read frame data from the graphics buffer and consume the frame data (synthesis and display, etc.). Figure 7 As shown, ⑨ is the frame loss point, and ⑩ is the frequency increase point determined by the load-based resource scheduling algorithm. Obviously, the frequency increase at ⑩ is too late to produce the corresponding frame data at ⑨, that is, it is too late to save the frame loss situation, and the timing of frequency increase is seriously delayed.

[0084] In the embodiment of the present application, the dwell time of each frame produced by the image producer can be monitored, and the frequency can be adjusted according to the change of the dwell time. Figure 7 As shown in the figure, the frame data at ①-⑤ stays in the graphics buffer for about 8ms, which is relatively stable. This shows that at ①-⑤, the work pace of the image producer (rendering thread) and the image consumer (SurfaceFlinger) is relatively consistent. After a buffer is produced, it will be consumed after a relatively stable period of time. The load of the rendering process is relatively uniform, the resource allocation is reasonable, and there is no need for frequency modulation. Figure 7 As shown, compared with the residence time at ①-⑤, the residence time of the frame data at ⑥⑦⑧ is significantly shorter, indicating that the workload of the image producer (rendering thread) has increased, and the production speed has slowed down, resulting in a significantly shorter residence time of the frame data. In an embodiment of the present application, when it is detected that the residence time of the frame data at ⑥ is significantly shortened (for example, the difference between the residence time and the target time is greater than the time difference tolerance and the residence time is less than the target time), the CPU where the rendering thread is located (such as CPU4) can be frequency-increased to prevent frame loss. Figure 7 As shown, compared with the scheme using a load-based resource scheduling algorithm, the scheme of frequency modulation based on the residence time of frame data used in the present application can perform frequency modulation several frame lengths in advance, thereby increasing the production speed of the image producer in advance to recover the unbalanced time and increase the probability of preventing subsequent frame loss.

[0085] The frequency modulation algorithm involved in the embodiment of the present application is described in detail below.

[0086] Please refer to Figure 8 , Figure 8 1 is a flow chart of a method for frequency modulation of a processor based on frame dwell time provided in an embodiment of the present application, wherein the execution subject of the method is an electronic device. Figure 8 As shown, the method includes:

[0087] 1. Set the target duration T.

[0088] The target duration T refers to the stable duration corresponding to the dwell duration, that is, when the frame rate is stable, the dwell duration of each frame is generally maintained at about the target duration T. The frame rate stability refers to a state where the application does not lose frames and the jitter rate of the frame rate is less than the preset jitter rate. The preset jitter rate can be set as needed, for example, it can be the maximum jitter rate that does not affect the display effect.

[0089] As an example, the dwell time of multiple historical frames rendered by the application at a stable frame rate can be obtained, and then the target duration can be determined based on the dwell time of the multiple historical frames. For example, the average dwell time of the multiple historical frames can be determined as the target duration.

[0090] As an example, the target duration T matches the frame rate of the application. In the embodiment of the present application, in order to ensure the accuracy of frequency modulation, different target durations T can be set in different frame rate scenarios. For example, in a scenario where the frame rate of the application is 60fps, the target duration T = 8.3ms. In a scenario where the frame rate of the application is 90fps, the target duration T = 5.5ms.

[0091] In a possible implementation, a correspondence between a frame rate and a target duration can be preset, and the correspondence includes multiple frame rates and target durations corresponding to different frame rates. When setting the target duration T, the target duration corresponding to the frame rate of the current application can be determined from the correspondence. Among them, the target duration corresponding to each frame rate can be determined based on the residence time of multiple historical frames rendered by the application at the corresponding frame rate and in a stable frame rate state, for example, based on the average residence time of multiple historical frames.

[0092] 2. During the application running process, monitor the dwell time t of each frame and use the negative feedback algorithm to adjust the frequency. The frequency adjustment process includes:

[0093] a) For the monitored dwell time t of the current frame, determine whether |tT| is less than the time difference tolerance.

[0094] b) If |tT| is less than the duration difference tolerance, indicating that the residence time of the current frame tends to be stable, the frequency of the processor where the rendering thread of the application is located is kept unchanged, that is, the frequency of the processor where the rendering thread of the application is located is not adjusted.

[0095] c) If |tT| is greater than or equal to the duration difference tolerance, determine whether t is greater than T.

[0096] d) If t is less than T, it means that the production speed of the image producer (rendering thread of the application) is slow, and the frequency of the processor where the rendering thread of the application is located is increased to appropriately increase the resource supply of the image producer.

[0097] After the frequency of the processor where the rendering thread of the application is located is increased, on the one hand, the production speed of the image producer is reduced and the frame residence time t becomes longer. On the other hand, the residence time t of each frame continues to be monitored and the above steps a)-d) are repeated until |tT<the time difference tolerance.

[0098] Furthermore, considering the high overhead caused by frequent frequency modulation, in order to save power consumption and avoid severe frequency jitter (sudden rise and fall), the frequency of the processor where the application's rendering thread is located can be gradually adjusted when the frequency of the processor where the application's rendering thread is located is modulated. In this way, the problem that the frequency point is raised to the highest frequency point when the frequency is increased based on the load resource scheduling algorithm can be improved, which is not friendly to power consumption.

[0099] As an example, when increasing the frequency of the processor where the rendering thread of the application is located, the frequency can be increased gradually. For example, the frequency value of the processor where the rendering thread is located can be increased by a first frequency value first, and then the residence time t of the next frame synthesized by the application is continuously obtained. If the residence time t of the next frame is less than the target time T, the frequency value of the processor where the rendering thread is located can be increased by the first frequency value until the residence time t of the latest synthesized frame of the application satisfies |tT|<time difference tolerance.

[0100] The first frequency value may be pre-set as required, for example, the first frequency value may be 200 KHz or 400 KHz.

[0101] In a possible implementation, the frequency value of the processor where the rendering thread is located is a frequency point. In this case, the frequency point of the processor where the rendering thread is located can be increased by m frequency points first, and then the residence time t of the next frame synthesized by the application is continuously obtained. If the residence time t of the next frame is less than the target time T, the frequency value of the processor where the rendering thread is located is continuously increased by m frequency points until the residence time t of the latest synthesized frame of the application satisfies |tT|<time difference tolerance. Wherein, m is a positive integer.

[0102] e) If t is greater than T, it means that the production speed of the image producer (rendering thread of the application) is faster, then the processor where the rendering thread of the application is located is down-clocked to appropriately reduce the resource supply of the image producer.

[0103] As an example, the frequency of the processor where the rendering thread of the application is located can be gradually reduced. For example, the frequency value of the processor where the rendering thread is located can be reduced by the second frequency value first, and then the residence time t of the next frame synthesized by the application is continuously obtained. If the residence time t of the next frame is greater than the target time T, the frequency value of the processor where the rendering thread is located is continuously reduced by the second frequency value until the residence time t of the latest synthesized frame of the application satisfies |tT|<the time difference tolerance.

[0104] The second frequency value may be pre-set as required, for example, the second frequency value may be 200 KHz or 400 KHz.

[0105] In a possible implementation, the frequency value of the processor where the rendering thread is located is a frequency point. In this case, the frequency point of the processor where the rendering thread is located can be increased by n frequency points first, and then the residence time t of the next frame synthesized by the application is continuously obtained. If the residence time t of the next frame is less than the target time T, the frequency value of the processor where the rendering thread is located is continuously increased by n frequency points until the residence time t of the latest synthesized frame of the application satisfies |tT|<time difference tolerance. Where n is a positive integer.

[0106] f) Repeat the above steps and maintain the rendering system in a state where |tT|< time difference tolerance through positive and negative frequency modulation.

[0107] In the embodiment of the present application, by adjusting the frequency of the processor based on the frame dwell time, the frame dwell time can be adjusted to a stable range. On the one hand, the frame dwell time can be prevented from being too short to cause frame loss, thereby improving the frame loss situation; on the other hand, the frame dwell time can be prevented from being too long to cause buffer accumulation and excessive supply of processor frequency points, thereby saving power consumption.

[0108] The electronic device provided in the embodiments of the present application may be a user equipment (UE), for example, a mobile terminal (such as a user's mobile phone), a tablet computer, a desktop computer, a laptop computer, a handheld computer, a netbook, a personal digital assistant (PDA), and the like.

[0109] In addition, an operating system is running on the above components, such as the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft. Application programs can be installed and run on the operating system.

[0110] The operating system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device.

[0111] Fig. 9 is a block diagram of a software system of an electronic device provided in an embodiment of the present application. Fig. 9 The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer (also called the application layer), the application framework layer, the native service layer, and the kernel layer.

[0112] The application layer can include a series of application packages. Fig. 9 As shown, the application package may include game applications, video applications, video call applications, and other applications that can display video images. Of course, the application package may also include other applications, such as camera, gallery, calendar, call, map, navigation, Bluetooth, music, video, short message, and other applications.

[0113] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 4 As shown, the application framework layer may include a view system. The view system is used to draw the display screen of the application. For example, the rendering thread of the game application can call the view system of the application framework layer, and use the view system to render the image frame to be displayed for the application. After the rendering is completed, the rendered image frame is cached in the graphics buffer.

[0114] In addition, the application framework layer may also include a window manager, a content provider, a phone manager, a resource manager, a notification manager, etc. The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data, and make these data accessible to applications, which may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The phone manager is used to provide communication functions for the electronic device 100, such as the management of call status (including connecting, hanging up, etc.). The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the application to display notification information in the status bar, which can be used to convey notification type messages, and can disappear automatically after a short stay without user interaction. For example, the notification manager can be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as a notification of an application running in the background.

[0115] The local service layer mainly includes some local services and link libraries. The local service layer can provide feature support for the Android system through some C / C++ libraries. In addition, the local service layer also includes the Android Runtime. Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for the scheduling and management of the Android system. The core library contains two parts: one part is the function functions that need to be called by the Java language, and the other part is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection and other functions.

[0116] Please refer to Fig. 9 The local service layer includes SurfaceFlinger and frequency modulation module. The frequency modulation module includes algorithm parameter parsing configuration module, frame buffer production and consumption monitoring module, frame buffer residence time statistics module, frequency adjustment feedback module and CPU frequency adjustment interface.

[0117] SurfaceFlinger is used to consume the image frames rendered by the application's rendering thread through the view system. For example, SurfaceFlinger can take image frames from the graphics buffer, synthesize the taken image frames, and send them to the display. It should be understood that Fig. 9 The image consumer is SurfaceFlinger as an example for illustration. In other embodiments, the image consumer may also be other image production modules, which is not limited in the embodiments of the present application.

[0118] The frame buffer production and consumption monitoring module is used to monitor the time when the view system stores the rendered image frame in the graphics buffer, and monitor the time when SurfaceFlinger takes out the image frame from the graphics buffer. That is, the frame buffer production and consumption monitoring module can monitor the storage time and retrieval time of each frame in the graphics buffer, and send the monitored storage time and retrieval time of each frame in the graphics buffer to the frame buffer residence time statistics module. The duration between the storage time and retrieval time of any image frame in the graphics buffer is the residence time of the image frame.

[0119] The frame buffer residence time statistics module is used to count the residence time of each frame according to the storage time and retrieval time of each frame in the graphics buffer monitored by the frame buffer production and consumption monitoring module.

[0120] The frequency adjustment feedback algorithm module is used to adjust the frequency of the processor where the rendering thread of the application is located according to the statistical residence time and the target time, according to the method provided in the embodiment of the present application. For example, the statistical residence time is compared with the target time, and it is determined whether to perform frequency adjustment, and whether to increase or decrease the frequency according to the comparison result. In addition, the frequency adjustment feedback algorithm module can also call the CPU frequency adjustment interface when it is determined that frequency adjustment is required, and send frequency adjustment instructions to the CPU frequency adjustment driver of the kernel layer through the CPU frequency adjustment interface.

[0121] The CPU frequency adjustment interface is an interface between the frequency adjustment module and the CPU frequency adjustment driver of the kernel layer, and is used to realize data transmission between the frequency adjustment module and the CPU frequency adjustment.

[0122] The algorithm parameter parsing configuration module is used to configure related parameters for other modules in the frequency modulation module, such as configuring the target duration and frequency adjustment algorithm for the frequency adjustment algorithm module.

[0123] The kernel layer is the layer between hardware and software. The kernel layer includes at least the CPU adjustment driver and GPU driver. The CPU adjustment driver is used to drive the CPU to adjust the frequency.

[0124] It should be noted that Fig. 9 The layers in the software structure shown and the components included in each layer do not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more layers than shown in the figure, such as a system library (FWK LIB) layer and a kernel layer. In addition, each layer may include more or fewer components than shown in the figure, which is not limited in this application.

[0125] It is understandable that in order to implement the frequency modulation method of the processor in the embodiment of the present application, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments.

[0126] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0127] The following briefly summarizes the processor frequency modulation method provided in the embodiment of the present application in combination with the specific game scenario and the above-mentioned software architecture.

[0128] After the game application is started, the game application creates a rendering thread and executes the rendering process through the rendering thread, that is, sequentially renders the image frames to be displayed by the game application, and stores the rendered image frames in the graphics buffer. For the image frames stored in the graphics buffer, SurfaceFlinger can read the cached image frames from the graphics buffer, and synthesize and display the read image frames so that the display screen displays the image frames. During the rendering process of the rendering thread, the frame buffer production and consumption monitoring module can monitor the buffer storage time of each frame rendered by the rendering thread, that is, the time when the frame data of each frame is stored in the graphics buffer after rendering, and send the monitored buffer storage time of the image frame to the frame buffer residence time statistics module. During the synthesis process of SurfaceFlinger, the frame buffer production and consumption monitoring module can monitor the buffer retrieval time of each frame synthesized by SurfaceFlinger, that is, the time when the frame data of each frame is taken out of the buffer, and send the monitored cache retrieval time of the image frame to the frame buffer residence time statistics module. The frame buffer residence time statistics module can determine the residence time of the image frame based on the buffer storage time and buffer retrieval time of the latest synthesized image frame, and send the residence time of the image frame to the frequency adjustment feedback algorithm module. The frequency adjustment feedback algorithm module determines whether frequency modulation (increasing or reducing the frequency) is required based on the comparison result of the residence time and the target time, and sends a frequency modulation instruction to the CPU frequency modulation driver through the CPU frequency adjustment interface when frequency modulation is required to control the CPU where the rendering thread is located to perform frequency modulation.

[0129] The execution subject of the video processing method provided in the embodiment of the present application can be the above-mentioned electronic device, or it can be a functional module and / or functional entity in the electronic device that can implement the video processing method, and the present application scheme can be implemented by hardware and / or software, which can be determined according to actual use requirements, and the embodiment of the present application is not limited. The following takes an electronic device as an example, and combines the accompanying drawings to exemplarily illustrate the video processing method provided in the embodiment of the present application.

[0130] The specific implementation process of the frequency modulation method of the processor provided in the embodiment of the present application is described below with reference to the accompanying drawings.

[0131] Fig.10 is a schematic flow chart of a frequency modulation method of a processor provided in an embodiment of the present application, and the method can be executed by an electronic device. The software structure block diagram of the electronic device can be as follows Fig. 9 As shown, but the embodiments of the present application are not limited thereto. Fig.10 As shown, the method comprises the following steps:

[0132] Step 101: A user performs a first operation.

[0133] The first operation is used to start the game application. For example, the first operation is an operation in which the user clicks on the icon of the game application.

[0134] Step 102: In response to the first operation, the game application is started.

[0135] In an embodiment of the present application, after the game application is started, in order to reduce the probability of frame loss of the game application and save power consumption, the frequency modulation module can monitor the residence time of each frame rendered by the game application, and the processor can be frequency modulated according to the monitored residence time.

[0136] Before frequency modulation is performed through the frequency modulation module, it is generally necessary to initialize the frequency modulation module, such as initializing the relevant algorithms and parameters of the frequency modulation module. For example, the specific initialization process can be implemented through the following steps 103 to 105.

[0137] Step 103: the game application sends a game application startup notification to the algorithm parameter parsing configuration module of the frequency modulation module.

[0138] The startup notification is used to instruct the frequency modulation module to start the game application.

[0139] It should be noted that the embodiment of the present application only takes the example of the game application notifying the game application of startup by sending a startup notification to the frequency modulation module. It should be understood that the frequency modulation module can also use other methods to obtain the startup of the game application, such as obtaining the startup of the game application through a related interface. The embodiment of the present application does not limit this.

[0140] Step 104: the algorithm parameter parsing configuration module determines initialization configuration information, where the initialization configuration information at least includes a target duration T and a duration difference tolerance.

[0141] As an example, the algorithm parameter parsing configuration module may determine the target duration T corresponding to the dwell time of the frame buffer of the game application according to the frame rate of the game application.

[0142] In a possible implementation, the startup notification sent by the game application carries the identification and frame rate of the game application, and the algorithm parameter parsing configuration module can obtain the frame rate of the game application according to the startup notification. Of course, the algorithm parameter parsing configuration module can also obtain the frame rate of the game application in other ways, such as obtaining the frame rate of the game application by calling a related interface, which is not limited in the embodiment of the present application.

[0143] In a possible implementation, after the algorithm parameter parsing configuration module determines the frame rate of the game application, it can obtain the target duration T corresponding to the frame rate of the game application from the correspondence table between the frame rate and the target duration. The correspondence table stores multiple frame rates and the target duration corresponding to each frame rate.

[0144] It should be understood that the correspondence table and the time difference tolerance can be pre-stored and obtained by the algorithm parameter parsing configuration module itself, or can be obtained by the algorithm parameter parsing configuration module through other methods, such as by calling relevant interfaces. The embodiment of the present application does not limit the method of obtaining the correspondence table and the time difference tolerance.

[0145] It should also be understood that the initialization configuration information may also include other configuration information, such as a specific frequency adjustment feedback algorithm, etc., which is not limited in the embodiments of the present application.

[0146] Step 105: the algorithm parameter parsing configuration module initializes other modules in the frequency modulation module according to the initialization configuration information.

[0147] As an example, the initialization operation at least includes: the algorithm parameter parsing configuration module sends parameter configuration information such as target duration T and duration difference tolerance to the frequency regulation feedback algorithm module, so that the frequency regulation feedback algorithm module initializes relevant parameters according to the parameter configuration information.

[0148] It should be understood that the algorithm parameter parsing configuration module can also send corresponding initialization configuration information to other modules other than the frequency adjustment feedback algorithm module to initialize the configuration of other modules, and the embodiments of the present application will not give examples one by one for this.

[0149] Step 106: The game application creates a rendering thread.

[0150] After the game application is started, a rendering thread can be created to render the application screen to be displayed through the rendering thread.

[0151] Step 107: The rendering thread of the game application starts to execute the rendering process.

[0152] The rendering process refers to the process of rendering image frames, and the image frames can be rendered sequentially according to the rendering logic. The rendering process of each image frame may include: obtaining a graphic buffer from the buffer queue, writing the frame data of the rendered image frame into the graphic buffer, and then submitting the graphic buffer to the buffer queue. The following will be illustrated by steps 108 and 109.

[0153] As an example, the rendering thread of the game application can be rendered through the view system of the application framework layer. For example, the rendering thread can send a rendering instruction to the view system, and after the view system receives the rendering instruction, it starts to execute the rendering process according to the rendering instruction to render the image frame.

[0154] Step 108: The rendering thread obtains graphic buffer 1 (hereinafter referred to as buffer 1) from the buffer queue.

[0155] For example, the rendering thread can get graphicsbuffer 1 from the buffer queue by calling the dequeue buffer method.

[0156] Step 109: The rendering thread writes the frame data of the rendered image frame 1 into buffer 1.

[0157] Step 110: The rendering thread submits buffer 1 to the buffer queue.

[0158] For example, the rendering thread can submit buffer 1 to the bufferqueue by calling the queue buffer method, that is, returning buffer 1 to the buffer queue, so that buffer 1 is re-queued.

[0159] In addition, after receiving the queue entry message of buffer1, buffer queue can also send a queue entry notification to Surfaceflinger. The queue entry notification is used to notify Surfaceflinger that a graphic buffer has been prepared for consumption by Surfaceflinger.

[0160] Step 111: When the Vsync signal arrives, Surfaceflinger takes buffer 1 from the buffer queue.

[0161] For example, Surfaceflinger can get buffer 1 from the buffer queue by calling the acquire buffer method.

[0162] Step 112: Surfaceflinger reads frame data from buffer 1.

[0163] In addition, after Surfaceflinger reads the cached frame data from buffer 1, it can also submit buffer 1 to the buffer queue by calling the release buffer method, that is, return graphic buffer 1 to the buffer queue so that graphic buffer 1 can be reused.

[0164] Step 113: Surfaceflinger synthesizes and displays the read frame data so that the display screen displays image frame 1.

[0165] As an example, Surfaceflinger can synthesize and display the frame data through HWC. For example, Surfaceflinger can send the read frame data to HWC, and HWC synthesizes and displays the received frame data.

[0166] Step 114: During the rendering process of the rendering thread, the frame buffer production and consumption monitoring module monitors the buffer storage time of each frame rendered by the rendering thread.

[0167] The buffer storage time of each frame refers to the time that the frame data of each frame is stored in the buffer after rendering. Specifically, it can be the time that the rendering process writes the frame data of each rendered frame into the graphic buffer, or it can be the time that the graphic buffer is submitted to the bufferqueue after the frame data of each rendered frame is written into the graphic buffer. The embodiment of the present application does not limit this.

[0168] Step 115: The frame buffer production and consumption monitoring module sends the buffer storage time 1 of the monitored image frame 1 to the frame buffer residence time statistics module.

[0169] For example, the frame buffer production and consumption monitoring module can determine the buffer storage time of image frame 1 when it monitors that the frame data of image frame 1 is stored in the buffer zone, and send the buffer storage time 1 of image frame 1 to the frame buffer residence time statistics module.

[0170] Step 116: During the synthesis process of Surfaceflinger, the frame buffer production and consumption monitoring module monitors the buffer fetching time of each frame synthesized by Surfaceflinger.

[0171] The buffer fetching time of each frame refers to the time when the frame data of each frame is fetched from the buffer. Specifically, it can be the time when the rendering process obtains the graphic buffer where the frame data of each frame is located from the buffer queue, or it can be the time when the frame data of each frame is read from the graphic buffer after the graphic buffer where the frame data of each frame is located is obtained. This embodiment of the application does not limit this.

[0172] Step 117: the frame buffer production and consumption monitoring module sends the monitored buffer fetching time 1 of the image frame 1 to the frame buffer residence time statistics module.

[0173] Step 118: The frame buffer residence time statistics module determines the residence time t of the image frame 1 according to the buffer storage time 1 and the buffer retrieval time 1 of the image frame 1.

[0174] The frame buffer residence time can be calculated based on the buffer storage time and buffer retrieval time of each frame monitored by the frame buffer production and consumption monitoring module. The residence time of each frame is the time between the buffer storage time and the buffer retrieval time of each frame, that is, the cache time of each frame in the graphics buffer.

[0175] For example, after receiving the buffer retrieval time 1 of image frame 1 monitored by the frame buffer production and consumption monitoring module, the frame buffer residence time statistics module can determine the residence time t of image frame 1 based on the buffer storage time 1 and buffer retrieval time 1 of image frame 1.

[0176] Step 119: the frame buffer dwell time statistics module sends the dwell time t of the image frame 1 to the frequency adjustment feedback algorithm module.

[0177] Step 120: The frequency adjustment feedback algorithm module compares |tT| with the time difference tolerance.

[0178] After receiving the dwell time t of the image frame 1 , the frequency adjustment feedback algorithm module may first compare |tT| with the time difference tolerance to determine whether |tT| is less than the time difference tolerance.

[0179] Step 121: When |tT|<tolerance, the frequency adjustment feedback algorithm does not perform frequency modulation.

[0180] When |tT|<tolerance, it means that the residence time t is not much different from the target time T and is relatively stable. Therefore, the frequency modulation operation may not be performed to keep the frequency of the CPU where the rendering thread of the game application is located unchanged.

[0181] In the embodiment of the present application, by introducing a variable time difference tolerance, frequency modulation is not performed when |tT|<tolerance, which can reduce the overhead caused by frequent frequency modulation and save power consumption.

[0182] Step 122: When |tT|>tolerance and t<T, the frequency regulation feedback algorithm generates a frequency increase instruction.

[0183] When |tT|>tolerance, it means that the residence time t is significantly different from the target time T, the load and production speed of Surfaceflinger are unstable, and frequency adjustment is required. Therefore, the residence time t can be compared with the target time T, and the frequency can be increased or decreased according to the comparison result of the residence time t and the target time T.

[0184] For example, if t<T, it means that the load of Surfaceflinger increases, the production speed slows down, and the resource supply of Surfaceflinger does not meet the load demand. Therefore, a frequency increase instruction can be generated at this time to increase the frequency of the CPU where the rendering thread is located through the frequency increase instruction to appropriately increase the resource supply of Surfaceflinger.

[0185] Step 123: The frequency adjustment feedback algorithm sends the frequency increase instruction to the CPU frequency adjustment driver of the kernel layer through the CPU frequency adjustment interface.

[0186] The frequency regulation feedback algorithm sends a frequency increase instruction to the CPU frequency regulation driver through the CPU frequency regulation interface to instruct the CPU frequency regulation driver to control the CPU where the rendering thread is located to increase the frequency.

[0187] For example, the frequency adjustment feedback algorithm first sends the frequency increase instruction to the CPU frequency adjustment interface, and the CPU frequency adjustment interface forwards the frequency increase instruction to the CPU frequency adjustment driver.

[0188] Step 124: The CPU frequency modulation driver sends a frequency increase driving instruction to the CPU where the rendering thread of the game application is located according to the frequency increase instruction.

[0189] The frequency increase driving instruction is used to drive the corresponding CPU to increase the frequency.

[0190] The CPU where the rendering thread of the game application is located may include one or more CPUs, that is, the rendering thread of the game application may be run by one or more CPUs. Figure 5 As shown, the CPUs where the rendering threads of the game application are located may include CPU4, CPU5 and CPU6.

[0191] Step 125: The CPU increases the frequency according to the frequency increasing driving instruction.

[0192] After receiving the frequency increase driving instruction, the CPU can increase the frequency according to the frequency increase driving instruction. For example, the current frequency point can be increased by m frequency points. Among them, m is a positive integer and can be set as needed, such as 1, 2 or 3. The difference between two adjacent frequency points is a fixed frequency, such as a difference of 200KHz.

[0193] For example, assuming that m is 1, the CPU where the rendering thread is located includes CPU4, and the original frequency point of CPU4 is 2 (the corresponding frequency band is 890.2MHz-890.2MHz), then CPU4 can increase the frequency point by 1 frequency point according to the frequency increase driving instruction, that is, adjust the frequency point from 2 to 3 (the corresponding frequency band is 890.4MHz-890.6MHz).

[0194] Step 126: When |tT|>tolerance and t>T, the frequency adjustment feedback algorithm generates a frequency reduction instruction.

[0195] If t>T, it means that the load of Surfaceflinger becomes smaller, the production speed becomes faster, and the resource supply of Surfaceflinger exceeds the load demand. Therefore, a frequency reduction instruction can be generated at this time to reduce the frequency of the CPU where the rendering thread is located through the frequency reduction instruction to appropriately reduce the resource supply of Surfaceflinger, thereby reducing power consumption.

[0196] Step 127: The frequency adjustment feedback algorithm sends the frequency reduction instruction to the CPU frequency adjustment driver of the kernel layer through the CPU frequency adjustment interface.

[0197] The frequency regulation feedback algorithm sends a frequency reduction instruction to the CPU frequency regulation driver through the CPU frequency regulation interface to instruct the CPU frequency regulation driver to control the CPU where the rendering thread is located to reduce the frequency.

[0198] For example, the frequency adjustment feedback algorithm first sends the frequency reduction instruction to the CPU frequency adjustment interface, and the CPU frequency adjustment interface forwards the frequency reduction instruction to the CPU frequency adjustment driver.

[0199] Step 128: The CPU frequency modulation driver sends a frequency reduction driving instruction to the CPU where the rendering thread is located according to the frequency reduction instruction.

[0200] The frequency increase driving instruction is used to drive the corresponding CPU to reduce the frequency.

[0201] Step 129: The CPU reduces the frequency point according to the frequency reduction driving instruction.

[0202] After receiving the frequency reduction driving instruction, the CPU can reduce the frequency according to the frequency reduction driving instruction. For example, the frequency point can be reduced by n frequency points. Among them, n is a positive integer, which can be set as needed, such as 1, 2 or 3. The difference between two adjacent frequency points is a fixed frequency, such as a difference of 200KHz.

[0203] For example, assuming that n is 1, the CPU where the rendering thread is located includes CPU4, and the original frequency point of CPU4 is 2 (the corresponding frequency band is 890.2MHz-890.2MHz), then CPU4 can reduce the frequency point by 1 frequency point according to the frequency reduction drive instruction, that is, adjust the frequency point from 2 to 1 (the corresponding frequency band is 890.0MHz-890.2MHz).

[0204] In the embodiment of the present application, on the one hand, by increasing the frequency of the processor where the rendering thread of the game application is located when the residence time of the latest synthesized image frame is less than the target time, a prediction of possible frame loss can be made at least one frame in advance, so that more timely intervention can be made for scheduling, reasonable frequency adjustment and other resource configurations can be made, such as timely improving the processing performance of the processor, thereby increasing the production speed of the image producer, thereby better reducing the possibility of frame loss in the game application, thereby reducing the freeze phenomenon of the game screen, and improving the user's gaming experience. On the other hand, by down-clocking the processor where the rendering thread of the game application is located when the residence time of the latest synthesized image frame is greater than the target time, that is, reducing the frequency of the processor where the rendering thread is located, the frequency of the processor can be reasonably configured without being too aggressive, and efforts can be made to ensure the stability of the frame residence time, thereby improving power consumption and performance, and saving power consumption.

[0205] The electronic device provided in the embodiments of the present application may be a user equipment (UE), for example, a mobile electronic device (such as a user's mobile phone), a tablet computer, a desktop computer, a laptop computer, a handheld computer, a netbook, a personal digital assistant (PDA), and the like.

[0206] Next, the electronic device involved in the embodiment of the present application is described.

[0207] Fig.11is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.11 The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0208] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0209] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0210] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0211] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0212] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0213] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0214] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0215] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0216] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0217] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0218] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. GNSS may include the global positioning system (GPS), the global navigation satellite system (GLONASS), the Beidou navigation satellite system (BDS), the quasi-zenith satellite system (QZSS) and / or the satellite based augmentation system (SBAS).

[0219] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0220] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is an integer greater than 1.

[0221] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0222] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0223] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is an integer greater than 1.

[0224] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0225] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0226] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0227] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0228] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created by the electronic device 100 during use (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0229] The electronic device 100 can implement audio functions, such as music playback, recording, etc., through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D and the application processor.

[0230] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than a pressure threshold acts on a short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the pressure threshold acts on a short message application icon, an instruction to create a new short message is executed.

[0231] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor 180K can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0232] The present application also provides a chip, which is coupled to a memory and is used to read and execute computer programs or instructions stored in the memory to execute the methods in the above embodiments.

[0233] The present application also provides an electronic device, which includes a chip, and the chip is used to read and execute a computer program or instruction stored in a memory, so that the method in each embodiment is executed.

[0234] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the video processing method in the above-mentioned embodiment.

[0235] This embodiment further provides a computer program product, wherein the computer-readable storage medium stores program codes. When the computer program product is run on a computer, the computer executes the above-mentioned related steps to implement the video processing method in the above-mentioned embodiment.

[0236] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the video processing method in the above-mentioned method embodiments.

[0237] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0238] The embodiments of the present application do not specifically limit the specific structure of the execution subject of the method provided in the embodiments of the present application, as long as the video processing can be performed according to the method provided in the embodiments of the present application by running a program that records the code of the method provided in the embodiments of the present application. For example, the execution subject of the method provided in the embodiments of the present application may be an electronic device, or a functional module in the electronic device that can call and execute the program.

[0239] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0240] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0241] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0242] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a computer software product, which is stored in a storage medium. The computer software product includes a number of instructions that enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium may include, but is not limited to: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0243] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A frequency modulation method for a processor, characterized in that: Applied to electronic equipment, the method comprises: In response to a first operation, starting a first application and displaying an application screen of the first application; During the running of the first application, obtaining the residence time of the latest synthesized image frame of the first application, where the residence time refers to the time from when the image frame is stored in the graphics buffer to when it is taken out of the graphics buffer for synthesis after rendering is completed; According to the comparison result of the residence time and the target time, the frequency of the target processor running the rendering thread of the first application is adjusted.

2. The method according to claim 1, characterized in that The adjusting, according to the comparison result of the residence time and the target time, the frequency of the target processor running the rendering thread of the first application includes: If the residence time is less than the target time, increasing the frequency of the target processor; If the residence time is greater than the target time, the frequency of the target processor is reduced.

3. The method according to claim 1 or 2, characterized in that Before adjusting the frequency of the target processor running the rendering thread of the first application according to the comparison result of the residence time and the target time, the method further includes: Determine a time difference between the dwell time and the target time; If the duration difference is greater than or equal to the duration difference tolerance, a step of adjusting the frequency of a target processor running the rendering thread of the first application according to a comparison result of the residence duration and the target duration is performed.

4. The method according to claim 2, characterized in that The step of increasing the frequency of the target processor comprises: Increasing the frequency value of the target processor by a first frequency value; Continuing to obtain the dwell time of the next image frame synthesized by the first application; If the residence time of the next image frame is less than the target time, the frequency value of the target processor continues to be increased by the first frequency value until the residence time of the image frame most recently synthesized by the first application is greater than or equal to the target time.

5. The method according to claim 2, characterized in that The step of reducing the frequency of the target processor comprises: reducing the frequency value of the target processor by a second frequency value; Continuing to obtain the dwell time of the next image frame synthesized by the first application; If the residence time of the next image frame is greater than the target time, the frequency value of the target processor continues to be reduced by the second frequency value until the residence time of the image frame most recently synthesized by the first application is less than or equal to the target time.

6. The method according to any one of claims 1 to 5, characterized in that: The obtaining of the dwell time of the latest synthesized image frame of the first application includes: Monitoring the rendering process of the rendering thread of the first application to obtain the time when the rendering thread stores the image frame in the graphics buffer after rendering the image frame, thereby obtaining the buffer storage time of the image frame; Monitoring the synthesis process of the image consumer corresponding to the rendering process to obtain the time when the image consumer takes out the image frame from the graphics buffer for synthesis, and obtaining the buffer fetching time of the image frame; The residence time of the image frame is determined according to the buffer storage time and the buffer retrieval time of the image frame.

7. The method according to any one of claims 1 to 6, characterized in that: Before adjusting the frequency of the target processor running the rendering thread of the first application according to the comparison result of the residence time and the target time, the method further includes: determining a frame rate of the first application; The target duration corresponding to the frame rate of the first application is determined from a correspondence between the frame rate and the target duration, wherein the correspondence includes a plurality of frame rates and target durations corresponding to different frame rates.

8. The method according to any one of claims 1 to 7, characterized in that: When the frame rate of the first application is 60fps, the target duration is 8.3ms; When the frame rate of the first application is 90fps, the target duration is 5.5ms.

9. The method according to any one of claims 1 to 8, characterized in that: Before adjusting the frequency of the target processor running the rendering thread of the first application according to the comparison result of the residence time and the target time, the method further includes: Obtaining the dwell time of multiple historical frames rendered by the first application in a state of stable frame rate, where the stable frame rate state refers to a state where the first application does not lose frames and the jitter rate of the frame rate is less than a preset jitter rate; The target duration is determined according to the residence durations of the multiple historical frames.

10. The method according to any one of claims 1 to 9, characterized in that: The first application is a game application.

11. An electronic device, characterized in that: The electronic device comprises: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method as described in any one of claims 1 to 10.

12. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, and when the instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 10.

Images