Frequency modulation method of processor, electronic device, chip system and storage medium

Abstract

The present application discloses a frequency modulation method for a processor, an electronic device, a chip system, and a storage medium, belonging to the technical field of terminals. The method includes: in response to a first operation, starting a first application and displaying an application screen of the first application; during the running of the first application, obtaining the residence duration of the latest synthesized image frame of the first application, where the residence duration refers to the duration from when the image frame is stored in the graphics buffer after rendering to when it is taken out of the graphics buffer for synthesis; and adjusting the frequency of the target processor of the rendering thread running the first application according to the comparison result between the residence duration and the target duration. By performing frequency modulation on the processor based on the frame buffer residence duration, the present application can improve the timeliness of frequency modulation and reduce or avoid random frame dropping phenomena.

Description

Technical Field

[0001] This application relates to the technical field of terminals, and particularly to a frequency modulation method for a processor, an electronic device, a chip system, and a storage medium. Background Art

[0002] Playing games is an important scenario in the use of terminals such as mobile phones and tablets, and the game experience is also an important indicator for evaluating the pros and cons of terminals. The frame rate and jitter rate of games are important indicators affecting the user's game experience. A poor frame rate or a high jitter rate will result in a poor game experience. At this time, it is necessary for the device manufacturers of the terminals to perform platform-level optimizations.

[0003] The direct cause of a poor frame rate and a high jitter rate is random frame dropping. Currently, terminals generally adopt a load-based resource scheduling algorithm to improve the random frame dropping phenomenon. Specifically, the load information in the rendering process of the game screen is collected in real time, and the load changes in the rendering process are statistically analyzed based on the load information of multiple time windows. Reasonable processor resources are allocated to the rendering process according to the load changes. For example, if the load changes indicate that the load in multiple consecutive time windows is greater than the load threshold, the frequency of the processor where the rendering thread is located is increased, so that the processor runs the rendering thread of the game screen at a higher frequency point, thereby reducing the subsequent frame dropping probability.

[0004] However, the load-based resource scheduling algorithm needs to accumulate and statistically analyze the load changes for a certain period of time before performing frequency modulation processing. Therefore, it may not be able to promptly perceive the load changes and accurately predict the frame dropping timing, and thus may not be able to promptly allocate the due resources to cooperate with the rendering process, resulting in the frequency point of the processor not being promptly increased when frame dropping is about to occur, and the improvement effect of the random frame dropping phenomenon is not good. Summary of the Invention

[0005] This application provides a frequency modulation method for a processor, an electronic device, a chip system, and a storage medium, which can improve the timeliness of frequency modulation and reduce or avoid random frame dropping. The technical solutions are as follows:

[0006] In a first aspect, a frequency modulation method for a processor is provided, which is applied to an electronic device. The method includes: in response to a first operation, starting a first application and displaying an application screen of the first application; during the running of the first application, obtaining the residence duration of the latest synthesized image frame of the first application; and adjusting the frequency of a target processor running the rendering thread of the first application according to the comparison result between the residence duration and a target duration.

[0007] Among them, the residence duration refers to the caching duration of the image frame in the graphics buffer, that is, the duration from when the image frame is stored in the graphics buffer after rendering to when it is taken out of the graphics buffer for synthesis. The target duration refers to the stable duration corresponding to the residence duration. That is, in the state of stable frame rate, the residence duration of each frame generally maintains around the target duration. Among them, the state of stable frame rate refers to the state where the application does not drop frames and the jitter rate of the frame rate is less than the preset jitter rate. The preset jitter rate can be set as needed. For example, it can be the maximum jitter rate that does not affect the display effect.

[0008] In the case of stable frame rate, the residence duration of each image frame should maintain around the target duration. By comparing the residence duration with the target duration, the load change and production speed change of the image producer can be sensed in a timely manner, so as to predict the possible frame drop situation in a timely and accurate manner. Therefore, according to the comparison result of the residence duration and the target duration, the processor of the rendering thread running the application is frequency modulated, and frequency modulation can be performed in a timely manner when frame drop may occur, so as to reduce or avoid the random frame drop phenomenon of the application, and further reduce or avoid the phenomenon of frame freeze in the application screen, improving the user's application experience.

[0009] In one embodiment, the residence durations of multiple historical frames rendered by the application in the state of stable frame rate can be obtained, and the target duration can be determined according to the residence durations of the multiple historical frames. For example, the average residence duration of the multiple historical frames is determined as the target duration. In this way, a more accurate target duration can be obtained, improving the accuracy of subsequent frequency modulation based on the residence duration.

[0010] In one embodiment, the target duration T matches the frame rate of the application. For example, to ensure the accuracy of frequency modulation, different target durations T can be set respectively in different frame rate scenarios. For example, when the frame rate of the application is 60fps, the target duration T = 8.3ms. When the frame rate of the application is 90fps, the target duration T = 5.5ms.

[0011] In a possible implementation manner, the corresponding relationship between the frame rate and the target duration can be preset. The corresponding relationship includes multiple frame rates and the target durations corresponding to different frame rates. When setting the target duration, the target duration corresponding to the current frame rate of the application can be determined from this corresponding relationship. Among them, the target duration corresponding to each frame rate can be determined according to the residence durations of multiple historical frames rendered by the application at the corresponding frame rate and in the state of stable frame rate. For example, it is determined according to the average residence duration of the multiple historical frames.

[0012] In one embodiment, when the residence duration of the latest synthesized image frame is less than the target duration, the frequency of the processor where the rendering thread of the first application is located is increased, that is, the frequency of the processor where the rendering thread of the first application is located is increased. In this way, the processing performance of the processor can be improved in a timely manner, and then the production speed of the image producer can be increased to prevent frame drops. On the contrary, when the residence duration of the latest synthesized image frame is greater than the target duration, the frequency of the processor where the rendering thread of the first application is located is decreased, that is, the frequency of the processor where the rendering thread is located is decreased, so as to reduce the production speed of the image producer and save power consumption.

[0013] In one embodiment, in order to avoid frequent frequency modulation of the processor and save power consumption, a duration difference tolerance can also be set in advance. Before performing frequency modulation, the duration difference between the residence duration and the target duration is first compared with the duration difference tolerance. If the duration difference between the two is less than or equal to the duration difference tolerance, the frequency of the processor where the rendering thread of the first application is located is not modulated. If the duration difference between the two is greater than the duration difference tolerance, the sizes of the residence duration and the target duration are compared again. If the residence duration of the latest synthesized image frame is less than the target duration, the frequency of the processor where the rendering thread of the first application is located is increased; if the residence duration of the latest synthesized image frame is greater than the target duration, the frequency of the processor where the rendering thread of the first application is located is decreased.

[0014] In one embodiment, considering that the overhead caused by frequent frequency modulation is relatively large, in order to save power consumption and avoid drastic fluctuations (sharp increases and decreases) in frequency, the frequency of the processor where the rendering thread of the first application is located can also be gradually adjusted when performing frequency modulation on the processor where the rendering thread of the first application is located. In this way, the problem that the frequency point is increased to the highest frequency point during frequency increase in the resource scheduling algorithm based on load, which is not friendly to power consumption, can be improved.

[0015] As an example, when increasing the frequency, the frequency value of the target processor can be first increased by a first frequency value, and then the residence duration of the next image frame synthesized by the first application is continuously obtained. If the residence duration of the next image frame is less than the target duration, the frequency value of the target processor is continuously increased by the first frequency value until the residence duration of the latest synthesized image frame of the first application is greater than or equal to the target duration.

[0016] As an example, when decreasing the frequency, the frequency value of the target processor can be first decreased by a second frequency value, and then the residence duration of the next image frame synthesized by the first application is continuously obtained. If the residence duration of the next image frame is greater than the target duration, the frequency value of the target processor is continuously decreased by the second frequency value until the residence duration of the latest synthesized image frame of the first application is less than or equal to the target duration.

[0017] In one embodiment, it is possible to monitor the rendering process of the rendering thread of the first application to obtain the time when the rendering thread stores the image frame in the graphics buffer after rendering the image frame, so as to obtain the buffer storage time of the image frame; monitor the composition process of the image consumer corresponding to the rendering process to obtain the time when the image consumer takes out the image frame from the graphics buffer for composition, so as to obtain the buffer extraction time of the image frame; and determine the residence duration of the image frame according to the buffer storage time and the buffer extraction time of the image frame.

[0018] By monitoring the rendering process of the rendering thread of the first application and the composition process of the image consumer (such as SurfaceFlinger), it is possible to timely monitor the buffer storage time and the buffer extraction time of the newly synthesized image frame, and timely determine the residence duration of the image frame according to the buffer storage time and the buffer extraction time of the image frame.

[0019] In one embodiment, the first application can be a game application, a video application, a video call application, etc., and the embodiments of the present application do not limit this.

[0020] In a second aspect, a frequency modulation device for a processor is provided. The frequency modulation device for the processor has the function of implementing the behavior of the frequency modulation method for the processor in the first aspect above. The frequency modulation device for the processor includes at least one module, and the at least one module is used to implement the frequency modulation method for the processor provided in the first aspect above.

[0021] In a third aspect, a frequency modulation device for a processor is provided. The structure of the frequency modulation device for the processor includes a processor and a memory. The memory is used to store a program for supporting the frequency modulation device for the processor to execute the frequency modulation method for the processor provided in the first aspect above, and to store data involved in implementing the frequency modulation method for the processor described in the first aspect above. The processor is configured to execute the program stored in the memory. The frequency modulation device for the processor may further include a communication bus, and the communication bus is used to establish a connection between the processor and the memory.

[0022] In a fourth aspect, a computer-readable storage medium is provided. Instructions are stored in the computer-readable storage medium, and when it runs on a computer, it causes the computer to execute the frequency modulation method for the processor described in the first aspect above.

[0023] In a fifth aspect, a computer program product containing instructions is provided. When it runs on a computer, it causes the computer to execute the frequency modulation method for the processor described in the first aspect above.

[0024] The technical effects obtained in the second, third, fourth, and fifth aspects above are similar to the technical effects obtained by the corresponding technical means in the first aspect above, and will not be elaborated here. Description of the Drawings

[0025] Figure 1 is a system schematic diagram of an image rendering system provided by an embodiment of the present application;

[0026] Figure 2 is a producer - consumer framework diagram of a buffer queue;

[0027] Figure 3 is a schematic diagram of a multi - core processor architecture of an electronic device provided by an embodiment of the present application;

[0028] Figure 4 is a schematic diagram of a game screen of a game application provided by an embodiment of the present application;

[0029] Figure 5 is a trace diagram of a rendering process of an application provided by an embodiment of the present application;

[0030] Figure 6 is a schematic diagram of the residence duration of an image frame provided by an embodiment of the present application;

[0031] Figure 7 Another trace diagram of the composition process of SurfaceFlinger provided by an embodiment of the present application;

[0032] Figure 8 is a flowchart of a method for frequency modulation of a processor based on the residence duration of a frame provided by an embodiment of the present application;

[0033] Figure 9 is a block diagram of a software system of an electronic device provided by an embodiment of the present application;

[0034] Figure 10 is a schematic flowchart of a method for frequency modulation of a processor provided by an embodiment of the present application;

[0035] Figure 11 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed Description of the Embodiment

[0036] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures, technologies, etc. are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details.

[0037] It should be understood that when used in the description of the present application specification and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0038] It should also be understood that in the embodiments of the present application, "one or more" means one, two, or more than two; "and / or" describes the association relationship of associated objects and indicates that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0039] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", "fourth", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0040] The reference to "one embodiment" or "some embodiments" etc. described in the present application specification means that in one or more embodiments of the present application, the specific features, structures, or characteristics described in connection with that embodiment are included. Thus, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized.

[0041] For ease of understanding, the nouns related to the embodiments of the present application are first described.

[0042] Frame rate: It refers to the number of frames generated per second, that is, the frequency at which bitmap images in frames continuously appear on the display. The unit of frame rate is fps (frame per second, frames per second). The frame rate represents the smoothness of the picture, and the higher the frame rate, the better the smoothness of the picture.

[0043] Due to the visual persistence effect of the human eye, when the frame rate is high enough (usually around 50fps to 60fps), the picture can look continuous and smooth. Currently, in order to improve the gaming experience, the frame rate of game applications is generally 60fps, 90fps, or 120fps, etc.

[0044] Jitter rate: The rate of change of the frame rate, generally calculated based on the frame rate difference between two adjacent frames. The higher the jitter rate, the greater the fluctuation of the frame rate, the worse the frame rate stability, and the worse the smoothness of the picture.

[0045] Dropped frame: Also known as dropped frame, it usually refers to the situation where some frames fail to be correctly displayed or processed in any application that requires a continuous frame rate, such as video playback, animation rendering, or games. Dropped frames will cause the picture to freeze and be incoherent.

[0046] Graphics buffer: A container for storing rendered and pending composite frame data.

[0047] Residence duration: The caching duration (residence duration) of the rendered frame data in the graphics buffer. That is, the duration from when the image producer finishes producing a frame of data to when the consumer retrieves this frame of data from the graphics buffer for consumption.

[0048] To facilitate the understanding of terms such as frame buffer and residence duration, the overall architecture of the image rendering system is described below.

[0049] Please refer to Figure 1 , Figure 1 which is a system schematic diagram of an image rendering system provided by an embodiment of this application. The image rendering system mainly adopts the producer-consumer mode, and the core of screen rendering is the production and consumption of image data. As Figure 1 shown, the image rendering system mainly includes: an image producer, a graphics buffer, and an image consumer. The image data generated by the image producer is first stored in the graphics buffer, and then retrieved and consumed by the image consumer from the graphics buffer. This is a typical producer-consumer mode.

[0050] Image producer: Used to generate image data. The main image producers include MediaPlayer, CameraPreview, open graphics library for embedded systems (OpenGL ES), or native developer kits (NDK), etc. Among them, MediaPlayer and Camera Preview generate image data by directly reading the image source. NDK (such as Skia) and OpenGL generate image data through their own drawing capabilities. In addition, for an application (APP), the application (hereinafter referred to as the app) can generate image data by calling OpenGL, etc. through the rendering thread. Therefore, the app or the rendering thread of the app can also be called an image producer.

[0051] Graphic buffer: It is used to store the image data generated by the image producer, mainly assisting in the transmission of image data between the image producer and the image consumer. That is, after the image producer generates the image data, the image data is first stored in the graphic buffer, and then the image consumer fetches and consumes the image data from the graphic buffer.

[0052] Image consumer: It is used to fetch the image data from the graphic buffer and display the image data on the screen. Among them, the most common image consumer is SurfaceFlinger. After SurfaceFlinger fetches the image data from the graphic buffer, it can use OpenGL and / or the hardware composer (HWC) to synthesize the fetched image data, and then send the synthesized image data for display. That is, the consumption process of SurfaceFlinger includes synthesis and sending for display.

[0053] The graphic buffer is managed by a buffer queue, and the core logic of the buffer queue is the producer-consumer logic. Please refer to Figure 2 , Figure 2 which is a producer-consumer framework diagram of a buffer queue. As Figure 2 shown, the working process of the buffer queue includes:

[0054] 1) The image producer requests a piece of graphic buffer content, that is, a graphic buffer (which can also be simply referred to as a buffer), from the buffer queue by calling the dequeue method.

[0055] 2) The image producer writes the generated image data into the graphic buffer and submits it to the buffer queue through the queue method.

[0056] 3) The buffer queue receives the in-queue message of the graphic buffer and notifies the image consumer to call the acquire method to obtain the filled graphic buffer for processing or display.

[0057] 4) After the image consumer finishes processing, it calls the release method to return the graphic buffer to the buffer queue, and this graphic buffer can be reused later.

[0058] 5) The image producer, buffer queue, and image consumer work in coordination. The graphic buffer circulates among the three in a coordinated manner, and the image is then smoothly displayed or processed.

[0059] In the rendering process of the APP, the rendering thread of the APP serves as the image producer, and SurfaceFlinger serves as the image consumer. In one embodiment, when the rendering thread of the APP needs a graphic buffer, it requests a graphic buffer from the buffer queue by calling the dequeue buffer method and specifying the width, height, pixel format, and usage flags of the buffer. The rendering thread of the APP performs rendering. After the rendering is completed, the generated image data is written into the graphic buffer, and the queue buffer method is called to return the graphic buffer to the buffer queue. After receiving the Vsync signal, SurfaceFlinger starts preparing for composition, calls the acquire buffer method to obtain the graphic buffer from the buffer queue, and composes the image data in the graphic buffer. After the composition is completed, SurfaceFlinger returns the graphic buffer to the buffer queue by calling the release buffer method.

[0060] It should be understood that the buffer queue can adopt a double-buffer architecture or a 3-buffer architecture. Of course, with the progress of technology, a 4-buffer architecture or a 5-buffer architecture, etc., can also be adopted. The embodiments of this application do not limit the buffer architecture adopted by the buffer queue. In the 3-buffer architecture, the 3-buffer architecture of the buffer queue includes three graphic buffers, namely the front buffer, the back buffer, and the idle buffer. The front buffer is the buffer that the image consumer is currently consuming; the back buffer is the buffer that the image producer is filling with the content of the next frame; when the image producer is preparing the content in the back buffer, the idle buffer is in an idle state. Once the back buffer is consumed by the image consumer, the idle buffer will become the new back buffer, waiting for the image producer to fill the content of the next frame. Through this 3-buffer architecture, the buffer queue can ensure that there is always a buffer ready during the production and consumption process, thus avoiding display delays caused by waiting for new content. At the same time, it can also effectively utilize system resources and improve the overall performance.

[0061] Frequency modulation: Frequency modulation refers to adjusting the frequency of the central processing unit (CPU), such as increasing the frequency of the central processing unit (overclocking) or decreasing the frequency of the central processing unit (downclocking). In the embodiments of the present application, frequency modulation can be performed on the central processing unit where the rendering thread of the application is located (hereinafter referred to as the processor) to reasonably utilize the processor resources, improve the rendering performance, and reduce or avoid frame drops.

[0062] Frequency point: A frequency point is the number assigned to a fixed frequency, and each frequency point can correspond to a fixed frequency band. The larger the number, the larger the upper limit value of the corresponding frequency band. For example, the frequency interval between adjacent frequency points is 200KHz.

[0063] The following is an example of the processor architecture involved in the frequency modulation scenario.

[0064] The processor is the operation and control core of an electronic device, mainly used to implement information processing and program operation. To improve the processing speed and performance of the processor, multi-core processors are generally used in electronic devices. Processors can be divided into large cores, medium cores, and small cores. Compared with medium cores, large cores have higher computing power but also consume more power. Similarly, medium cores have higher computing power than small cores but also consume more power. For example, a multi-core processor includes small cores, medium cores, and large cores. The frequency of the medium core is greater than that of the small core, and the frequency of the large core is greater than that of the medium core.

[0065] As an example, please refer to Figure 3 , Figure 3 which is a schematic diagram of the multi-core processor architecture of an electronic device provided by the embodiments of the present application. As shown in Figure 3 , the electronic device has 12 processors (CPU0 - CPU11) with different architectures. These 12 processors include 4 large cores (CPU0 - 3), 4 medium cores (CPU4 - 7), and 4 small cores (CPU8 - 11). The frequencies of the large core, medium core, and small core increase in sequence. For example, the frequency band of the small core is 600M - 1.5G, that is, the minimum frequency is 600M and the maximum frequency is 1.5G; the frequency band of the medium core is 1.5G - 3.0G; the frequency band of the large core is 3.0G - 5.0G.

[0066] For a multi-core processor, the rendering thread of an application can be run by one or several processors in the multi-core processor, that is, the rendering thread of the application runs on all or part of the processors in the multi-core processor. However, the frequency of the processor where the rendering thread is located may not match the rendering thread, resulting in frame drops. In the embodiments of the present application, frequency modulation can be performed on the processor where the rendering thread of the application is located at an appropriate time to reasonably utilize the processor resources, improve the rendering performance, and reduce or avoid frame drops.

[0067] The frequency modulation method of the processor provided by the embodiments of the present application can be applied to an electronic device. The electronic device includes an electronic device, which can also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The electronic device can be a mobile phone, smart TV, wearable device, tablet computer (Pad), computer with wireless transceiver function, virtual reality (VR) electronic device, augmented reality (AR) electronic device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, and so on. The embodiments of the present application do not limit the specific technologies and specific device forms adopted by the electronic device.

[0068] Next, the application scenarios involved in the embodiments of the present application will be described.

[0069] Playing games is an important scenario in the use of terminals such as mobile phones and tablet computers, and the game experience is also an important indicator for evaluating the quality of terminals. The frame rate and jitter rate of games are basic and important evaluation items. With the continuous development of intelligent terminal devices, users' game experience requirements are also getting higher and higher. With the continuous development of terminal technologies, users' game experience requirements are also getting higher and higher. Currently, there are a variety of game applications in the application market. Some have complex game scenarios and randomly changing loads, and some have uneven development levels of developers, resulting in unsatisfactory performance in terms of frame rate and jitter rate, such as a low frame rate and / or a high jitter rate, thereby leading to a poor game experience. At this time, it is necessary for the device manufacturers of the terminals to make platform-level optimizations.

[0070] For example, due to the complexity of the game scene, the content volume of different game screens varies greatly, resulting in a large difference in the load required to render different game screens. As a result, frame drops may occur due to the timeout of the rendering duration when rendering a game screen with a large content volume. Exemplarily, when the game scene enters a new map, due to the complexity of the new map scene and the large amount of data, the load required to render the new map screen is large and the required duration is long. There may be a frame drop situation caused by the failure to render the new map in time, resulting in the game interface remaining on the current game screen after the user performs the operation of switching to the new map and failing to switch to the new map screen in time, causing the screen to freeze.

[0071] Please refer to Figure 4 , Figure 4 which is a schematic diagram of the game interface of a game application provided by an embodiment of the present application. As Figure 4 shown, the game interface of the game application first displays the game screen 310, and the game screen 310 includes a mini-map Figure 10 , and the game object operated by the user is currently located at point A in the mini-map. If the user wants to switch to other positions in the mini-map (i.e., switch to a new map), the user can click on point C in the mini-map on the game screen 310. In response to the user's operation, the new map corresponding to point C should be immediately entered, that is, the game screen 310 should be switched to the new map screen corresponding to point C. However, due to the complexity of the new map scene, the large amount of new content, and the large amount of data, the load required to render the new map screen is large and the required duration is long, resulting in the failure to render the new map screen in time when the screen refresh signal arrives. Therefore, the current game screen (game screen 311) is still displayed during screen refresh, and the rendered new map screen (game screen 312) is displayed until the next screen refresh signal arrives. For the user, after the user performs the operation of switching to the new map, the user will see the game screen freeze, and then switch to the new map after the freeze, and the user perceives that the operation is not smooth and the game experience is poor.

[0072] In order to improve the user's game experience, it is necessary for the device manufacturers of the terminal to perform platform-level optimization. The direct cause of high frame rate difference and high jitter rate is random frame drops. Currently, the terminal generally uses a load-based resource scheduling algorithm to improve the random frame drop phenomenon. However, the load-based resource scheduling algorithm is not accurate enough in perceiving and predicting the load changes in the rendering process of the game screen, and thus cannot allocate the necessary resources in time to cooperate with the rendering process, resulting in the failure to timely boost the frequency point of the processor when frame drops are about to occur, and the improvement effect of the random frame drop phenomenon is not good.

[0073] Please refer to Figure 5 , Figure 5 which is a behavior (trace) diagram of the rendering process of an application provided by an embodiment of the present application, and it shows the logs of each key point in the rendering process of application A. As Figure 5As shown, the key tasks ①②③ of the rendering thread of Application A are respectively running on CPU4, CPU5, and CPU6, and the maximum frequencies of CPU4, CPU5, and CPU6 are 2.5 GHz. Assume the frame rate of Application A is 60 fps, then the average rendering time interval between two frames should be 16.6 ms. As Figure 5 As shown, the rendering completion times of the i-th frame and the (i + 1)-th frame differ by 16.6 ms, and after the (i + 1)-th frame is rendered, the frequencies of CPU4, CPU5, and CPU6 drop to 600 MHz. In the case of a stable frame rate, the rendering of the (i + 2)-th frame should start at a time point (shown as ④ in the figure) that also differs by 16.6 ms from the rendering completion time of the (i + 1)-th frame. However, due to the large data volume of the (i + 2)-th frame and the relatively high required load, the frequency point at ⑤ when the load starts to run is too low, so the rendering of the (i + 3)-th frame fails to start in time at ④, resulting in the failure to render the (i + 3)-th frame in time and causing the loss of the (i + 3)-th frame. To reduce frame loss, a load-based resource scheduling algorithm is generally adopted currently. Based on this algorithm, it is necessary to continuously collect the load information for a period of time (such as the time period from ⑤ to ⑥ in the figure) and statistically analyze the load change situation during this period. If the statistical result indicates that the load continuously increases and is greater than the load threshold during this period, then the frequencies of CPU4, CPU5, and CPU6 are increased at time point ⑥ to raise the frequency to 2.5 GHz. However, this scheduling algorithm only increases the frequency after passing ④ and only pulls the frequency point to 2.5 GHz until time point ⑥, but frame loss has already occurred at this time. Therefore, this scheduling algorithm cannot predict frame loss in time and cannot increase the frequency in time.

[0074] Therefore, the load-based resource scheduling algorithm has the following problems: 1. The frequency increase timing is too late, and the probability of saving the current frame is not high, resulting in poor effect of improving the frame loss phenomenon. 2. When increasing the frequency, the frequency point is generally raised to the highest frequency point, which is not friendly to power consumption.

[0075] Regarding the above problem 1, that is, the load-based resource scheduling algorithm has the problem that the frequency increase timing is too late, the probability of saving the current frame is not high, and the effect of improving the frame loss phenomenon is poor. In the embodiments of this application, during the operation of the application, the residence duration of the latest synthesized image frame (which can also be simply referred to as a frame) of the application can be obtained, and according to this residence duration and the target duration, the processor running the rendering thread of the application is frequency-modulated. Among them, the frame data of the image frame is generally cached in the graphics buffer first after being rendered, and then is taken out from the graphics buffer for composition and display. The residence duration refers to the caching duration of the frame data of the image frame in the graphics buffer after being rendered. The target duration is the overall residence duration of the image frame when the frame rate of the application is stable, that is, the stable value of the residence duration.

[0076] When the frame rate is stable, the dwell time of each image frame should be the target time, or not differ much from the target time. By comparing the dwell time with the target time, it is possible to timely sense the changes in the workload and production speed of the image producer, and thus be able to predict frame loss situations timely and accurately. Therefore, according to the comparison result of the dwell time and the target time, the processor of the rendering thread running the application is frequency modulated, so that frequency modulation can be performed in a timely manner when frame loss may occur, thereby reducing or avoiding the probability of frame loss, improving the effect of improving random frame loss phenomenon, and further reducing or avoiding the phenomenon of application screen freezing, thus enhancing the user's application experience.

[0077] As an example, if the dwell time of the latest synthesized image frame is less than the target time, the frequency of the processor where the rendering thread of the application is located is increased, that is, the frequency (also called the working frequency) of the processor where the rendering thread is located is increased, so that the processing performance of the processor can be timely improved, and further the production speed of the image producer can be increased to prevent frame loss. On the contrary, if the dwell time of the latest synthesized image frame is greater than the target time, the frequency of the processor where the rendering thread of the application is located is decreased, that is, the frequency of the processor where the rendering thread is located is decreased, to reduce the production speed of the image producer and save power consumption.

[0078] Furthermore, in order to avoid frequent frequency modulation of the processor and save power consumption, a time difference tolerance can also be preset. Before performing frequency modulation, the time difference between the dwell time and the target time is first compared with the time difference tolerance. If the time difference between the two is less than or equal to the time difference tolerance, the processor where the rendering thread of the application is located is not frequency modulated. If the time difference between the two is greater than the time difference tolerance, then the sizes of the dwell time and the target time are compared. If the dwell time of the latest synthesized image frame is less than the target time, the frequency of the processor where the rendering thread of the application is located is increased; if the dwell time of the latest synthesized image frame is greater than the target time, the frequency of the processor where the rendering thread of the application is located is decreased.

[0079] When the frame rate is stable, the image producer (rendering thread) and the image consumer (SurfaceFlinger) work at a relatively consistent pace. After a relatively stable period of time has passed since a buffer is produced, this buffer is consumed, that is, the residence duration of each frame produced by the image producer is also relatively stable. If the residence duration of a certain frame produced by the image producer becomes shorter or longer, it means that the working rhythms of the image producer (rendering thread) and the image consumer (SurfaceFlinger) have changed. Considering that the workload of the image consumer (SurfaceFlinger) is relatively low and it works under an accurate vsync clock, it generally does not affect the change in the above supply-demand relationship. Therefore, it can be inferred that the main factor causing the above change is the change in the workload of the image producer (rendering thread), resulting in a change in the production speed. For example, if the residence duration of the frame produced by the image producer in the graphics buffer becomes shorter, it means that the workload of the image producer at this time has increased and the production speed has slowed down, which leads to the shorter residence duration of the produced frame in the graphics buffer. At this time, by increasing the frequency of the processor where the rendering thread of the application is located, the processing performance of the processor can be improved in a timely manner, thereby increasing the production speed of the image producer and preventing frame drops. Another example is that if the residence duration of the frame produced by the image producer in the graphics buffer becomes longer, it means that the workload of the image producer at this time has decreased and the production speed has increased, which leads to the shorter residence duration of the produced frame in the graphics buffer. At this time, by decreasing the frequency of the processor where the rendering thread of the application is located, the production speed of the image producer can be reduced and power consumption can be saved.

[0080] It should be understood that the processor where the rendering thread of the application is located may include one or more processors, that is, the rendering thread of the game application may be run by one or more processors. As Figure 5 shown, the processor where the rendering thread of the game application is located may include CPU4, CPU5, and CPU6.

[0081] As an example, frequency modulation of the processor running the rendering thread of the application includes: frequency modulation of the processor running the key tasks of the rendering thread of the application, that is, frequency modulation of the processor where the key tasks of the rendering thread of the application are located.

[0082] Please refer to Figure 6 , Figure 6 is a schematic diagram of the residence duration of an image frame provided by an embodiment of the present application. As Figure 6 shown, the black rectangle in the figure represents the state where a frame of data is cached in the graphic buffer after rendering, and its length represents the residence duration of the frame data in the graphic buffer.

[0083] Please refer to Figure 7 ,Figure 7 Another trace graph of the synthesis process of SurfaceFlinger provided by an embodiment of the present application shows the logs of key points during the image consumption process of SurfaceFlinger. SurfaceFlinger can obtain cached image data from the graphic buffer and consume the obtained image data (synthesize and display). As Figure 7 shown, the black rectangle in the figure represents the state of a frame of data cached in the graphic buffer, and its length represents the residence duration of the frame data in the graphic buffer. The squares vertically stacked after the black rectangle represent the methods called by SurfaceFlinger to read the frame data from the graphic buffer and consume the frame data (synthesize, display, etc.). As Figure 7 shown, the point at ⑨ is the frame loss point, and the point at ⑩ is the frequency increase point determined by the load-based resource scheduling algorithm. Obviously, it is too late to increase the frequency at ⑩ to produce the corresponding frame data at ⑨, that is, it is too late to save the frame loss situation, and the frequency increase timing is severely lagged.

[0084] In an embodiment of the present application, the residence duration of each frame produced by the image producer can be monitored, and the frequency can be adjusted according to the change of the residence duration. As Figure 7 shown, the residence duration of the frame data at ①-⑤ in the graphic buffer is about 8 ms, which is relatively stable. This shows that at ①-⑤, the working paces of the image producer (rendering thread) and the image consumer (SurfaceFlinger) are relatively consistent. After producing a buffer, this buffer is consumed after a relatively stable duration, the load of the rendering process is relatively uniform, the resource configuration is reasonable, and there is no need to adjust the frequency. As Figure 7 shown, compared with the residence duration at ①-⑤, the residence duration of the frame data at ⑥⑦⑧ is significantly shorter, indicating that the load of the image producer (rendering thread) increases and the production speed slows down, resulting in a significant shortening of the residence duration of the frame data. In an embodiment of the present application, when it is detected that the residence duration of the frame data at ⑥ is significantly shortened (for example, the duration difference between the residence duration and the target duration is greater than the duration difference tolerance and the residence duration is less than the target duration), the CPU where the rendering thread is located (such as CPU4) can be increased in frequency to prevent frame loss. As Figure 7 shown, compared with the solution using the load-based resource scheduling algorithm, the solution using frequency adjustment based on the residence duration of frame data in the present application can adjust the frequency several frame lengths in advance, increase the production speed of the image producer in advance, retrieve the unbalanced time, and improve the probability of preventing subsequent frame loss.

[0085] The frequency adjustment algorithm involved in an embodiment of the present application will be described in detail below.

[0086] Please refer toFigure 8 , Figure 8 is a flowchart of a method for adjusting the frequency of a processor based on the frame residence duration provided by an embodiment of the present application. The execution subject of this method is an electronic device. As Figure 8 shown, this method includes:

[0087] 1. Set the target duration T.

[0088] Among them, the target duration T refers to the stable duration corresponding to the residence duration, that is, in the state where the frame rate is stable, the residence duration of each frame generally maintains around the target duration T. Among them, the state where the frame rate is stable refers to the state where the application does not have dropped frames and the jitter rate of the frame rate is less than the preset jitter rate. The preset jitter rate can be set as needed, for example, it can be the maximum jitter rate that does not affect the display effect.

[0089] As an example, the residence durations of multiple historical frames rendered by the application in the state where the frame rate is stable can be obtained, and then the target duration can be determined according to the residence durations of the multiple historical frames. For example, the average residence duration of the multiple historical frames is determined as the target duration.

[0090] As an example, the target duration T matches the frame rate of the application. In the embodiments of the present application, in order to ensure the accuracy of frequency modulation, different target durations T can be set respectively in different frame rate scenarios. For example, in the scenario where the frame rate of the application is 60fps, the target duration T = 8.3ms. In the scenario where the frame rate of the application is 90fps, the target duration T = 5.5ms.

[0091] In a possible implementation manner, the corresponding relationship between the frame rate and the target duration can be set in advance. This corresponding relationship includes multiple frame rates and the target durations corresponding to different frame rates. When setting the target duration T, the target duration corresponding to the frame rate of the current application can be determined from this corresponding relationship. Among them, the target duration corresponding to each frame rate can be determined according to the residence durations of multiple historical frames rendered by the application at the corresponding frame rate and in the state where the frame rate is stable, for example, determined according to the average residence duration of the multiple historical frames.

[0092] 2. During the running of the application, monitor the residence duration t of each frame and use a negative feedback algorithm for frequency modulation. The frequency modulation process includes:

[0093] a) For the monitored residence duration t of the current frame, determine whether |t - T| is less than the duration difference tolerance.

[0094] b) If |t - T| is less than the duration difference tolerance, it means that the residence duration of the current frame tends to be stable, then keep the frequency of the processor where the rendering thread of the application is located unchanged, that is, do not perform frequency modulation on the processor where the rendering thread of the application is located.

[0095] c) If |t - T| is greater than or equal to the duration difference tolerance, then determine whether t is greater than T.

[0096] d) If t is less than T, it means that the production speed of the image producer (the rendering thread of the application) is slow. Then, boost the frequency of the processor where the rendering thread of the application is located to appropriately increase the resource supply to the image producer.

[0097] After boosting the frequency of the processor where the rendering thread of the application is located, on the one hand, the production speed of the image producer decreases, and the residence duration t of the frame becomes longer. On the other hand, continue to monitor the residence duration t of each frame and repeat the above steps a)-d) until |t - T| < the duration difference tolerance.

[0098] Furthermore, considering that the overhead caused by frequent frequency modulation is relatively large, in order to save power consumption and avoid drastic fluctuations (sharp rises and falls) in frequency, when adjusting the frequency of the processor where the rendering thread of the application is located, the frequency of the processor where the rendering thread of the application is located can be gradually adjusted. In this way, the problem that the frequency point is raised to the highest frequency point during frequency boosting in the load-based resource scheduling algorithm, which is not friendly to power consumption, can be improved.

[0099] As an example, when boosting the frequency of the processor where the rendering thread of the application is located, the frequency can be gradually increased. For example, the frequency value of the processor where the rendering thread is located can be first increased by the first frequency value. Then, continue to obtain the residence duration t of the next frame synthesized by the application. If the residence duration t of the next frame is less than the target duration T, then continue to increase the frequency value of the processor where the rendering thread is located by the first frequency value until the residence duration t of the latest frame synthesized by the application satisfies |t - T| < the duration difference tolerance.

[0100] Among them, the first frequency value can be set in advance according to needs. For example, the first frequency value can be 200KHz or 400KHz, etc.

[0101] In a possible implementation manner, the frequency value of the processor where the rendering thread is located is the frequency point. In this case, the frequency point of the processor where the rendering thread is located can be first increased by m frequency points. Then, continue to obtain the residence duration t of the next frame synthesized by the application. If the residence duration t of the next frame is less than the target duration T, then continue to increase the frequency value of the processor where the rendering thread is located by m frequency points until the residence duration t of the latest frame synthesized by the application satisfies |t - T| < the duration difference tolerance. Here, m is a positive integer.

[0102] e) If t is greater than T, it means that the production speed of the image producer (the rendering thread of the application) is fast. Then, lower the frequency of the processor where the rendering thread of the application is located to appropriately reduce the resource supply to the image producer.

[0103] As an example, the processor where the rendering thread of the application is located can be gradually downclocked. For example, the frequency value of the processor where the rendering thread is located can be first reduced by a second frequency value, and then the residence duration t of the next frame synthesized by the application is continuously obtained. If the residence duration t of the next frame is greater than the target duration T, the frequency value of the processor where the rendering thread is located is continuously reduced by the second frequency value until the residence duration t of the frame synthesized most recently by the application satisfies |t - T| < duration difference tolerance.

[0104] Among them, the second frequency value can be set in advance according to needs. For example, the second frequency value can be 200KHz or 400KHz, etc.

[0105] In a possible implementation manner, the frequency value of the processor where the rendering thread is located is a frequency point. In this case, the frequency point of the processor where the rendering thread is located can be first increased by n frequency points, and then the residence duration t of the next frame synthesized by the application is continuously obtained. If the residence duration t of the next frame is less than the target duration T, the frequency value of the processor where the rendering thread is located is continuously increased by n frequency points until the residence duration t of the frame synthesized most recently by the application satisfies |t - T| < duration difference tolerance. Among them, n is a positive integer.

[0106] f) Repeat the above actions to keep the rendering system in a state where |t - T| < duration difference tolerance through positive and negative frequency modulation.

[0107] In the embodiments of the present application, by performing frequency modulation on the processor based on the frame residence duration, the frame residence duration can be adjusted to a stable interval. On the one hand, it can prevent the frame residence duration from being too short to cause frame loss, thus improving the frame loss situation; on the other hand, it can prevent the frame residence duration from being too long to cause buffer accumulation and excessive supply of the processor frequency point, thus saving power consumption.

[0108] The electronic device provided in the embodiments of the present application can be a user equipment (UE), for example, it can be a mobile terminal (such as a user's mobile phone), a tablet computer, a desktop type, a laptop computer, a handheld computer, a netbook, a personal digital assistant (PDA), etc.

[0109] In addition, an operating system runs on the above components. For example, the iOS operating system developed by Apple Inc., the Android open-source operating system developed by Google Inc., the Windows operating system developed by Microsoft Corporation, etc. Application programs can be installed and run on this operating system.

[0110] The operating system of an electronic device can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of this application, taking the Android system with a layered architecture as an example, the software structure of the electronic device is illustratively described.

[0111] Figure 9 It is a block diagram of a software system of an electronic device provided by an embodiment of this application. Refer to Figure 9 , the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely the application layer (also called the app layer), the application framework (Framework) layer, the native service (Native) layer, and the kernel (Kernal) layer.

[0112] The application layer can include a series of application packages. As Figure 9 shown, the application packages can include application programs that can display video images, such as game applications, video applications, video call applications, etc. Of course, the application packages can also include other application programs, such as camera, gallery, calendar, call, map, navigation, Bluetooth, music, video, short message and other application programs.

[0113] The application framework layer provides application programming interfaces (application programming interface, API) and programming frameworks for the application programs in the application layer. The application framework layer includes some predefined functions. As Figure 4 shown, the application framework layer can include a view system. The view system is used to draw the display screen of the application program. For example, the rendering thread of a game application can call the view system of the application framework layer, and through the view system, render the image frames to be displayed for the application. After the rendering is completed, the rendered image frames are cached in the graphics buffer.

[0114] In addition, the application framework layer may further include a window manager, a content provider, a telephone manager, a resource manager, a notification manager, etc. The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data, and enable these data to be accessed by application programs. These data may include videos, images, audio, dialed and received calls, browsing history and bookmarks, phone books, etc. The telephone manager is used to provide the communication function of the electronic device 100, such as the management of call states (including connection, hanging up, etc.). The resource manager provides various resources for application programs, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables application programs to display notification information in the status bar. It can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction. For example, the notification manager can be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as a notification of a background-running application program.

[0115] The native service layer mainly includes some native services and link libraries. The native service layer can provide feature support for the Android system through some C / C++ libraries. In addition, the native service layer also includes the Android Runtime. The Android Runtime includes a core library and a virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system. The core library contains two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core library of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0116] Please refer to Figure 9 , the native service layer includes SurfaceFlinger and a frequency modulation module. The frequency modulation module includes an algorithm parameter parsing and configuration module, a frame buffer production and consumption monitoring module, a frame buffer residence duration statistics module, a frequency adjustment feedback module, and a CPU frequency adjustment interface.

[0117] SurfaceFlinger is used to consume the image frames rendered by the rendering thread of the application through the view system. For example, SurfaceFlinger can take out the image frames from the graphic buffer, and perform synthesis and display of the taken-out image frames, etc. It should be understood that Figure 9 only taking SurfaceFlinger as an image consumer as an example for illustration. In other embodiments, the image consumer may also be other image production modules, and the embodiments of the present application do not limit this.

[0118] The frame buffer production and consumption monitoring module is used to monitor the time when the view system stores the rendered image frame in the graphics buffer, and the time when SurfaceFlinger retrieves this image frame from the graphics buffer. That is, the frame buffer production and consumption monitoring module can monitor the storage time and retrieval time of each frame in the graphics buffer, and send the monitored storage time and retrieval time of each frame in the graphics buffer to the frame buffer residence duration statistics module. The duration between the storage time and the retrieval time of any image frame in the graphics buffer is the residence duration of the image frame.

[0119] The frame buffer residence duration statistics module is used to calculate the residence duration of each frame according to the storage time and retrieval time of each frame in the graphics buffer monitored by the frame buffer production and consumption monitoring module.

[0120] The frequency adjustment feedback algorithm module is used to adjust the frequency of the processor where the rendering thread of the application is located according to the calculated residence duration and the target duration, according to the method provided in the embodiments of the present application. For example, compare the calculated residence duration with the target duration, and determine whether to adjust the frequency and whether to increase or decrease the frequency according to the comparison result. In addition, the frequency adjustment feedback algorithm module can also call the CPU frequency adjustment interface when it is determined that frequency adjustment is required, and send a frequency adjustment instruction to the CPU frequency adjustment driver in the kernel layer through the CPU frequency adjustment interface.

[0121] The CPU frequency adjustment interface is an interface between the frequency adjustment module and the CPU frequency adjustment driver in the kernel layer, and is used to implement data transmission between the frequency adjustment module and the CPU frequency adjustment.

[0122] The algorithm parameter parsing and configuration module is used to configure relevant parameters for other modules in the frequency adjustment module, such as configuring the target duration and the frequency adjustment algorithm for the frequency adjustment algorithm module.

[0123] The kernel layer is the layer between hardware and software. The kernel layer at least includes a CPU adjustment driver and a GPU driver, etc. The CPU adjustment driver is used to drive the CPU to adjust the frequency.

[0124] It should be noted that Figure 9 The layers shown in the software structure and the components included in each layer do not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more layers than shown, such as a system library (FWK LIB) layer and a kernel layer. In addition, each layer may include more or fewer components than shown, and the present application does not make a limitation.

[0125] It can be understood that in order for an electronic device to implement the frequency modulation method of the processor in the embodiments of the present application, it includes the corresponding hardware and / or software modules for performing various functions. Combining the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments.

[0126] It should be noted that although the embodiments of the present application are described by taking the Android system as an example, the basic principle also applies to electronic devices based on operating systems such as iOS or Windows.

[0127] Next, in combination with a specific game scenario and the above software architecture, a brief overview of the frequency modulation method of the processor provided by the embodiments of the present application is given.

[0128] After the game application is started, the game application creates a rendering thread and executes the rendering process through the rendering thread, that is, sequentially renders the image frames to be displayed by the game application and stores the rendered image frames in the graphics buffer. For the image frames stored in the graphics buffer, SurfaceFlinger can read the cached image frames from the graphics buffer, synthesize and display the read image frames, so that the display screen displays the image frames. During the rendering process of the rendering thread, the frame buffer production and consumption monitoring module can monitor the buffer storage time of each frame completed by the rendering thread, that is, the time when the frame data of each frame is stored in the graphics buffer after being rendered, and send the monitored buffer storage time of the image frame to the frame buffer residence duration statistics module. During the synthesis process of SurfaceFlinger, the frame buffer production and consumption monitoring module can monitor the buffer extraction time of each frame synthesized by SurfaceFlinger, that is, the time when the frame data of each frame is extracted from the buffer, and send the monitored buffer extraction time of the image frame to the frame buffer residence duration statistics module. The frame buffer residence duration statistics module can determine the residence duration of the image frame according to the buffer storage time and buffer extraction time of the latest synthesized image frame, and send the residence duration of the image frame to the frequency adjustment feedback algorithm module. The frequency adjustment feedback algorithm module can judge whether frequency modulation (frequency increase or frequency decrease) is required according to the comparison result of the residence duration and the target duration, and send a frequency modulation instruction to the CPU frequency modulation driver through the CPU frequency adjustment interface when frequency modulation is required to control the CPU where the rendering thread is located to perform frequency modulation.

[0129] The execution subject of the video processing method provided in the embodiments of this application can be the above-mentioned electronic device, or a functional module and / or functional entity in the electronic device that can implement the video processing method. Moreover, the solution of this application can be implemented in a hardware and / or software manner, and specifically can be determined according to actual usage requirements, which are not limited in the embodiments of this application. Hereinafter, taking the electronic device as an example, the video processing method provided in the embodiments of this application will be described exemplarily with reference to the accompanying drawings.

[0130] The following will describe the specific implementation process of the frequency modulation method of the processor provided in the embodiments of this application with reference to the accompanying drawings.

[0131] Figure 10 is a schematic flowchart of a frequency modulation method of a processor provided in the embodiments of this application, and this method can be executed by an electronic device. The software structure block diagram of the electronic device can be as Figure 9 shown, but the embodiments of this application are not limited thereto. As Figure 10 shown, this method includes the following steps:

[0132] Step 101: The user performs a first operation.

[0133] Among them, the first operation is used to start the game application. For example, the first operation is the operation of the user clicking on the icon of the game application.

[0134] Step 102: In response to the first operation, the game application starts.

[0135] In the embodiments of this application, after the game application is started, in order to reduce the frame drop probability of the game application and save power consumption, the frequency modulation module can monitor the residence duration of each frame rendered by the game application and perform frequency modulation on the processor according to the monitored residence duration.

[0136] Before performing frequency modulation through the frequency modulation module, it is generally necessary to initialize the frequency modulation module, such as initializing the relevant algorithms and parameters of the frequency modulation module. Exemplarily, the specific initialization process can be implemented through the following steps 103-step 105.

[0137] Step 103: The game application sends a startup notification of the game application to the algorithm parameter parsing and configuration module of the frequency modulation module.

[0138] Among them, the startup notification is used to indicate to the frequency modulation module that the game application has started.

[0139] It should be noted that the embodiments of this application only take the game application sending a startup notification to the frequency modulation module to inform the start of the game application as an example for illustration. It should be understood that the frequency modulation module can also use other methods to learn that the game application has started, such as learning the start of the game application through relevant interfaces, and the embodiments of this application do not limit this.

[0140] Step 104: The algorithm parameter parsing and configuration module determines the initialization configuration information, which at least includes the target duration T and the duration difference tolerance tolerance.

[0141] As an example, the algorithm parameter parsing and configuration module can determine the target duration T corresponding to the residence duration of the frame buffer of the game application according to the frame rate of the game application.

[0142] In a possible implementation, the startup notification sent by the game application carries the identifier and frame rate of the game application, and the algorithm parameter parsing and configuration module can obtain the frame rate of the game application according to the startup notification. Of course, the algorithm parameter parsing and configuration module can also obtain the frame rate of the game application through other means, such as obtaining the frame rate of the game application by calling relevant interfaces. The embodiments of the present application do not limit this.

[0143] In a possible implementation, after the algorithm parameter parsing and configuration module determines the frame rate of the game application, it can obtain the target duration T corresponding to the frame rate of the game application from the correspondence table between the frame rate and the target duration. The correspondence table stores multiple frame rates and the target durations corresponding to each frame rate.

[0144] It should be understood that the correspondence table and the duration difference tolerance tolerance can be pre-stored and obtained by the algorithm parameter parsing and configuration module itself, or can be obtained by the algorithm parameter parsing and configuration module through other means, such as obtaining by calling relevant interfaces. The embodiments of the present application do not limit the manner of obtaining the correspondence table and the duration difference tolerance tolerance.

[0145] It should also be understood that the initialization configuration information may further include other configuration information, such as a specific frequency adjustment feedback algorithm, etc. The embodiments of the present application do not limit this.

[0146] Step 105: The algorithm parameter parsing and configuration module initializes other modules in the frequency modulation module according to the initialization configuration information.

[0147] As an example, the initialization operation at least includes: the algorithm parameter parsing and configuration module sends parameter configuration information such as the target duration T and the duration difference tolerance tolerance to the frequency adjustment feedback algorithm module, so that the frequency adjustment feedback algorithm module initializes relevant parameters according to the parameter configuration information.

[0148] It should be understood that the algorithm parameter parsing and configuration module can also send the corresponding initialization configuration information to other modules other than the frequency adjustment feedback algorithm module to perform initialization configuration on other modules. The embodiments of the present application will not list them one by one here.

[0149] Step 106: The game application creates a rendering thread.

[0150] After the game application is launched, a rendering thread can be created to render the application screen to be displayed through the rendering thread.

[0151] Step 107: The rendering thread of the game application starts to execute the rendering process.

[0152] Among them, the rendering process refers to the process of rendering image frames, and the image frames can be rendered in sequence according to the rendering logic. The rendering process of each image frame can include: obtaining a graphic buffer from the buffer queue, writing the frame data of the rendered image frame into the graphic buffer, and then submitting the graphic buffer to the buffer queue. The following will be illustrated through Step 108 - Step 109.

[0153] As an example, the rendering thread of the game application can perform rendering through the view system of the application framework layer. For example, the rendering thread can send a rendering instruction to the view system. After receiving the rendering instruction, the view system starts to execute the rendering process according to the rendering instruction to render the image frames.

[0154] Step 108: The rendering thread obtains graphic buffer 1 (hereinafter referred to as buffer 1) from the buffer queue.

[0155] For example, the rendering thread can obtain graphic buffer 1 from the buffer queue by calling the dequeue buffer method.

[0156] Step 109: The rendering thread writes the frame data of the rendered image frame 1 into buffer 1.

[0157] Step 110: The rendering thread submits buffer 1 to the buffer queue.

[0158] For example, the rendering thread can submit buffer 1 to the buffer queue by calling the queue buffer method, that is, return buffer 1 to the buffer queue to make buffer 1 re-enter the queue.

[0159] In addition, after receiving the in-queue message of buffer 1, the buffer queue can also send an in-queue notification to Surfaceflinger. This in-queue notification is used to notify Surfaceflinger that there is a ready graphic buffer for Surfaceflinger to consume.

[0160] Step 111: When the Vsync signal arrives, Surfaceflinger fetches buffer 1 from the buffer queue.

[0161] For example, Surfaceflinger can obtain buffer 1 from the buffer queue by calling the acquire buffer method.

[0162] Step 112: Surfaceflinger reads the frame data from buffer 1.

[0163] In addition, after Surfaceflinger reads the cached frame data from buffer 1, it can also submit buffer 1 to the buffer queue by calling the release buffer method, that is, return the graphic buffer 1 to the buffer queue, so that the graphic buffer 1 can be reused.

[0164] Step 113: Surfaceflinger synthesizes and displays the read frame data, so that the display screen shows image frame 1.

[0165] As an example, Surfaceflinger can synthesize and display the frame data through HWC. For example, Surfaceflinger can send the read frame data to HWC, and HWC synthesizes and displays the received frame data.

[0166] Step 114: During the rendering process of the rendering thread, the frame buffer production and consumption monitoring module monitors the buffer storage time of each frame completed by the rendering thread.

[0167] Among them, the buffer storage time of each frame refers to the time when the frame data of each frame is stored in the buffer after rendering. Specifically, it can be the time when the rendering process writes the frame data of each rendered frame into the graphic buffer, or the time when the graphic buffer is submitted to the bufferqueue after the frame data of each rendered frame is written into the graphic buffer. The embodiments of this application do not limit this.

[0168] Step 115: The frame buffer production and consumption monitoring module sends the monitored buffer storage time 1 of image frame 1 to the frame buffer residence duration statistics module.

[0169] For example, when the frame buffer production and consumption monitoring module monitors that the frame data of image frame 1 is stored in the buffer, it can determine the buffer storage time of image frame 1 and send the buffer storage time 1 of image frame 1 to the frame buffer residence duration statistics module.

[0170] Step 116: During the composition process of Surfaceflinger, the frame buffer production and consumption monitoring module monitors the buffer extraction time of each frame synthesized by Surfaceflinger.

[0171] Among them, the buffer extraction time of each frame refers to the time when the frame data of each frame is extracted from the buffer. Specifically, it can be the time when the rendering process obtains the graphic buffer where the frame data of each frame is located from the buffer queue, or it can be the time when the frame data of each frame is read from the graphic buffer after obtaining the graphic buffer where the frame data of each frame is located. The embodiments of the present application do not limit this.

[0172] Step 117: The frame buffer production and consumption monitoring module sends the monitored buffer extraction time 1 of image frame 1 to the frame buffer residence duration statistics module.

[0173] Step 118: The frame buffer residence duration statistics module determines the residence duration t of image frame 1 according to the buffer storage time 1 and buffer extraction time 1 of image frame 1.

[0174] The frame buffer residence duration can statistically calculate the residence duration of each frame according to the buffer storage time and buffer extraction time of each frame monitored by the frame buffer production and consumption monitoring module. The residence duration of each frame is the duration between the buffer storage time and buffer extraction time of each frame, that is, the cache duration of each frame in the graphic buffer.

[0175] For example, after receiving the buffer extraction time 1 of image frame 1 monitored by the frame buffer production and consumption monitoring module, the frame buffer residence duration statistics module can determine the residence duration t of image frame 1 according to the buffer storage time 1 and buffer extraction time 1 of image frame 1.

[0176] Step 119: The frame buffer residence duration statistics module sends the residence duration t of image frame 1 to the frequency adjustment feedback algorithm module.

[0177] Step 120: The frequency adjustment feedback algorithm module compares |t - T| with the duration difference tolerance tolerance.

[0178] After receiving the residence duration t of image frame 1, the frequency adjustment feedback algorithm module can first compare |t - T| with the duration difference tolerance tolerance to determine whether |t - T| is less than the duration difference tolerance tolerance.

[0179] Step 121: When |t - T| < tolerance, the frequency adjustment feedback algorithm does not perform frequency modulation.

[0180] When |t - T| < tolerance, it indicates that the dwell time t is not much different from the target time T and is relatively stable. Therefore, frequency modulation operations can be avoided to keep the frequency of the CPU where the rendering thread of the game application is located unchanged.

[0181] In the embodiments of this application, by introducing the variable time difference tolerance tolerance, when |t - T| < tolerance, no frequency modulation is performed, which can reduce the overhead caused by frequent frequency modulation and save power consumption.

[0182] Step 122: When |t - T| > tolerance and t < T, the frequency adjustment feedback algorithm generates a frequency increase instruction.

[0183] When |t - T| > tolerance, it indicates that the dwell time t is quite different from the target time T, and the load and production speed of Surfaceflinger are unstable, so frequency modulation is required. Therefore, at this time, the dwell time t can continue to be compared with the target time T, and whether to increase or decrease the frequency is determined according to the comparison result of the dwell time t and the target time T.

[0184] For example, if t < T, it means that the load of Surfaceflinger increases and the production speed slows down, and the resource supply of Surfaceflinger does not meet the load demand. Therefore, at this time, a frequency increase instruction can be generated to increase the frequency of the CPU where the rendering thread is located through the frequency increase instruction to appropriately increase the resource supply of Surfaceflinger.

[0185] Step 123: The frequency adjustment feedback algorithm sends the frequency increase instruction to the CPU frequency modulation driver in the kernel layer through the CPU frequency adjustment interface.

[0186] The frequency adjustment feedback algorithm sends a frequency increase instruction to the CPU frequency modulation driver through the CPU frequency adjustment interface to instruct the CPU frequency modulation driver to control the CPU where the rendering thread is located to increase the frequency.

[0187] For example, the frequency adjustment feedback algorithm first sends the frequency increase instruction to the CPU frequency adjustment interface, and the CPU frequency adjustment interface forwards the frequency increase instruction to the CPU frequency modulation driver.

[0188] Step 124: The CPU frequency modulation driver sends a frequency increase driver instruction to the CPU where the rendering thread of the game application is located according to the frequency increase instruction.

[0189] Among them, the frequency boosting drive instruction is used to drive the corresponding CPU to boost the frequency.

[0190] Among them, the CPUs where the rendering threads of the game application are located can include one or more CPUs. That is, the rendering threads of the game application can be run by one or more CPUs. As Figure 5 shown, the CPUs where the rendering threads of the game application are located can include CPU4, CPU5, and CPU6.

[0191] Step 125: The CPU increases the frequency point according to the frequency boosting drive instruction.

[0192] After receiving the frequency boosting drive instruction, the CPU can boost the frequency according to the instruction. For example, the frequency point where it is located can be increased by m frequency points. Among them, m is a positive integer and can be set as needed. For example, it can be 1, 2, or 3, etc. The difference between two adjacent frequency points is a fixed frequency. For example, the difference is 200 KHz.

[0193] Exemplarily, assume m is 1, the CPU where the rendering thread is located includes CPU4, and the original frequency point of CPU4 is 2 (corresponding frequency band is 890.2 MHz - 890.2 MHz). Then, according to the frequency boosting drive instruction, CPU4 can increase the frequency point where it is located by 1 frequency point, that is, adjust the frequency point where it is located from 2 to 3 (corresponding frequency band is 890.4 MHz - 890.6 MHz).

[0194] Step 126: When |t - T| > tolerance and t > T, the frequency adjustment feedback algorithm generates a frequency reduction instruction.

[0195] If t > T, it means that the load of Surfaceflinger becomes smaller, the production speed becomes faster, and the resource supply of Surfaceflinger exceeds the load demand. Therefore, at this time, a frequency reduction instruction can be generated to reduce the frequency of the CPU where the rendering thread is located through the frequency reduction instruction, so as to appropriately reduce the resource supply of Surfaceflinger and thus reduce power consumption.

[0196] Step 127: The frequency adjustment feedback algorithm sends the frequency reduction instruction to the CPU frequency modulation drive in the kernel layer through the CPU frequency adjustment interface.

[0197] The frequency adjustment feedback algorithm sends a frequency reduction instruction to the CPU frequency modulation drive through the CPU frequency adjustment interface to instruct the CPU frequency modulation drive to control the CPU where the rendering thread is located to reduce the frequency.

[0198] For example, the frequency adjustment feedback algorithm first sends the frequency reduction instruction to the CPU frequency adjustment interface, and the CPU frequency adjustment interface forwards the frequency reduction instruction to the CPU frequency modulation drive.

[0199] Step 128: The CPU frequency modulation driver sends a frequency reduction driver instruction to the CPU where the rendering thread is located according to the frequency reduction instruction.

[0200] Among them, the frequency increase driver instruction is used to drive the corresponding CPU to reduce the frequency.

[0201] Step 129: The CPU reduces the frequency point according to the frequency reduction driver instruction.

[0202] After receiving the frequency reduction driver instruction, the CPU can reduce the frequency according to the frequency reduction driver instruction. For example, the frequency point where it is located can be reduced by n frequency points. Among them, n is a positive integer and can be set according to needs. For example, it can be 1, 2, or 3, etc. The difference between two adjacent frequency points is a fixed frequency, for example, the difference is 200KHz.

[0203] Exemplarily, assuming n is 1, the CPU where the rendering thread is located includes CPU4, and the original frequency point of CPU4 is 2 (corresponding frequency band is 890.2MHz - 890.2MHz), then CPU4 can reduce the frequency point where it is located by 1 frequency point according to the frequency reduction driver instruction, that is, adjust the frequency point where it is located from 2 to 1 (corresponding frequency band is 890.0MHz - 890.2MHz).

[0204] In the embodiment of the present application, on the one hand, by increasing the frequency of the processor where the rendering thread of the game application is located when the residence time of the newly synthesized image frame is less than the target time, it is possible to make a prediction of possible frame loss at least one frame in advance, so as to intervene in scheduling more timely, make reasonable frequency point adjustments and other resource configurations, such as timely improving the processing performance of the processor, thereby improving the production speed of the image producer, and thus better reducing the possibility of frame loss in the game application, and further reducing the stuttering phenomenon of the game screen, improving the user's game experience. On the other hand, by reducing the frequency of the processor where the rendering thread of the game application is located when the residence time of the newly synthesized image frame is greater than the target time, that is, reducing the frequency of the processor where the rendering thread is located, it is possible to reasonably configure the frequency points of the processor, not too radical, and try to ensure the smoothness of the frame residence time, thereby improving power consumption and performance and saving power consumption.

[0205] The electronic device provided by the embodiment of the present application can be a user equipment (UE), for example, it can be a mobile electronic device (such as a user's mobile phone), a tablet computer, a desktop type, a laptop computer, a handheld computer, a netbook, a personal digital assistant (PDA), etc.

[0206] Next, the electronic device involved in the embodiment of the present application will be described.

[0207] Figure 11It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Refer to Figure 11 , the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0208] It can be understood that the structure schematically shown in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0209] The processor 110 may include one or more processing units. For example: the processor 110 may include an application processor (AP), a modulation and demodulation processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0210] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0211] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0212] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0213] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0214] The antenna 1 and the antenna 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0215] The mobile communication module 150 may provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 may receive electromagnetic waves through the antenna 1, filter, amplify, and process the received electromagnetic waves, and then transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 may also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least some functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be disposed in the same device.

[0216] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modulation and demodulation processor may be an independent device. In other embodiments, the modulation and demodulation processor may be independent of the processor 110 and disposed in the same device as the mobile communication module 150 or other functional modules.

[0217] The wireless communication module 160 may provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 for radiation.

[0218] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with a network and other devices via wireless communication technologies. The wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0219] Electronic device 100 implements display functions through the GPU, display screen 194, and application processor, etc. The GPU is a microprocessor for image processing, connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0220] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is an integer greater than 1.

[0221] The electronic device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, etc.

[0222] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light passes through the lens and is transmitted to the camera's photosensitive element. The optical signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise, brightness, and skin color of the image through algorithms. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0223] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, etc. format. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is an integer greater than 1.

[0224] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0225] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0226] The NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between human brain neurons, it can quickly process input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0227] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to achieve the data storage function. For example, files such as music and videos are saved in the external memory card.

[0228] The internal memory 121 can be used to store computer-executable program code, and the computer-executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a storage program area and a storage data area. Among them, the storage program area can store the operating system, application programs required for at least one function (such as the sound playback function, image playback function, etc.), etc. The storage data area can store the data created during the use of the electronic device 100 (such as audio data, phone book, etc.), etc. In addition, the internal memory 121 can include high-speed random access memory and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0229] The electronic device 100 can implement audio functions, such as music playback, recording, etc., through the audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone interface 170D, and the application processor, etc.

[0230] The pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor may include at least two parallel plates with conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the pressure threshold acts on the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0231] The touch sensor 180K, also known as the "touch panel". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also known as the "touch display screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100, at a different position from that of the display screen 194.

[0232] This application also provides a chip, which is coupled to a memory. The chip is used to read and execute computer programs or instructions stored in the memory to execute the methods in the above embodiments.

[0233] This application also provides an electronic device, which includes a chip. The chip is used to read and execute computer programs or instructions stored in the memory, so that the methods in the embodiments are executed.

[0234] This embodiment also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions run on an electronic device, the electronic device executes the above-related method steps to implement the video processing method in the above embodiments.

[0235] This embodiment also provides a computer program product. The computer-readable storage medium stores program codes. When the computer program product runs on a computer, the computer is enabled to execute the above related steps to implement the video processing method in the above embodiment.

[0236] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, a component or a module. The device may include a processor and a memory connected to each other. The memory is used to store computer-executable instructions. When the device runs, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the video processing method in each of the above method embodiments.

[0237] The electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be elaborated here.

[0238] The embodiments of the present application do not particularly limit the specific structure of the execution subject of the method provided in the embodiments of the present application. As long as it can run a program recording the code of the method provided in the embodiments of the present application to perform video processing according to the method provided in the embodiments of the present application. For example, the execution subject of the method provided in the embodiments of the present application may be an electronic device, or a functional module in the electronic device that can call and execute the program.

[0239] In the several embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces. The indirect coupling or communication connection of the device or unit may be in an electrical, mechanical or other form.

[0240] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0241] In addition, in each embodiment of the present application, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0242] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a computer software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium can include, but is not limited to, various media that can store program codes such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs.

[0243] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A frequency modulation method for a processor, characterized in that: Applied to electronic equipment, the method includes: In response to a first operation, starting a first application and displaying an application screen of the first application; During the execution of the first application, obtaining a dwell time of a latest synthesized image frame of the first application, where the dwell time refers to a time from when the image frame is stored in a graphics buffer after rendering is completed to when the image frame is retrieved from the graphics buffer for synthesis; Adjusting the frequency of a target processor running a rendering thread of the first application based on a comparison result of the dwell time and a target time, wherein the target time is determined based on the dwell times of multiple historical frames rendered by the first application in a stable frame rate state, where the stable frame rate state refers to a state in which the first application does not experience frame loss and a frame rate jitter ratio is less than a preset jitter ratio; The adjusting, according to the comparison result of the residence time and the target time, the frequency of the target processor running the rendering thread of the first application includes: If the dwell time is less than the target time, increasing the frequency of the target processor; If the residency duration is greater than the target duration, the frequency of the target processor is reduced.

2. The method according to claim 1, wherein Before adjusting the frequency of the target processor running the rendering thread of the first application according to the comparison result of the residence time and the target time, the method further includes: Determining a time difference between the dwell time and the target time; If the duration difference is greater than or equal to the duration difference tolerance, a step of adjusting the frequency of a target processor running the rendering thread of the first application according to a comparison result of the residence duration and the target duration is performed.

3. The method according to claim 1, wherein Increasing the frequency of the target processor includes: Increasing the frequency value of the target processor by a first frequency value; Continuing to obtain the dwell time of the next image frame synthesized by the first application; If the dwell time of the next image frame is less than the target time, the frequency value of the target processor continues to be increased by the first frequency value until the dwell time of the image frame most recently synthesized by the first application is greater than or equal to the target time.

4. The method according to claim 1, wherein The reducing the frequency of the target processor comprises: reducing the frequency value of the target processor by a second frequency value; Continuing to obtain the dwell time of the next image frame synthesized by the first application; If the dwell time of the next image frame is greater than the target time, the frequency value of the target processor is continuously reduced by the second frequency value until the dwell time of the image frame most recently synthesized by the first application is less than or equal to the target time.

5. The method according to claim 1, wherein The obtaining of the dwell time of the latest synthesized image frame of the first application includes: monitoring a rendering process of a rendering thread of the first application to obtain a time for storing the image frame in the graphics buffer after the rendering thread completes rendering the image frame, thereby obtaining a buffer storage time of the image frame; Monitoring the synthesis process of the image consumer corresponding to the rendering process to obtain the time when the image consumer takes out the image frame from the graphics buffer for synthesis, thereby obtaining the buffer fetching time of the image frame; The residence time of the image frame is determined according to the buffer storage time and the buffer retrieval time of the image frame.

6. The method according to any one of claims 1 to 5, characterized in that: Before adjusting the frequency of the target processor running the rendering thread of the first application according to the comparison result of the residence time and the target time, the method further includes: determining a frame rate of the first application; The target duration corresponding to the frame rate of the first application is determined from a correspondence between frame rates and target durations, where the correspondence includes multiple frame rates and target durations corresponding to different frame rates.

7. The method according to any one of claims 1 to 5, characterized in that: When the frame rate of the first application is 60 fps, the target duration is 8.3 ms; When the frame rate of the first application is 90 fps, the target duration is 5.5 ms.

8. The method according to any one of claims 1 to 5, characterized in that: Before adjusting the frequency of the target processor running the rendering thread of the first application according to the comparison result of the residence time and the target time, the method further includes: Obtaining dwell time of multiple historical frames rendered by the first application in a stable frame rate state, where the stable frame rate state refers to a state in which the first application does not lose frames and a frame rate jitter rate is less than a preset jitter rate; The target duration is determined according to the dwell durations of the multiple historical frames.

9. The method according to any one of claims 1 to 5, wherein: The first application is a game application.

10. An electronic device, characterized in that: The electronic device includes: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method as described in any one of claims 1 to 9.

11. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium comprises instructions, which, when executed on an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 9.

Images