Frequency modulation method, electronic device and storage medium

Abstract

The present application provides a frequency modulation method, electronic device and storage medium. The method is applied to an electronic device in a camera scene. When the electronic device starts drawing the Nth image frame, it calculates the CPU load value from the first moment to the second moment; the first moment is the moment when the N-1th image frame starts drawing, and the second moment is the moment when the N-1th image frame starts drawing; according to the CPU load value and the frame loss mark of the N-1th image frame, the CPU frequency is determined to modulate the CPU frequency. The frequency modulation method uses the image frame drawing start time as the CPU frequency modulation opportunity, and determines the CPU frequency by predicting the CPU load value, so that the CPU frequency is adjusted more appropriately to avoid power consumption waste.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent terminals, and in particular to a frequency modulation method, an electronic device and a storage medium. Background Art

[0002] In today's era, smart terminals such as mobile phones have become an indispensable tool in people's lives. Among them, the camera functions of smart terminals such as mobile phones are also constantly improving, making video recording more convenient and efficient. However, the high power consumption of the camera function causes smart terminals such as mobile phones to consume power quickly and easily heat up, resulting in poor user experience. Summary of the invention

[0003] In order to solve the above technical problems, the present application provides a frequency modulation method, an electronic device and a storage medium. The frequency modulation method modulates the CPU core to a suitable frequency point within a suitable frequency modulation interval, which can not only ensure the camera performance, but also reduce the power consumption of the electronic device in the camera scene.

[0004] In a first aspect, an embodiment of the present application provides a frequency modulation method. The method is applied to an electronic device, and the current scene of the electronic device is implemented based on a camera function. The method includes:

[0005] When the Nth image frame starts to be drawn, the electronic device calculates the CPU load value from the first moment to the second moment; N is an integer greater than 1, the first moment is the moment when the N-1th image frame starts to be drawn, and the second moment is the moment when the N-1th image frame starts to be drawn; the electronic device determines the CPU frequency according to the CPU load value and the frame loss flag of the N-1th image frame; the frame loss flag is used to indicate whether a frame is lost; the electronic device adjusts the CPU frequency according to the CPU frequency.

[0006] The frequency modulation method uses the image frame drawing start time as the CPU frequency modulation opportunity and determines the CPU frequency by predicting the CPU load value, so that the CPU frequency can be adjusted more appropriately and power consumption can be avoided.

[0007] According to the first aspect, the method also includes: when the N+1th image frame starts to be drawn, the electronic device calculates the CPU load value from the second moment to the third moment; the third moment is the moment when the N+1th image frame starts to be drawn; the electronic device determines the CPU frequency based on the CPU load value and the frame loss mark of the Nth image frame; and adjusts the CPU according to the CPU frequency.

[0008] In this way, the CPU frequency modulation interval is aligned with the frame rate of the camera application. Compared with the EAS scheduling method, the frequency modulation interval is reduced, and the CPU frequency is adjusted more appropriately, which not only ensures the camera performance but also reduces the power consumption of the device.

[0009] According to the first aspect, or any implementation method of the first aspect above, the electronic device determines the CPU frequency based on the CPU load value and the frame loss flag of the N-1th image frame, including: the electronic device queries the pre-trained RL model based on the CPU load value and the frame loss flag of the N-1th image frame to obtain the CPU frequency.

[0010] In this embodiment, the electronic device determines the CPU frequency based on the pre-trained RL model, so that the CPU frequency is adjusted more appropriately, avoiding the problem of power consumption wasted due to excessive frequency adjustment strategy.

[0011] According to the first aspect, or any implementation of the first aspect above, the electronic device calculates the CPU load value from the first moment to the second moment, including: the electronic device calculates the CPU medium core load value and the CPU large core load value from the first moment to the second moment;

[0012] The electronic device queries the pre-trained reinforcement learning RL model according to the CPU load value and the frame loss flag of the N-1th image frame to obtain the CPU frequency point, including:

[0013] The electronic device queries a pre-trained RL model according to the load value of the CPU middle core, the load value of the CPU large core, and the frame loss flag of the N-1th image frame to obtain the frequency point of the CPU middle core and the frequency point of the CPU large core.

[0014] According to the first aspect, or any implementation of the first aspect above, the electronic device queries the pre-trained RL model according to the CPU middle core load value, the CPU large core load value, and the frame loss identifier of the N-1th image frame to obtain the frequency point of the CPU middle core and the frequency point of the CPU large core, including:

[0015] The electronic device determines a CPU medium core load interval to which the CPU medium core load value belongs and a CPU large core load interval to which the CPU large core load value belongs;

[0016] The electronic device queries the pre-trained RL model according to the load range of the CPU middle core, the load range of the CPU large core, and the frame loss flag of the N-1th image frame to obtain the frequency point of the CPU middle core and the frequency point of the CPU large core.

[0017] According to the first aspect, or any implementation of the first aspect above, in the RL model, the CPU frequency corresponding to the first CPU medium core load interval and the second CPU large core load interval is the first frequency, and the CPU frequency corresponding to the second CPU medium core load interval and the first CPU large core load interval is the second frequency, and the CPU frequency corresponding to the second CPU medium core load interval and the second CPU large core load interval is not less than the first frequency, nor less than the second frequency;

[0018] The medium core load interval of the first CPU is the left adjacent interval of the medium core load interval of the second CPU, and the large core load interval of the first CPU is the left adjacent interval of the large core load interval of the second CPU.

[0019] According to the first aspect, or any implementation of the first aspect above, the method further includes: the electronic device queries the target CPU core to which the critical drawing thread is bound; and the electronic device increases the priority of the target CPU core.

[0020] In this way, the operating efficiency of the key drawing thread can be further improved.

[0021] According to the first aspect, or any implementation of the first aspect above, the CPU load value is a normalized load value.

[0022] According to the first aspect, or any implementation of the first aspect, when the Nth image frame starts to be drawn, it includes: when the Nth image frame starts to be drawn based on the input-output control ioctl message mechanism.

[0023] According to the first aspect, or any implementation of the first aspect above, the current scene of the electronic device may be a camera preview scene, a video recording scene, or the like.

[0024] In a second aspect, an embodiment of the present application provides an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device executes the frequency modulation method of the first aspect and any one of the first aspects.

[0025] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0026] In a third aspect, an embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium includes a computer program, and when the computer program is executed on an electronic device, the electronic device executes the frequency modulation method of the first aspect and any one of the first aspects.

[0027] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0028] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed, enables a computer to execute a frequency modulation method as described in the first aspect or any one of the first aspects.

[0029] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0030] In a fifth aspect, the present application provides a chip, the chip comprising a processing circuit and a transceiver pin, wherein the transceiver pin and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the frequency modulation method as in the first aspect or any one of the first aspects to control the receiving pin to receive a signal, and to control the transmitting pin to send a signal.

[0031] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The figure is an exemplary diagram of the drawing and display process of an image frame when the electronic device is in a camera scene;

[0033] Figure 2 The figure is a schematic diagram showing the time consumption of drawing the image frame interval and the complete drawing process;

[0034] Figure 3 This is a schematic diagram showing an exemplary electronic device performing frequency modulation based on EAS scheduling;

[0035] Figure 4 is a schematic diagram of the hardware structure of an electronic device shown as an example;

[0036] Figure 5 A schematic diagram of the software structure of an electronic device is shown as an example;

[0037] Figure 6a A schematic diagram of a frequency modulation method provided in an embodiment of the present application;

[0038] Figure 6b A schematic diagram of CPU frequency adjustment based on an RL model provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of the training of an RL model is shown as an example;

[0040] Figure 8 A schematic diagram of module interaction of a frequency modulation method provided by an embodiment of the present application is exemplarily shown;

[0041] Fig. 9 is a schematic diagram showing a comparison of frequency modulation modes;

[0042] Fig.10a An exemplary diagram showing the load value distribution comparison of a core in a CPU under different frequency modulation modes is shown;

[0043] Fig.10b An exemplary diagram showing the comparison of the core frequency ratio in the CPU under different frequency modulation modes;

[0044] Fig.11a The following is a schematic diagram showing the load distribution comparison of a CPU large core under different frequency modulation modes;

[0045] Fig.11b The figure shows an example of a comparison of the CPU large core frequency ratio under different frequency modulation modes. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0048] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0049] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0050] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0051] In today's era, smart terminals have become an indispensable tool in people's lives. Among them, the camera function of smart terminals is also constantly improving, making video recording more convenient and efficient. However, when the camera function of the smart terminal is turned on and in the camera scene, the smart terminal consumes a lot of power and is prone to heat, which affects the user experience.

[0052] The above-mentioned smart terminal can be a mobile phone, a tablet computer, a wearable device, a car-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA) or a dedicated camera (such as a SLR camera, a card camera), etc. This application does not impose any restrictions on the specific types of the above-mentioned smart terminal devices.

[0053] The following explanation is given using a mobile phone as an example of the smart terminal.

[0054] When the mobile phone is in a camera scene, its default scheduling mode is EAS (Energy Aware Scheduling). Among them, the camera scene can be a camera preview scene, a camera photo scene, a camera video scene, and other scenes based on the camera function to draw frames and output frames. This embodiment does not specifically limit the camera scene. Exemplarily, when the mobile phone runs a camera application, the mobile phone is in a camera scene. Another example is that when the mobile phone runs other applications and the operation of the application requires calling the mobile phone camera function, for example, the mobile phone runs the WeChat application and the WeChat application is in a video scene or a shooting scene, the mobile phone is in a camera scene.

[0055] Taking the camera preview scene of a mobile phone as an example, the process of drawing frames and sending them to the display based on the camera function of the mobile phone can be roughly referred to as Figure 1 As shown. Figure 1As shown, the image frame data exposed by the camera sensor is sent to the camera provider in the hardware abstraction layer (HAL) through the camera driver. The image frame data is processed by the ISP (Image Signal Processing) pipeline in the camera provider to obtain a preview frame, which is sent to the camera service in the application framework layer. The camera service puts the preview frames received in sequence into the BufferQueue. Continue to refer to Figure 1 As shown, BBQ (Blast Buffer Queue) provides a communication bridge between APP (such as camera application) and SurfaceFlinger in the Framework layer, so that the preview frame image of the camera application can be passed to SurfaceFlinger. Among them, the preview frame image can be understood as a Surface of the camera application window. SurfaceFlinger receives the Surface of all windows (window) as input, calculates the position of each Surface in the final composite image according to parameters such as ZOrder, transparency, size, position, and so on, and hands it over to Hardware Composer (hardware synthesizer) or OpenGL to generate the final display Buffer, and then displays it on the display screen through the display driver. Among them, the display principle of the Android system is based on the vsync mechanism. Under the vsync mechanism, there are 2 types of vsync signals in the Android system: one is the vsync signal generated by hardware (ie HW vsync), and the other is the vsync signal simulated by software, including APP vsync signal and sf vsync signal. For the parts that are not explained in detail in this process, please refer to the existing technology, and no further description will be given here.

[0056] It should be pointed out that the ISP pipeline processing performed in the camera provider includes but is not limited to IFE (Image Singnal Processing Front End) processing, IPE (Image Singnal Processing Post End) processing, SIT (Self-supervised Vision Transformer) processing, SAT (Spatial Alignment Transform) processing, EIS (Electronic Image Stabilization) processing, etc., among which some processing only involves software processing, some processing involves hardware processing, and some processing involves software and hardware processing, which is not limited in this embodiment. Moreover, this embodiment does not limit the image processing algorithms involved in the IFE module, IPE module, SIT module, SAT module, and EIS module, as well as the order in which each module is processed.

[0057] In addition, it should be pointed out that for each preview frame, the drawing starting point is at the starting point of the ISP pipeline processing in the camera provider. For example, in the ISP pipeline processing, the IFE processing is executed first, then for each preview frame, the drawing starting point is at the starting point of the IFE processing. Exemplarily, a stake point can be added at the starting point of the ISP pipeline processing, such as the starting point of the IFE processing, so that other modules can know the drawing start time of each image frame in the camera scene through the corresponding stake point function.

[0058] Among them, when the EAS scheduling method is applied to the camera scene, the frequency modulation interval of its CPU (Central Processing Unit) is unreasonable.

[0059] The default frequency modulation interval of the EAS scheduling method is 9ms, and the CPU frequency will be adjusted every time an interruption occurs. However, the exposure and output time of an image frame in the mobile phone camera scene is usually tens of ms to tens of ms. For example, when the camera application frame rate is 30fps, the exposure and output time of an image frame is 33ms (or the frame output interval is 33ms), and when the camera application frame rate is 60fps, the exposure and output time of an image frame is 16ms (or the frame output interval is 16ms). In this way, the preview image frame output interval is greater than the frequency modulation interval of the EAS scheduling, and the CPU frequency changes multiple times within the output time of an image frame, which increases the frequency points with high energy consumption, thereby increasing the energy consumption of the mobile phone.

[0060] According to the preview frame drawing process, the duration of a preview frame from the start to the end of drawing (that is, the processing flow from cameraprovider to SurfaceFlinger) is about 100ms. After a preview frame drawing starts, before the preview frame drawing ends, several preview frames have already started drawing. The image frame drawing interval can be as follows Figure 2 As shown in the figure, a complete frame drawing process lasts for about several frames of image output time. According to the existing EAS scheduling method, the CPU frequency will be adjusted multiple times within the output time of an image frame, but this cannot significantly shorten the frame drawing time.

[0061] In addition, when the EAS scheduling method is applied to camera scenarios, its scheduling strategy is relatively aggressive, resulting in unreasonable CPU frequency adjustment.

[0062] Take the camera recording scene as an example. Figure 3 The schematic diagram of the electronic device adjusting the CPU frequency based on the EAS scheduling method is shown as an example. Among them, CPU3, CPU4, CPU5, and CPU6 are medium cores, and CPU7 is a large core. Figure 3 As shown in Figure 2, when the mobile phone adjusts the CPU frequency based on the EAS scheduling method, the load of the CPU core does not reach the maximum (such as Figure 3 301 in the middle area), but the frequency of the CPU core has been adjusted to the highest (as shown in Figure 3 As shown in the middle area 302, the power consumption loss at high frequency points increases. Figure 3 When the mobile phone adjusts the CPU frequency based on the EAS scheduling method, no tasks are running on the CPU core (such as Figure 3 303), but its frequency is adjusted to the highest (as shown in FIG. Figure 3 304), resulting in power consumption loss.

[0063] It is understandable that the CPU core of an electronic device can be divided into a small CPU core, a medium CPU core, a large CPU core, and an extra large CPU core, etc. The frequencies of the small CPU core, the medium CPU core, the large CPU core, and the extra large CPU core increase successively, and their operating heat problems will also increase successively. Among them, the frequency of the CPU core can be understood as the number of instructions processed per unit time. In some examples, the CPU core of an electronic device can also be divided into a small CPU core, a small CPU core, a medium CPU core, a large CPU core, etc. Regarding the division of CPU cores, this embodiment does not limit it. For example, in the example Figure 3 In the example shown, CPU3, CPU4, CPU5, and CPU6 may also be referred to as large cores, and CPU7 may be referred to as an extra large core.

[0064] It should be noted that the frequency point can be understood as a number for a fixed frequency. For example, the larger the frequency point, the higher the frequency value.

[0065] Through analysis, it is found that when the scheduling module in the mobile phone performs scheduling based on the EAS scheduling method, if it detects that there are multiple threads in the waiting queue, the CPU core frequency will be increased to the maximum frequency. Since the camera application is a multi-threaded pool application, there will be multiple threads in the waiting queue, which will cause the CPU core frequency to be increased to the maximum frequency frequently, causing the problem of excessive power consumption of the mobile phone.

[0066] Therefore, when the electronic device is in a camera scene, how to reduce the power consumption of the electronic device while ensuring the camera performance is a problem that needs to be solved.

[0067] In order to solve the above problems, an embodiment of the present application provides a frequency modulation method in which the electronic device can modulate the CPU core to a suitable frequency point within a suitable frequency modulation interval, which can not only ensure the camera performance but also reduce the power consumption of the electronic device in the camera scene.

[0068] like Figure 4 FIG. 1 is a schematic diagram of the structure of the electronic device 100. Optionally, the electronic device 100 may be a terminal, which may also be referred to as a terminal device. The terminal may be a device with a camera such as a cellular phone or a tablet computer, which is not limited in this application. It should be understood that Figure 4 The electronic device 100 shown is only one example of an electronic device, and the electronic device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different configuration of components. Figure 4 The various components shown in the drawings may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0069] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a temperature sensor, a motion sensor, an air pressure sensor, a magnetic sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0070] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0071] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0072] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

[0073] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0074] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0075] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0076] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0077] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0078] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0079] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement a data storage function.

[0080] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0081] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0082] The pressure sensor is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor can be arranged on the display screen 194. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor.

[0083] The touch sensor is also called a "touch panel". The touch sensor can be set on the display screen 194. The touch sensor and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event.

[0084] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0085] Figure 5 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0086] The layered architecture of the electronic device 100 divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android Runtime) and the native system library, the HAL and the kernel layer.

[0087] The application layer can include a series of application packages.

[0088] like Figure 5 As shown, the application package may include applications such as camera application, gallery, application with camera function, etc. The application package may also include applications such as call, calendar, map, navigation, music, video, short message, etc.

[0089] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0090] like Figure 5 As shown, the application framework layer may include a camera service and SurfaceFlinger, wherein the camera service may be called by a camera application or other applications with camera functions, thereby providing services related to the camera function. SurfaceFlinger is used to receive the Surface of all windows as input, calculate the position of each Surface in the final composite image according to parameters such as ZOrder, transparency, size, and position, and pass it to Hardware Composer or OpenGL to generate the final display Buffer.

[0091] In addition, the application framework layer can also include window managers, content providers, view systems, telephony managers, resource managers, notification managers, etc.

[0092] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0093] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0094] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0095] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0096] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0097] The notification manager allows applications to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, notification information is used to inform the completion of downloads, message reminders, etc. Notification information can also appear in the system top status bar in the form of icons or scroll bar text, such as notifications of applications running in the background, or in the form of dialog windows on the screen. Notification information can also be, for example, text information prompted in the status bar, a prompt sound, vibration of electronic equipment, flashing indicator lights, etc.

[0098] It should be pointed out that the camera application or other applications with camera functions can also call the content provider, resource manager, notification manager, window manager, view system, etc. according to actual business needs, and the embodiments of the present application do not impose any restrictions on this.

[0099] Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for scheduling and management of the Android system.

[0100] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0101] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0102] The native system library may include multiple functional modules, such as native camera functional module, surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0103] Among them, the native camera function module can communicate with the Camera Service upward and with the CameraProvider downward.

[0104] In this embodiment, the native camera function module can be used to monitor the drawing start time of each frame in the camera scene, and notify the frequency modulation scheduling module in the kernel layer of the drawing start time of each frame.

[0105] In this embodiment, the native camera function module can also determine whether there is a frame loss problem in the current frame under the camera scene, and notify the frequency modulation scheduling module in the kernel layer of the result of whether the current frame is lost.

[0106] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0107] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0108] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0109] A 2D graphics engine is a drawing engine for 2D drawings.

[0110] HAL can encapsulate the drivers in the kernel layer and provide a calling interface to the application framework layer, shielding the implementation details of the low-level hardware. Figure 5 As shown, HAL may include Camera Provider, HardwareComposer (hardware synthesizer), DispSync module, etc.

[0111] Among them, Camera Provider can communicate with Camera Service. Camera Provider is the key process of camera function, involving image processing algorithms, image cropping, video saving and other functions. Exemplarily, ISP pipeline processing can be performed in CameraProvider, including but not limited to IFE processing, IPE processing, SIT processing, SAT processing, EIS processing, etc. Moreover, pile points can be added in Camera Provider to obtain nodes related to image frame processing.

[0112] In this embodiment, a stake point is set in the Camera Provider, and the corresponding stake point function is used to obtain the drawing start time of the image frame. Exemplarily, the stake point can be set at the starting point of the ISP pipeline processing, such as the starting point of the IFE processing.

[0113] Hardware Composer is used by SurfaceFlinger to synthesize Surface to the screen. Hardware Composer can abstract overlays and 2D block transmitters, etc. Its main purpose is to assist the GPU in completing some tasks.

[0114] The DispSync module is used to receive the HW-VSYNC signal and update the calculated SW-VSYNC interval.

[0115] The kernel layer is the layer between hardware and software. The kernel layer at least includes frequency modulation scheduling module, CPU driver, display driver, sensor driver, etc. Among them, the hardware at least includes processor, display screen, sensor, etc.

[0116] In this embodiment, the frequency modulation scheduling module calculates the load information of the electronic device in the current frame, and uses the trained RL model to determine a better CPU frequency point based on the load information and whether the current frame is lost, so as to perform frequency modulation operation on the CPU.

[0117] In this embodiment, the CPU driver can perform frequency modulation operation on the CPU according to the CPU frequency point determined by the frequency modulation scheduling module.

[0118] Understandably, Figure 5 The layers in the software structure shown and the components included in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in the present application.

[0119] It is understandable that in order to implement the frequency modulation method in the embodiment of the present application, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0120] The frequency modulation method provided in the embodiment of the present application is applied to the camera scene of the electronic device, and the frequency modulation interval of the CPU is aligned with the frame rate of the camera application, that is, the frequency modulation interval is equal to or approximately equal to the image frame interval of the camera application. It can be understood that the frequency modulation interval of the CPU frequency point is equal to or approximately equal to the image frame interval when the electronic device is in the camera scene.

[0121] The electronic device adjusts the CPU frequency every time it starts drawing an image frame, that is, the CPU frequency is not adjusted during the exposure time of an image frame.

[0122] like Figure 6aAs shown, the electronic device is in the camera scene, and the moment when the N-1th (N is an integer greater than 1) image frame starts to be drawn is t N-1 , the time when the Nth image frame starts to be drawn is t N When the electronic device starts drawing the Nth image frame, the time t is calculated. N-1 ~ time t N The CPU medium core load value and the CPU large core load value are calculated, and it is determined whether the N-1th image frame is lost. In this way, the electronic device can obtain the CPU medium core frequency point optimization value and the CPU large core frequency point optimization value corresponding to the CPU medium core load value, the CPU large core load value, and the result of whether the N-1th image frame is lost based on the pre-trained RL (Reinforcement learning) model. Then, the electronic device performs frequency modulation operations on the corresponding CPU core according to the CPU medium core frequency point optimization value and the CPU large core frequency point optimization value.

[0123] Continue to refer to Figure 6b , when the electronic device includes multiple CPU medium cores and one CPU large core, the electronic device can calculate the time t N-1 ~ time t N The load value of each CPU core in the CPU, and according to the average of the load values ​​of the cores in these multiple CPUs, at time t N-1 ~ time t N The CPU large core load value and the result of whether the N-1th image frame is dropped are used to query the pre-trained RL model to obtain the CPU medium core frequency point optimization value and the CPU large core frequency point optimization value. Then, the electronic device can adjust the frequency of each CPU medium core according to the CPU medium core frequency point optimization value, and adjust the frequency of the CPU large core according to the CPU large core frequency point optimization value.

[0124] RL (Reinforcement Learning), also known as Reinforcement Learning, Evaluation Learning or Enhanced Learning, is one of the paradigms and methodologies of machine learning. It is used to describe and solve the problem of how an agent (Agent) can maximize its rewards or achieve specific goals through learning strategies during its interaction with the environment. Reinforcement learning regards learning as a trial and evaluation process. The agent selects an action for the environment. After the environment accepts the action, its state changes, and a reinforcement signal ("reward" or "punishment") is generated and fed back to the agent. The agent selects the next action based on the reinforcement signal and the current state of the environment. The principle of selection is to increase the probability of receiving positive reinforcement (i.e., "reward"). Among them, the selected action not only affects the immediate reinforcement value, but also affects the state of the environment at the next moment and the final reinforcement value.

[0125] Figure 7 The training diagram of the RL model provided in the embodiment of the present application is shown as an example. Figure 7As shown in (1), the training input data of the agent is the average load of each CPU core when drawing a frame, the load value of the CPU large core when drawing a frame, and the result of whether the image frame is lost (such as the frame loss mark). The agent generates random frequency points (including random frequency points of the CPU core and random frequency points of the CPU large core), and feeds back to the agent based on the performance status (i.e., whether the frame is lost) and the power consumption of the electronic device at the random frequency point, so that the agent (Agent) gradually determines a better frequency point (including the preferred frequency point of the CPU core and the preferred frequency point of the CPU large core) according to the feedback. Among them, the more training data the agent has, the more reasonable the frequency point learned by the agent, and the more accurate the RL model trained to select the optimal frequency point of the CPU.

[0126] Among them, when the RL model is trained, the weighted cumulative sum of the camera performance (i.e., whether the frame is dropped) and the power consumption level of the electronic device at the random frequency point can be fed back to the intelligent agent as feedback information. Optionally, the smaller the value of the cumulative sum, the better the condition, and the larger the value of the cumulative sum, the worse the condition. In order to ensure that the intelligent agent can take into account both the device power consumption and the camera performance (i.e., reduce the frame loss rate) when learning the optimal frequency point, the weight of the camera performance is equal to the weight of the power consumption level of the electronic device at the random frequency point.

[0127] In an optional implementation, in the training input data, the result of whether the image frame is lost or the frame loss mark can be represented by a value "0" and a value "1". Among them, the value "1" represents the situation of frame loss, and the value "0" represents the situation of no frame loss. At this time, in order to take into account the power consumption of the device and the performance of the camera, the power consumption level of the electronic device can be represented by a value of "0" to "1", such as 0.1, 0.3, etc.

[0128] Continue to refer to Figure 7 In (2), when training the RL model for selecting the optimal CPU frequency, an ADEB environment is built in the mobile phone so that the mobile phone can communicate with the PC based on a socket connection. The mobile phone transmits its load information (including the CPU medium core load value and the CPU large core load value) and the result of whether the frame is lost to the PC so that the PC can train the RL model.

[0129] After the RL model training is completed, the CPU medium core frequency point optimization value and the CPU large core frequency point optimization value corresponding to a CPU medium core load value, a CPU large core load value, and a frame loss flag (whether a frame is lost) can be queried based on the RL model.

[0130] Optionally, based on the RL model, the optimal value of the CPU medium core frequency point and the optimal value of the CPU large core frequency point corresponding to a CPU medium core load value interval, a CPU large core load value interval, and a frame loss flag (the result of whether a frame is lost) can be queried.

[0131] In the embodiment of the present application, the load value may be represented by a normalized load value (utility).

[0132] Exemplarily, four preferred frequency point tables can be obtained based on the RL model obtained after training. Among them, the first CPU preferred frequency point table is used to query the CPU core frequency point preferred value corresponding to a CPU core load value (or load value interval) and a CPU large core load value (or load value interval) when the frame loss mark indicates "frame loss", the second CPU preferred frequency point table is used to query the CPU core frequency point preferred value corresponding to a CPU core load value (or load value interval) and a CPU large core load value (or load value interval) when the frame loss mark indicates "no frame loss", the third CPU preferred frequency point table is used to query the CPU core load value (or load value interval) and a CPU large core load value (or load value interval) when the frame loss mark indicates "frame loss", and the fourth CPU preferred frequency point table is used to query the CPU core frequency point preferred value corresponding to a CPU core load value (or load value interval) and a CPU large core load value (or load value interval) when the frame loss mark indicates "no frame loss".

[0133] Table 1 shows a CPU preferred frequency table by way of example. The values ​​in Table 1 are preferred CPU frequencies. As shown in Table 1, the CPU medium core utility can be divided into n1 intervals, and the CPU large core utility can be divided into n2 intervals. One CPU medium core utility interval and one CPU large core utility interval can uniquely determine a preferred CPU frequency. n1 and n2 can be equal or unequal, and this embodiment does not limit this.

[0134] Table 1

[0135]

[0136] Another exemplary example is that the RL model obtained after training can be four preferred frequency point maps. Among them, the first CPU preferred frequency point map is used to query the CPU core frequency point preferred value corresponding to the core load value (or load value interval) of a certain CPU and the large core load value (or load value interval) of a certain CPU when the frame loss mark indicates "frame loss", the second CPU preferred frequency point map is used to query the CPU core frequency point preferred value corresponding to the core load value (or load value interval) of a certain CPU and the large core load value (or load value interval) of a certain CPU when the frame loss mark indicates "no frame loss", the third CPU preferred frequency point map is used to query the CPU core frequency point preferred value corresponding to the core load value (or load value interval) of a certain CPU and the large core load value (or load value interval) of a certain CPU when the frame loss mark indicates "frame loss", and the fourth CPU preferred frequency point map is used to query the CPU core frequency point preferred value corresponding to the core load value (or load value interval) of a certain CPU and the large core load value (or load value interval) of a certain CPU when the frame loss mark indicates "no frame loss".

[0137] Optionally, in any preferred frequency point diagram, the x-axis data is the CPU mid-core utility, the y-axis data is the CPU large-core utility, and the z-axis data is the preferred CPU frequency point. Among them, in any preferred frequency point diagram, the CPU mid-core utility can also be divided into n1 intervals, and the CPU large-core utility can be divided into n2 intervals. One CPU mid-core utility interval and one CPU large-core utility interval can uniquely determine a preferred CPU frequency point. n1 and n2 can be equal or unequal, and this embodiment does not limit this.

[0138] Another example is that four preferred frequency matrixes can be obtained based on the RL model obtained after training. Among them, the first CPU preferred frequency matrix is ​​used to query the CPU core frequency preferred value corresponding to a CPU core load value (or load value interval) and a CPU large core load value (or load value interval) when the frame loss mark indicates "frame loss", the second CPU preferred frequency matrix is ​​used to query the CPU core frequency preferred value corresponding to a CPU core load value (or load value interval) and a CPU large core load value (or load value interval) when the frame loss mark indicates "no frame loss", the third CPU preferred frequency matrix is ​​used to query the CPU core load value (or load value interval) and a CPU large core load value (or load value interval) when the frame loss mark indicates "frame loss", and the fourth CPU preferred frequency matrix is ​​used to query the CPU core frequency preferred value corresponding to a CPU core load value (or load value interval) and a CPU large core load value (or load value interval) when the frame loss mark indicates "no frame loss".

[0139] Optionally, in any preferred frequency matrix, the horizontal axis is the CPU mid-core utility, the vertical axis is the CPU large-core utility, and the matrix elements are the preferred CPU frequencies. Among them, in any preferred frequency matrix, the CPU mid-core utility can also be divided into n1 intervals, and the CPU large-core utility can be divided into n2 intervals. One CPU mid-core utility interval and one CPU large-core utility interval can uniquely determine a preferred CPU frequency. n1 and n2 can be equal or unequal, and this embodiment does not limit this.

[0140] After the RL model is trained, there may be untrained CPU mid-core utility intervals and CPU large-core utility intervals. That is, based on these untrained CPU mid-core utility intervals and CPU large-core utility intervals, it is impossible to determine a preferred CPU frequency. Therefore, based on the CPU frequency corresponding to the trained CPU mid-core utility interval and CPU large-core utility interval, the CPU frequency corresponding to the untrained CPU mid-core utility interval and CPU large-core utility interval can be supplemented.

[0141] Among them, when the CPU frequency points corresponding to the CPU medium core utility interval a and the CPU large core utility interval b are supplemented, the CPU frequency points are not lower than the CPU frequency points corresponding to the CPU medium core utility interval a-1 and the CPU large core utility interval b, and are also not lower than the CPU frequency points corresponding to the CPU medium core utility interval a and the CPU large core utility interval b-1. The CPU medium core utility interval a-1 is the left adjacent interval of the CPU medium core utility interval a, and the CPU large core utility interval b-1 is the left adjacent interval of the CPU large core utility interval b.

[0142] For example, assuming that the CPU frequency corresponding to the CPU mid-core utility interval 2 and the CPU large-core utility interval 3 is 4, and the CPU frequency corresponding to the CPU mid-core utility interval 3 and the CPU large-core utility interval 2 is 6, then when supplementing the CPU frequency corresponding to the CPU mid-core utility interval 3 and the CPU large-core utility interval 2, the frequency cannot be lower than the CPU frequency "4" corresponding to the CPU mid-core utility interval 2 and the CPU large-core utility interval 3, nor lower than the CPU frequency "6" corresponding to the CPU mid-core utility interval 3 and the CPU large-core utility interval 2.

[0143] like Figure 8 The following is a schematic diagram of the interaction between the modules. Figure 8 The process of the frequency modulation method provided in the embodiment of the present application specifically includes:

[0144] S401, in response to a user starting a camera application operation, the Native camera function module monitors the first image frame and starts drawing.

[0145] Exemplarily, the operation of the user starting the camera application can be understood as the operation of the user opening the camera application. The camera scene of the electronic device can be a camera preview scene, a video recording scene, etc.

[0146] In this embodiment, the Native camera function module can monitor whether there is an image frame to start drawing based on the plug-in function corresponding to the target point in the Camera Provider.

[0147] It should be pointed out that the first image frame here refers to the first image frame whose drawing start time is first sent by the Native camera function module to the frequency modulation scheduling module, and is not a limitation on the sequence or number of the image frames.

[0148] S402: The Native camera function module transmits the drawing start time t1 of the first image frame to the frequency modulation scheduling module.

[0149] In one example, the Native camera function module may transmit the drawing start time of each image frame to the frequency modulation scheduling module based on an ioctl (input / output control) message mechanism.

[0150] S403: The Native camera function module transmits a frame loss flag of the first image frame to the frequency modulation scheduling module.

[0151] The frame loss flag is used to indicate whether there is a frame loss problem in the image frame. For example, the Native camera function module reads the image frame from the relevant cache or register at the beginning of each image frame drawing. If the image frame can be read, it indicates that the current frame is not lost, otherwise it indicates that the current frame is lost.

[0152] Exemplarily, when the frame loss flag is "1", it indicates frame loss, and when the frame loss flag is "0", it indicates no frame loss.

[0153] In this embodiment, the Native camera function module may also transmit the frame loss flag of each image frame to the frequency modulation scheduling module based on the ioctl message mechanism.

[0154] In addition, it should be pointed out that if there is a frame loss problem in the current image frame, the drawing process for the current image frame is still executed, the difference is that the drawing process draws a blank frame at this time.

[0155] This embodiment does not limit the timing sequence of S402 and S403.

[0156] S404 , the frequency modulation scheduling module writes the drawing start time t1 of the first image frame into the counter.

[0157] In this embodiment, the counter is used to store the drawing start time of the image frame. The initial state of the counter is empty. It can be understood that the drawing start time written into the timer for the first time is the drawing start time of the first image frame.

[0158] In the technical solution of the present application, the frequency modulation scheduling module predicts the CPU load when the current image frame is drawn by calculating the CPU load information when the previous image frame is drawn, and then queries the optimal CPU frequency based on the trained RL model to adjust the CPU frequency. Therefore, the frequency modulation scheduling module records the drawing start time of the previous image frame in order to calculate the CPU load value when the previous image frame is drawn in combination with the drawing start time of the current image frame. That is, the frequency modulation scheduling module records the drawing start time t1 of the first image frame in order to calculate the CPU load value when the first image frame is drawn in combination with the drawing start time of the second image frame.

[0159] S405, the Native camera function module monitors the Nth (N>1) image frame and starts drawing.

[0160] S406, the Native camera function module transmits the drawing start time t of the Nth image frame to the frequency modulation scheduling module N .

[0161] S407, the Native camera function module transmits the frame loss flag of the Nth image frame to the frequency modulation scheduling module.

[0162] This embodiment does not limit the timing sequence of S406 and S407.

[0163] S408: The frequency modulation scheduling module reads the drawing start time t of the N-1th image frame in the counter. N-1 .

[0164] S409: The frequency modulation scheduling module clears the counter and sets the drawing start time t N Write counter.

[0165] S410, the frequency modulation scheduling module calculates t N-1 ~t N The CPU medium core load value and CPU large core load value within the time period.

[0166] In this embodiment, the CPU medium core load value and the CPU large core load value are both normalized load values ​​(utility).

[0167] Among them, at t N-1 ~t N The normalized load value taskutil of a task in a time period is:

[0168]

[0169] Δtask is the task at t N-1 ~t N The running time in the time period, curfreq is the frequency of the CPU core when running the task, maxfreq is the maximum frequency of the CPU core running the task, and maxcap is the maximum normalized load value of the CPU core running the task.

[0170] If at t N-1 ~t N If only one task is running on a CPU core during a time period, then the CPU core will be N-1 ~t N The normalized load value during the time period is equal to the normalized load value of this task.

[0171] If at t N-1 ~t N If multiple tasks (task_1, task_2, ..., task_n) are running on a CPU core during a time period, then the CPU core will N ~t N-1 The normalized load value cpuutil during the time period is;

[0172]

[0173] Δtask_i is the value of task_i at t N-1 ~t N The running time in the time period, curfreq_i is the frequency of the CPU core when running task_i, maxfreq is the maximum frequency of the CPU core, and maxcap is the maximum normalized load value of the CPU core.

[0174] In one case, the number of cores in the CPU is multiple, and the electronic device is N-1 ~t N The CPU core load value in the time period can be the CPU core load value in t N-1 ~t NThe average value of the normalized load value within a time period. Taking 4 CPU cores as an example, namely CPU3, CPU4, CPU5, and CPU6, the electronic device is N-1 ~t N The CPU core load value in the time period is (cpu3util+cpu4util+cpu5util+cpu6util) / 4. cpu3util means that CPU3 is N-1 ~t N Normalized load value during the time period. cpu4util indicates the CPU4 load during t N-1 ~t N Normalized load value during the time period. cpu4util indicates that CPU5 is N-1 ~t N Normalized load value in a time period. cpu6util indicates the load of CPU6 in t N-1 ~t N Normalized load value over the time period.

[0175] S411, the frequency modulation scheduling module queries the target CPU core bound to the frame drawing key thread, and sends an instruction to increase the priority of the target CPU core to the CPU driver.

[0176] The instruction is used to instruct the CPU driver to increase the priority of the target CPU core. Optionally, the instruction can be used to instruct to increase the priority of the target CPU core by one level, or to instruct to increase the priority of the target CPU core to the highest priority, which is not limited in this embodiment.

[0177] S412: The CPU driver increases the priority of the target CPU core.

[0178] After receiving the priority increase indication information sent by the frequency modulation scheduling module, the CPU driver increases the priority of the target CPU core running the frame drawing critical thread, for example, by one level, or increases the priority to the highest priority.

[0179] S413, the frequency modulation scheduling module queries the RL model according to the CPU medium core load value, the CPU large core load value, and the frame loss flag of the N-1th image frame, determines the CPU medium core frequency point and the CPU large core frequency point, and sends a frequency modulation instruction to the CPU driver.

[0180] Exemplarily, the frequency modulation scheduling module queries a corresponding CPU preferred frequency table or CPU preferred frequency graph or CPU preferred frequency matrix in the RL model according to the CPU medium core load value, the CPU large core load value, and the frame loss flag of the N-1th image frame, and obtains the CPU medium core preferred frequency corresponding to the CPU medium core load value and the CPU large core load value, and the CPU large core preferred frequency corresponding to the CPU medium core load value and the CPU large core load value.

[0181] Optionally, the frequency modulation scheduling module queries a corresponding CPU preferred frequency point table or CPU preferred frequency graph or CPU preferred frequency matrix in the RL model according to the CPU medium core load value, the CPU large core load value, and the frame loss flag of the N-1th image frame, and obtains the CPU medium core preferred frequency points corresponding to the interval where the CPU medium core load value and the interval where the CPU large core load value are located, and the CPU large core preferred frequency points corresponding to the interval where the CPU medium core load value and the interval where the CPU large core load value are located.

[0182] After querying the CPU medium core frequency point and the CPU large core frequency point, the frequency modulation scheduling module generates a corresponding frequency modulation indication according to the CPU medium core frequency point and the CPU large core frequency point, and sends the frequency modulation indication to the CPU driver. Exemplarily, the frequency modulation indication includes but is not limited to the frequency point corresponding to the CPU large core and the CPU frequency point corresponding to the CPU medium core.

[0183] S414: The CPU driver adjusts the frequencies of the CPU middle core and the CPU large core according to the frequency adjustment instruction.

[0184] After receiving the frequency adjustment instruction, the CPU driver adjusts the operating frequency of each CPU core according to the analyzed CPU core frequency point, and adjusts the operating frequency of the CPU large core according to the analyzed CPU large core frequency point.

[0185] This embodiment does not limit the timing sequence of S411 and S413.

[0186] It can be understood that S405-S413 is a process that needs to be repeatedly executed, that is, every time an image frame starts to be drawn, the frequency modulation scheduling module queries the better CPU medium core frequency point and CPU large core frequency point based on the CPU medium core load value and CPU large core load value when the previous frame was drawn, as well as the frame loss mark, to adjust the frequency of the CPU medium core and the CPU large core.

[0187] Taking N=2 as an example, when the Native camera function module monitors the start of drawing of the second image frame, it obtains the drawing start time t2 of the second image frame, and passes the drawing start time t2 of the second image frame and the frame loss flag of the second image frame to the frequency modulation scheduling module. After receiving the drawing start time t2 of the second image frame, the frequency modulation scheduling module reads the drawing start time t1 of the first image frame in the counter, and writes the drawing start time t2 of the second image frame into the counter after clearing the counter. Then, the frequency modulation scheduling module calculates the CPU middle core load value and the CPU large core load value in the time period t1 to t2, and queries the RL model according to the CPU middle core load value, the CPU large core load value, and the frame loss flag of the first image frame, determines the CPU middle core frequency point and the CPU large core frequency point, and sends a frequency modulation instruction to the CPU driver so that the CPU driver adjusts the frequency of the CPU middle core and the CPU large core.

[0188] Similarly, taking N=3 as an example, when the Native camera function module monitors the start of drawing of the third image frame, it obtains the drawing start time t3 of the third image frame, and passes the drawing start time t3 of the third image frame and the frame loss flag of the third image frame to the frequency modulation scheduling module. After receiving the drawing start time t3 of the third image frame, the frequency modulation scheduling module reads the drawing start time t2 of the second image frame in the counter, and writes the drawing start time t3 of the third image frame into the counter after clearing the counter. Then, the frequency modulation scheduling module calculates the CPU middle core load value and the CPU large core load value in the time period t2 to t3, and queries the RL model according to the CPU middle core load value, the CPU large core load value, and the frame loss flag of the second image frame, determines the CPU middle core frequency point and the CPU large core frequency point, and sends a frequency modulation instruction to the CPU driver so that the CPU driver adjusts the frequency of the CPU middle core and the CPU large core.

[0189] Similarly, every time an image frame starts to be drawn, the frequency modulation scheduling module queries the better CPU medium core frequency point and CPU large core frequency point based on the CPU medium core load value and CPU large core load value when the previous frame was drawn, as well as the frame loss flag, to adjust the frequency of the CPU medium core and CPU large core.

[0190] In the embodiment of the present application, the image frame drawing start time is used as the CPU frequency modulation opportunity, so that the frequency modulation interval matches the frame rate. Compared with the EAS scheduling method, the frequency modulation interval is reduced and the frequency point is adjusted more appropriately, which can not only ensure the camera performance, but also reduce the power consumption of the device.

[0191] Fig. 9 The frequency modulation comparison diagram is shown as an example. Taking the electronic device in the camera scene as an example, Fig. 9The example (1) shows a schematic diagram of CPU frequency change when the electronic device modulates the frequency based on the EAS scheduling method. Fig. 9 The example in (2) shows a schematic diagram of CPU frequency change when the electronic device modulates the frequency based on the frequency modulation method provided by this embodiment. Fig. 9 It can be seen from (1) and (2) that when the electronic device performs frequency modulation based on the frequency modulation method provided in this embodiment, the frequencies of the CPU middle cores (CPU3, CPU4, CPU5, CPU6) and the CPU large core (CPU7) are more stable, which reduces power consumption waste, and the frequencies of the CPU middle cores (CPU3, CPU4, CPU5, CPU6) and the CPU large core (CPU7) are also relatively lower.

[0192] Fig.10a The following is an exemplary diagram showing the load distribution comparison of a CPU core under different frequency modulation modes. Fig.10a As shown, when the electronic device modulates the frequency based on the frequency modulation method provided in this embodiment, the core load value of the CPU hardly increases suddenly, and is relatively more stable.

[0193] Take the frame rate of the camera recording scene as 60Hz as an example. Fig.10b The following is a schematic diagram showing the comparison of the core frequency ratio in the CPU under different frequency modulation modes. Fig.10b The example in (1) shows a schematic diagram of the proportion of core frequencies in the CPU when the electronic device modulates the frequency based on the EAS scheduling method. Fig.10b The example in (2) shows a schematic diagram of the proportion of core frequencies in the CPU when the electronic device modulates the frequency based on the frequency modulation method provided by this embodiment. Fig.10b Zhong (1) and Fig.10b As can be seen from (2), when the electronic device performs frequency modulation based on the frequency modulation method provided by this embodiment, the median of the core frequency of the CPU is lower, which will undoubtedly reduce the power consumption of the device.

[0194] Fig.11a The following is an exemplary diagram showing the load distribution comparison of the CPU large core under different frequency modulation modes. Fig.11a As shown, when the electronic device modulates the frequency based on the frequency modulation method provided in this embodiment, the CPU large core load value hardly increases suddenly, is relatively more stable, and can also increase the load according to the actual situation to meet the usage requirements.

[0195] Take the frame rate of the camera recording scene as 60Hz as an example. Fig.11b The following is an example of a comparison of the CPU large core frequency ratio under different frequency modulation methods. Fig.11b The example (1) shows a schematic diagram of the CPU large core frequency ratio when the electronic device adjusts the frequency based on the EAS scheduling method. Fig.11bThe example in (2) shows a schematic diagram of the CPU large core frequency ratio when the electronic device is frequency modulated based on the frequency modulation method provided by this embodiment. Fig.11b Zhong (1) and Fig.11b As can be seen from (2), when the electronic device modulates the frequency based on the frequency modulation method provided in this embodiment, the proportion of the low frequency of the CPU large core is larger (increased from 94.2% to 97.3%), which means that the frequency point of the CPU large core is reduced, and the corresponding device power consumption will also be reduced.

[0196] This embodiment further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the frequency modulation method in the above-mentioned embodiment.

[0197] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the frequency modulation method in the above-mentioned embodiment.

[0198] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor can execute the computer execution instructions stored in the memory so that the chip executes the frequency modulation method in the above-mentioned method embodiments.

[0199] Among them, the electronic device (such as a mobile phone, etc.), computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0200] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0201] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0202] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A frequency modulation method, characterized in that: Applied in an electronic device, the current scene of the electronic device is realized based on a camera function, and the method includes: When the Nth image frame starts to be drawn, the CPU load value from the first moment to the second moment is calculated; N is an integer greater than 1, the first moment is the moment when the N-1th image frame starts to be drawn, and the second moment is the moment when the Nth image frame starts to be drawn; Determine the CPU frequency point according to the CPU load value and the frame loss flag of the N-1th image frame; the frame loss flag is used to indicate whether a frame is lost; The CPU frequency is adjusted according to the CPU frequency point.

2. The method according to claim 1, characterized in that Also includes: When the N+1th image frame starts to be drawn, the CPU load value from the second moment to the third moment is calculated; the third moment is the moment when the N+1th image frame starts to be drawn; Determine a CPU frequency point according to the CPU load value and a frame loss flag of the Nth image frame; The CPU frequency is adjusted according to the CPU frequency point.

3. The method according to claim 1, characterized in that Determining a CPU frequency point according to the CPU load value and a frame loss flag of the N-1th image frame includes: According to the CPU load value and the frame loss flag of the N-1th image frame, the pre-trained RL model is queried to obtain the CPU frequency point.

4. The method according to claim 3, characterized in that Calculate the CPU load value from the first moment to the second moment, including: Calculate the CPU medium core load value and the CPU large core load value from the first moment to the second moment; According to the CPU load value and the frame loss flag of the N-1th image frame, a pre-trained reinforcement learning RL model is queried to obtain a CPU frequency point, including: According to the load value of the CPU medium core, the load value of the CPU large core, and the frame loss identifier of the N-1th image frame, a pre-trained RL model is queried to obtain the frequency point of the CPU medium core and the frequency point of the CPU large core.

5. The method according to claim 4, characterized in that According to the load value of the CPU medium core, the load value of the CPU large core, and the frame loss identifier of the N-1th image frame, a pre-trained RL model is queried to obtain the frequency point of the CPU medium core and the frequency point of the CPU large core, including: Determine the CPU medium core load interval to which the CPU medium core load value belongs, and the CPU large core load interval to which the CPU large core load value belongs; According to the load interval of the CPU medium core, the load interval of the CPU large core, and the frame loss identifier of the N-1th image frame, a pre-trained RL model is queried to obtain the frequency point of the CPU medium core and the frequency point of the CPU large core.

6. The method according to claim 5, characterized in that In the RL model, the CPU frequency corresponding to the first CPU medium core load interval and the second CPU large core load interval is the first frequency, and the CPU frequency corresponding to the second CPU medium core load interval and the first CPU large core load interval is the second frequency, and the CPU frequency corresponding to the second CPU medium core load interval and the second CPU large core load interval is not less than the first frequency, nor less than the second frequency; The medium core load interval of the first CPU is the left adjacent interval of the medium core load interval of the second CPU, and the large core load interval of the first CPU is the left adjacent interval of the large core load interval of the second CPU.

7. The method according to any one of claims 1 to 6, characterized in that: Also includes: Query the target CPU core to which the key drawing thread is bound; Raise the priority of the target CPU core.

8. The method according to any one of claims 1 to 6, characterized in that: The CPU load value is a normalized load value.

9. The method according to any one of claims 1 to 6, characterized in that: When the Nth image frame starts drawing, it includes: When the Nth image frame starts to be drawn based on the input and output control ioctl message mechanism.

10. The method according to any one of claims 1 to 6, characterized in that: The current scene is a camera preview scene or a video recording scene.

11. An electronic device, characterized in that: include: one or more processors; Memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device performs the frequency modulation method according to any one of claims 1 to 10.

12. A computer-readable storage medium comprising a computer program, characterized in that: When the computer program is executed on an electronic device, the electronic device is enabled to execute the frequency modulation method according to any one of claims 1 to 10.

Images