Computing Devices

Abstract

To efficiently set the clock frequency of a processor installed in a computing device. [Solution] A computing device (100) includes a processor (110) and a control unit (120) that controls a clock frequency of the processor. The control unit controls the clock frequency using Q-learning, where the state of the processor is the current state in the Q-learning, and the clock frequency is the behavior in the Q-learning. If the execution time of a software process by the processor exceeds a predetermined maximum allowable time, a reward for the Q-learning is negative, and if the execution time does not exceed the predetermined maximum allowable time, a positive reward is set so that the smaller the power consumption of the processor, the larger the reward for the Q-learning.

Description

[Technical Field]

[0001] The present invention relates to computing devices. [Background technology]

[0002] Patent Document 1 describes a system and method for using reinforcement learning to design a data path circuit in a processor that is optimized for area, power, and / or delay. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-68601 Summary of the Invention [Problem to be solved by the invention]

[0004] Dynamic Voltage Frequency Scaling (DVFS) is a CPU power saving technology that changes the power supply voltage and clock frequency of a CPU depending on the CPU load.

[0005] In view of the need for further improvement, an object of the present invention is to efficiently set the clock frequency of a processor using DVFS that is installed in a computing device. [Means for solving the problem]

[0006] A computing device according to the present invention includes a processor and a control unit that controls a clock frequency of the processor, the control unit controlling the clock frequency by Q-learning, the state of the processor being the current state in the Q-learning, and the clock frequency being the behavior in the Q-learning. If the execution time of a software process by the processor exceeds a predetermined maximum allowable time, a reward for the Q-learning is set to a negative value, and if the execution time does not exceed the predetermined maximum allowable time, a positive reward is set so that the smaller the power consumption of the processor, the larger the reward for the Q-learning. [Effects of the Invention]

[0007] According to the present invention, it is possible to efficiently set the clock frequency of a processor installed in a computing device. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of a computing device. [Figure 2] FIG. 10 is an explanatory diagram showing the flow of updating the Q table in the search phase. [Figure 3] FIG. 10 is an explanatory diagram showing the flow of updating the Q table in the utilization phase. [Figure 4A] FIG. 10 is an explanatory diagram illustrating an example of a Q table. [Figure 4B] FIG. 10 is an explanatory diagram illustrating an example of a Q table. [Figure 4C] FIG. 10 is an explanatory diagram illustrating an example of a Q table. [Figure 4D] FIG. 10 is an explanatory diagram illustrating an example of a Q table. [Figure 4E] FIG. 10 is an explanatory diagram illustrating an example of a Q table. [Figure 4F] FIG. 10 is an explanatory diagram illustrating an example of a Q table. [Figure 5] 10 is a flowchart showing the flow of processing in a search phase. [Figure 6] 10 is a flowchart showing the flow of processing in the utilization phase. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described below based on the illustrated embodiments, but the present invention is not limited to the embodiments described below.

[0010] The functionality of software installed in computing devices with limited processing power and resources (e.g., edge devices installed in vehicles) is evolving, increasing the processing load. The increased processing load on the CPU results in increased power consumption and heat generation.

[0011] The DVFS mentioned above has various modes. The following are some of the most common DVFS modes: Performance mode: Sets the maximum CPU frequency (increases power consumption) Power Save Mode: Sets minimum CPU frequency (reduced performance) On-demand mode: Changes CPU frequency depending on processing load

[0012] Conventional DVFS uses simple logic, so it is not considered optimal when the CPU load fluctuates due to complex application processing. Therefore, in the embodiment described below, machine learning is used to change the clock frequency in response to complex application processing.

[0013] 1 shows a computing device 100 according to this embodiment. The computing device 100 includes a CPU 110 with a variable clock frequency, and a control unit 120 that monitors the CPU and controls the clock frequency of the CPU. The power consumption of the CPU 110 is measured by a power sensor, which may be built into the computing device 100 or may be external to the computing device 100.

[0014] Machine learning includes supervised learning, unsupervised learning, and reinforcement learning. In this embodiment, Q-learning, which is one of the reinforcement learning methods, is used. In the exploration phase, a reward for a specific action is calculated according to the power consumption and processing time when that action is taken in relation to the CPU state, and the Q-table is updated. The "CPU state" corresponds to the "state" in Q-learning, and the "selection of CPU frequency" corresponds to the "action" in Q-learning.

[0015] <Exploration Phase> Figure 2 shows the processing flow in the search phase. The Q table QT1 is a table that manages the Q value (value of an action) corresponding to the CPU frequency selected in a certain CPU state. In this Q table, states s0, s1, s2, and s3 are represented as rows, and actions a0, a1, and a2 are represented as columns. The initial value of the Q value is zero. s0, s1, s2, and s3, as well as a0, a1, and a2, are examples, and any number of states and actions can be set.

[0016] In step ST11, the control unit 120 acquires the state of the CPU 110 (one of states s0, s1, s2, and s3) from system parameters (CPU frequency, memory usage, etc.) according to the processing load of the application. In step ST12, the control unit 120 randomly selects one of the actions a0, a1, and a2 in the Q table QT1. In step ST13, the control unit 120 controls the CPU 110 at the frequency corresponding to the action selected in step ST12. In step ST14, the control unit 120 calculates a reward based on the power consumption of the CPU 110 under the control of the CPU 110 performed in step ST13. The reward is calculated so that the smaller the power consumption, the larger the reward, and the larger the power consumption, the smaller the reward. In step ST15, the control unit 120 updates the Q value in the Q table QT1.

[0017] As an example, assume that the system state s0 is acquired in step ST11 and the action a0 is selected in step ST12. In this case, the Q value corresponding to the system state s0 and the action a0 in the Q table QT1 is updated to q1 in step ST15 using the reward calculated in step ST14.

[0018] In this way, steps ST11 to ST15 are repeatedly executed in the search phase, thereby updating the Q table QT1. As a result, a Q table is obtained in which good behaviors have high Q values ​​and bad behaviors have low Q values. An example of the obtained Q table is shown as Q table QT2 in Figure 3.

[0019] <Utilization phase> The processing flow in the utilization phase is shown in Figure 3. The Q table QT2 is the table obtained in the search phase described above.

[0020] In step ST21, the control unit 120 acquires the state of the CPU 110 (one of states s0, s1, s2, and s3) from system parameters (CPU frequency, memory usage, etc.) according to the processing load of the application. In step ST22, the control unit 120 selects from the Q table QT2 the action with the largest Q value among the actions that can be taken for the state acquired in step ST21. For example, when the state s0 is acquired in step ST21, the action with the largest Q value in the Q table QT2 is action a2, and therefore action a2 is selected in step ST22. In step ST23, the control unit 120 controls the CPU 110 at the frequency corresponding to the action selected in step ST22. In step ST24, the control unit 120 calculates a reward based on the power consumption of the CPU 110 under the control of the CPU 110 performed in step ST23. The reward is calculated so that the smaller the power consumption, the larger the reward, and the larger the power consumption, the smaller the reward. In step ST25, the control section 120 updates the Q value in the Q table QT2.

[0021] The degree to which the Q value is affected differs between the update of the Q table in the search phase (step ST15) and the update of the Q table in the utilization phase (step ST25), as will be described in detail below.

[0022] In the search phase and the exploitation phase, the Q table is updated using the following formula (steps ST15 and ST25): Q(S,A)=Q(S,A)+α*(Reward) (1) where S is the state of the CPU 110 acquired in step ST11 or step ST21. A is the action (frequency) selected in step ST12 or step ST22. Q(S, A) is the Q value of the entry in the Q table corresponding to the state S and action A. Q(S, A) on the right side of equation (1) is the Q value before updating, and Q(S, A) on the left side of equation (1) is the Q value after updating. α is the learning rate. Reward is the reward calculated in step ST14 or step ST24.

[0023] The learning rate α is determined as follows:

number

[0024] The learning rate varies depending on the number of updates. In the exploration phase, the learning rate α is 1. In the exploitation phase, the learning rate α is less than 1, and as the number of updates increases, the learning rate α decreases. Thus, in the exploitation phase, as the number of updates increases, the degree to which the Q value is affected decreases.

[0025] Three additional points are mentioned below regarding the application of the frequency change method using Q-learning to computing devices.

[0026] <1. Switching according to load> When a frequency change is performed using Q-learning, a computational load associated with the Q-learning algorithm occurs each time. Therefore, in the utilization phase, when the processing load of the CPU 110 is small (for example, when the CPU usage rate is less than a predetermined threshold (for example, 10%)), the frequency of the CPU 110 may be set using conventional DVFS (for example, the above-mentioned power save mode) instead of Q-learning. This reduces the computational load and power consumption when the processing load of the CPU 110 is small.

[0027] <2.Methods for reducing memory usage> The size of the Q-table depends on the number of states and the number of actions (CPU frequency). As the number of states or actions increases, the size of the Q-table grows exponentially, which may not meet the memory constraints of computing devices such as edge devices.

[0028] For example, suppose that six parameters (PMCs (Performance Monitoring Counters)) are selected as states in the search phase, and a total of six Q-tables shown in FIGS. 4A to 4F are obtained. The states in the Q table shown in FIG. 4A are normalized CPU clock frequencies. The states in the Q table shown in FIG. 4B are the number of instruction memory accesses (normalized values) of the CPU cache memory. The states in the Q table shown in FIG. 4C are the number of instruction memory accesses (normalized values) of the CPU's TLB. The states in the Q table shown in FIG. 4D are the number of read and write accesses (normalized values) to the CPU cache memory. The states in the Q table shown in FIG. 4E are the number of reads and writes (normalized values) of the CPU cache memory. The states in the Q table shown in FIG. 4F are the number of reads and writes (normalized values) of the CPU's TLB. The behavior in the six Q tables is the CPU clock frequency (unit: GHz).

[0029] A cache is a high-speed memory device installed inside a microprocessor (MPU or CPU). By storing frequently used data, it is possible to reduce access to the relatively slow main memory (main storage device), thereby speeding up processing. A TLB (translation lookaside buffer) is a cache memory that temporarily stores the correspondence between virtual and physical addresses in memory on a computer that uses virtual memory.

[0030] For each entry that stores a Q value, grayed-out entries indicate that the Q value is positive, dotted entries indicate that the Q value is negative, and blacked-out entries indicate that the Q value has not yet been searched (not updated). In each table, the percentages of entries with positive Q values, entries with negative Q values, and unsearched entries are as follows:

[0031] Positive Q Negative Q Unexplored Q table in Figure 4A 56.3% 20.3% 23.4% Q table in Figure 4B 19.8% 65.9% 14.3% Q-table in Figure 4C 12.3% 74.1% 13.6% Q table in Figure 4D 27.1% 53.4% ​​19.5% Q table in Figure 4E 31.4% 46.9% 21.7% Q table in Figure 4F 43.1% 32.7% 24.2%

[0032] The correlation between each parameter and power consumption is visualized from the Q table for the six parameters obtained in the exploration phase. Then, only the Q table for the parameter with the greatest correlation can be used in the utilization phase. In other words, only the Q table in Figure 4A can be used in the utilization phase. This is because it makes it easier to take actions that improve power consumption efficiency rather than using all parameters.

[0033] As mentioned above, using six Q-tables in the utilization phase increases memory usage. Therefore, it is possible to use only the Q-table related to the one parameter most related to power consumption among the six system parameters in the utilization phase. This reduces memory usage while achieving efficient power consumption.

[0034] <3. Calculation of Reward> Actions that reduce power consumption and minimize performance degradation are given positive rewards, while actions that increase power consumption or reduce performance are given negative rewards. If rewards were determined solely from the perspective of whether power consumption was reduced, a positive reward would be given regardless of the calculation time if power consumption was reduced, which would increase calculation time and reduce performance, resulting in an impractical Q-table. Therefore, if the calculation time exceeds a predetermined threshold, the reward function is set so that the reward becomes a negative value, and the updated Q value becomes smaller than the Q value before the update. If a negative Q value is included in the Q table created in this way, the action corresponding to that Q value should not be selected in the exploitation phase. The reason why this negative Q value is not selected is because the action with the largest Q value is taken in the exploitation phase.

[0035] The calculation of the reward in steps ST14 and ST24 can be performed using the following formula.

number

[0036] Processing time T i is the threshold T c1 and the actual power consumption P is less than the maximum possible power consumption P High If the processing time is less than T, the reward will be positive. i is the threshold T c1 If the processing time is equal to T, the reward is zero. i is the threshold T c1 If the reward exceeds T i is the threshold T c2 Above this, the reward becomes a biased negative reward.

[0037] In the exploration phase, a negative Q value is set in the Q table created by the above-mentioned reward calculation. Then, in the utilization phase, actions corresponding to a negative Q value are not selected, which reduces the learning time in the utilization phase.

[0038] According to the above-described embodiment, the CPU frequency is selected using the Q-learning algorithm, so that power consumption can be reduced while suppressing an increase in the calculation time of the computing device in the utilization phase.

[0039] We will now explain the exploration phase and the exploitation phase. Figure 5 shows the flow of the exploration phase, and Figure 6 shows the flow of the exploitation phase.

[0040] In step ST51 of FIG. 5, the control unit 120 executes the Q-learning algorithm as shown in FIG. 1 based on the load of the software executed by the CPU 110 (the performance monitoring counter described above). In step ST52, the control unit 120 creates a plurality of heat maps (for example, the six Q tables shown in FIGS. 4A to 4F). In step ST53, the control unit 120 selects one Q table that is highly correlated with power consumption reduction from the plurality of heat maps (that is, the plurality of Q tables) created in step ST52. In step ST54, the control unit 120 further executes the Q-learning algorithm as shown in FIG. 1 for one Q table selected in step ST53.

[0041] 6, the control unit 120 determines whether the processor utilization rate of the CPU 110 is equal to or greater than a threshold value. If it is determined that the processor utilization rate of the CPU 110 is equal to or greater than the threshold value, the control unit 120 proceeds to step ST62, and if not, the control unit 120 proceeds to step ST64. In step ST62, the control unit 120 selects the execution of the Q-learning algorithm, and in step ST63, the control unit 120 executes the Q-learning algorithm, after which step ST61 is executed again. In step ST64, the control unit 120 controls the frequency of the CPU 110 using the power save mode of DVFS, rather than the Q-learning algorithm, and then step ST61 is executed again.

[0042] The effects of the above embodiment will be described below. Q-learning is a machine learning method that determines the best action to take based on the current state. By using Q-learning to control the frequency, it is possible to reduce power consumption while minimizing the degradation of CPU processing performance. This is particularly useful for edge devices with limited resources.

[0043] By changing the learning rate α between the exploration phase and the exploitation phase, it is possible to more easily reflect the learning results in the exploration phase and update the Q-table appropriately. In the exploitation phase, learning can proceed that reflects the actual usage in the exploitation phase, but it is possible to make it more difficult to reflect the learning results compared to the exploration phase. This prevents deviation from the learning results obtained in the exploration phase.

[0044] By allowing negative rewards to be calculated, it becomes less likely that an action whose calculation time may exceed a predetermined time will be selected. If the calculation time is regulated by law, it becomes possible to select an action that complies with the law.

[0045] By determining whether to execute the Q-learning algorithm depending on the CPU load, Q-learning is performed only when necessary. When Q-learning is not required, further power savings are possible. The CPU load can also be reduced.

[0046] By selecting one Q table with a high power-saving effect, it is possible to reduce memory capacity while achieving power-saving effects in the utilization phase.

[0047] In the exploitation phase, unnecessary elements (actions with negative Q values) are less likely to be selected. This reduces the number of times the learning algorithm is executed for actions that are not selected, leading to savings in computational resources.

[0048] The embodiments of the present invention are not limited to CPUs and can be applied to any processor with a variable clock frequency.

[0049] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications and changes can be made based on the technical concept of the present invention. [Explanation of symbols]

[0050] 100 computing devices 110 CPU 120 control section

Claims

1. a processor; a control unit that controls a clock frequency of the processor; Equipped with the control unit controls the clock frequency by Q learning; the state of the processor is the current state in the Q-learning; the clock frequency is a behavior in the Q-learning; When the execution time of the software processing by the processor exceeds a predetermined maximum allowable time, the reward of the Q-learning becomes a negative value; If the execution time does not exceed the predetermined maximum allowable time, a positive value of the reward is determined so that the smaller the power consumption of the processor, the larger the reward of the Q-learning. Computing devices.

2. a learning rate in the exploration phase of the Q-learning is greater than a learning rate in the exploitation phase of the Q-learning; In the utilization phase, the learning rate decreases each time the Q table is updated by the Q-learning. The computing device of claim 1 .

3. In the utilization phase, the control unit controls the clock frequency of the processor by the Q-learning only when the utilization rate of the processor is lower than a predetermined threshold. The computing device of claim 2 .

4. a Q table when the state of the processor is the frequency of the processor; a Q table when the state of the processor is the number of instruction memory accesses of the cache memory of the processor; a Q table when the state of the processor is the number of instruction memory accesses of the TLB of the processor; a Q table when the state of the processor is the number of read and write accesses to the cache memory of the processor; a Q table when the state of the processor is the number of reads and writes to the cache memory of the processor; A Q table when the state of the processor is the number of reads and writes to the TLB of the processor. The control unit selects one of the following: The control unit uses the selected Q table in the utilization phase. The computing device of claim 3 .

5. The computing device according to claim 4 , wherein the control unit uses only entries in the selected Q table that have positive Q values ​​as targets for Q learning in the exploitation phase.

6. If the learning rate is α, [Equation 1] where M represents the number of updates to the Q value to be updated, and M ｅｘｐｌｏｒｅ is a predetermined maximum number of updates in the search phase, and M ｅｘｐｌｏｉｔ is a predetermined maximum number of updates in the exploitation phase, If the reward is Reward, [Equation 2] and T ｉ is the processing time of the application by the processor, and T ｃ１ and T ｃ２ is a predetermined threshold value for the processing time, and P Ｈｉｇｈ is the maximum possible power consumption, P is the actual power consumption of the processor, and w ｐ and w ｎ is a positive coefficient, The computing device of claim 5 .

Images