Real-time memory allocation model and method for multi-virtual machine and multi-process architecture of automobile domain controller

Abstract

The invention discloses a real-time memory allocation method for a multi-virtual machine and multi-process architecture of an automobile domain controller, which comprises the following steps of: converting a real-time memory allocation process oriented to the multi-virtual machine and multi-process architecture in the automobile domain controller into a Markov decision process, and fitting a value function Q (S, A) by utilizing a reinforcement learning depth Q network; a memory real-time allocation model of the multi-virtual machine and multi-process architecture of the automobile domain controller is constructed and obtained; performing domain controller multi-virtual machine and multi-task process memory real-time allocation based on the constructed memory real-time allocation model, including static and dynamic classification of calculation tasks and pre-sorting of the calculation tasks; judging a memory fragmentation degree, inputting memory and calculation task state information, executing a memory real-time allocation rule output by a model, and the like; according to the method, the real-time memory allocation model can be ensured to run when necessary, the energy consumption of the automobile is reduced, and the fragmentation degree of the memory is reduced by flexibly switching various memory allocation rules, so that the real-time memory allocation with real-time response and adaptability is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of resource allocation in a virtual environment, and specifically relates to a real-time memory allocation model and method for a multi-virtual machine and multi-process architecture of an automobile domain controller. Background Art

[0002] The central domain controller architecture has become the mainstream computing architecture for intelligent connected vehicles because of its simple structure, high integration, small amount of hardware, and support for multiple operating systems and architectures. It can well adapt to the needs of the future customized development of intelligent connected vehicles and has gradually become the mainstream computing architecture for intelligent connected vehicles. As the core computing unit of the vehicle, the central domain controller is responsible for the centralized management and coordination of multiple systems. Its memory resource management presents significant characteristics such as multi-system multi-task concurrency, memory sharing and isolation, and dynamic memory allocation. Therefore, the central domain controller and the memory manager on its virtual machine need to dynamically allocate memory between virtual machines and task processes based on the real-time status of tasks and memory.

[0003] At the memory management level, the automotive domain controller needs to adopt a three-level memory mapping mechanism: physical memory, virtual memory, and process memory, and implement the above mapping relationship through the memory page table mechanism. This architecture makes memory allocation based on the minimum memory page as the basic unit, and adopts a two-level memory dynamic allocation mechanism based on the three-level mapping mechanism to efficiently manage memory resources.

[0004] At the computing task level, the computing tasks generated by intelligent connected vehicles during operation are first divided into separable tasks and indivisible tasks, among which indivisible tasks are further divided into independent tasks and dependent tasks. The dependency between tasks may exist not only within the same virtual machine, but also between different virtual machines. Since the type of computing task is usually determined by the input and output data it processes, the dynamic data changes during vehicle operation and the uncertainty of task completion time may cause dynamic changes in task types. This dynamism makes the real-time memory allocation of the central domain controller not only affected by the performance of the computing unit and data communication conditions, but also closely related to the vehicle's operating status. In addition, the fine-grained size of computing tasks during vehicle operation, the high frequency of task switching, and the existence of a large number of virtual machines and concurrent tasks have significantly increased the complexity of real-time memory allocation of the domain controller, making it exponentially difficult.

[0005] The existing technologies usually use local optimization strategies (such as the best fit rule and the first fit rule) for real-time memory allocation under multi-virtual machine and multi-process architectures to quickly meet the memory requirements of current computing tasks. However, these strategies lack a global optimization perspective and do not fully consider the possible future memory requirements and the disturbance effects of dynamic events. In the virtual environment of the automotive domain controller, task computing has higher real-time performance requirements and energy consumption optimization requirements than the traditional virtual environment to support long driving range; at the same time, there are many types of virtual operating systems and the number of parallel tasks has increased sharply, further exacerbating the complexity of memory management. Even the global memory real-time allocation technology in the traditional virtualization scenario is difficult to meet the special requirements of the automotive domain controller in terms of high real-time performance, low latency tolerance, low energy consumption requirements, and multi-virtual machine parallel task processing, resulting in serious memory fragmentation in the later stage of computing task processing, a significant increase in computing latency, and excessive memory management energy consumption. These problems restrict the further improvement of the performance of intelligent connected vehicles. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a real-time memory allocation model and method for a multi-virtual machine and multi-process architecture of an automobile domain controller, which reduces the degree of memory fragmentation by flexibly switching between multiple memory allocation rules, thereby achieving real-time response and adaptive real-time memory allocation.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] The present invention first proposes a real-time memory allocation model for a multi-virtual machine and multi-process architecture of an automobile domain controller, and the construction method thereof comprises the following steps:

[0009] 11) The real-time memory allocation process for multi-virtual machines and multi-process architectures in automotive domain controllers is transformed into a Markov decision process, and a Markov decision model (S, A, π, R, γ) is constructed; where: S represents the state space that can characterize the domain controller computing tasks and physical memory state information in the real-time memory allocation process; A represents the action space composed of multiple specific allocation rules when the automotive domain controller resource management mechanism allocates memory; π represents the allocation strategy of the domain controller resource management mechanism; R represents the reward function; γ∈(0,1) represents the reward discount rate;

[0010] 12) Define the state space S, action space A, reward function R and allocation strategy π:

[0011] The state space S is defined as:

[0012] S={S TASK +S MEMORY}

[0013] Where: S TASKrepresents the computing task state information matrix; S MEMORY represents the memory state information matrix; and:

[0014]

[0015] in: and They respectively represent the virtual machine code and computing task code to which the j-th memory page of the i-th memory block belongs; and The codes respectively represent the memory block and memory page of the memory in its corresponding virtual machine;

[0016]

[0017] in: It indicates the time from the mth virtual machine to the moment when it processes its nth computing task. Indicates that the computing task has been running for a period of time. Indicates that the calculation task will take some time to start; Indicates the remaining time required to complete the calculation task; P mn Indicates the priority level of the task; Indicates the task code that has a dependency relationship with this task; since tasks between different virtual machines also have dependencies, The virtual machine code corresponding to the dependent task; Indicates the maximum delay allowed for processing the task;

[0018] The action space A is defined as:

[0019] A={rule 1 ,rule 2 ,rule 3 ,rule 4 ,rule 5 ,rule 6}

[0020] Among them: rule 1 Indicates that if the first adaptation rule condition is not met, the best rule for the fragment is executed; rule 2 Indicates the first rule of the fragment; rule 3 Indicates the best fit rule; rule 4 Indicates that the best fit rule is considered first, and then the fragment first rule is considered; rule 5 Indicates that the best adaptation rule is considered first; rule 6 Indicates that the fragment first rule is considered first;

[0021] First fit rule: When a single continuous memory is satisfied, traverse the free area of ​​the memory, find the first free block that is large enough, and allocate it to the requesting virtual machine or process;

[0022] Optimal fragmentation rule: When the first fit rule is not met, the largest possible memory fragment is allocated to the requested virtual machine or process;

[0023] First fragment rule: When the first adaptation rule cannot be met, the memory is traversed and the first few memory fragments that meet the requirements are allocated to the requesting virtual machine or process;

[0024] Best fit rule: When a single continuous memory is satisfied, traverse the free area of ​​the memory to find the free block with the most suitable size, otherwise, perform the best fit rule for fragmentation;

[0025] The reward function R is defined as:

[0026]

[0027] in: When the process memory is allocated at time step t, the environment state S t Change to S t+1 The reward value calculated by the agent of the mth virtual machine through the reward function; L represents the number of tasks completed by the mth virtual machine during the period from time step t to time step t+1; Indicates the time taken to complete the lth computing task on the mth virtual machine; represents the maximum delay time of the lth computing task completed by the mth virtual machine from time step t to time step t+1; P ml Indicates the priority level of the lth computing task completed by the mth virtual machine from time step t to time step t+1. The larger the priority level value, the higher the priority of the computing task;

[0028] The allocation strategy π is defined as: in state S t What action to take next? t A strategy to maximize cumulative rewards;

[0029] Optimizing the allocation strategy π(A k |S k ) is used to make the action value function Q π (S t ,A t ) is minimal, and the action value function Q π (S t ,A t ) is equal to in state S t Take action A t The expected value of the cumulative discounted reward is expressed as:

[0030] Q π (S t ,A t )=E π {R t +γQ π (S t+1 ,A t+1 )}

[0031] Among them, E π represents the expected value of the cumulative discounted reward obtained under the allocation strategy π;

[0032] 13) Train the deep Q network to fit the value function Q(S,A).

[0033] Further, in step 13), the training steps of the deep Q network are:

[0034] 131) Initialize the algorithm hyperparameters of the Q network, including learning rate, discount factor, exploration rate, experience replay pool size, batch size, number of training steps, hyperparameters of the Adam optimizer, random seed, and target network update frequency; initialize the weight parameters of the main body of each agent, where: Agent 0 Deployed on the host computer of the automotive domain controller computing system, the intelligent agent m Deployed on virtual machine m;

[0035] 132) According to the set exploration rate, the agents on the host machine and each virtual machine randomly generate the neural network of the current training step; obtain the best action of the host machine and each virtual machine at the current time step t And interact with the environment to get the state s t+1 And feedback reward value The data generated Store to Agent 0 In the experience replay pool, the data generated Store to Agent m Experience replay pool; among them: Represents the host agent 0 The best action to output, Agent m The best action to output; Represents the environment to give feedback to the Agent 0 The reward value of Represents the reward value fed back to the virtual machine m agent by the environment;

[0036] 133) Determine whether the data in the experience replay pool of any agent has reached the set amount; if so, execute step 134); if not, set t=t+1 and execute step 132);

[0037] 134) For the agents whose experience replay pool data reaches the set amount, randomly sample the batch amount of data from the corresponding experience replay pool data to the Q network and the target network; the Q network uses the state-action pair data (S t ,A t ) predicts the corresponding Q value; the target network uses the state S t+1 The data calculates the corresponding maximum Q value in various actions;

[0038] 135) Parameter update:

[0039] Use the loss function L to reversely calculate the loss function gradient of each parameter θ in the deep Q network To update the weight parameters of each node in the Q network;

[0040] Determine whether the number of random Batch sampling after the last parameter update on the target network is equal to the preset number H: if so, update the parameters of the target network; if not, do not update the parameters of the target network;

[0041] 136) Determine whether the time step reaches the set maximum time step: if so, the deep Q network training is completed; if not, set t = t + 1, and execute step 142).

[0042] Further, in step 132), the optimal action A is selected at time step t with the maximum output of the Q network as the goal. t for:

[0043]

[0044] Where: Q(S,A) represents the value function of the deep Q network.

[0045] Further, in step 135), the parameters of the deep Q network approximator are updated by instant differential learning, the principle is:

[0046]

[0047] Among them, α represents the learning rate.

[0048] Furthermore, Q(S,A,θ) represents the action value function fitted by the deep neural network, and the loss function L is defined as:

[0049]

[0050] Among them, θ z represents the zth parameter in the deep Q network; Indicates that in state S t When taking action A t Then extract the reward R from the domain controllert and in state S t+1 The expected value of the cumulative discounted reward when taking the highest Q-value action; y i represents the target value, and:

[0051]

[0052] in, Indicates that the reward R is extracted from the domain controller t and in state S t+1 The expected value of the cumulative discounted reward when taking the highest Q-value action.

[0053] The present invention also proposes a real-time memory allocation method for a multi-virtual machine and multi-process architecture of an automobile domain controller, comprising the following steps:

[0054] Step 1: deploy the memory real-time allocation model of the multi-virtual machine and multi-process architecture of the vehicle domain controller as described above to the vehicle domain controller;

[0055] Step 2: Real-time memory allocation for multiple virtual machines and multi-task processes on the domain controller

[0056] 21) Task static instruction pre-ordering and dynamic ordering during execution:

[0057] Classify computing tasks into divisible tasks and indivisible tasks, split divisible tasks into indivisible tasks; divide all indivisible tasks into independent tasks and dependent tasks;

[0058] Sort computing tasks by priority level; for computing tasks with the same priority level, give priority to computing tasks with smaller maximum allowable delay; for dependent tasks, sort them by dependency relationship;

[0059] During the execution of computing tasks, the computing system dynamically sorts the computing tasks in real time;

[0060] 22) Determine whether the number of free memory blocks is greater than or equal to a set number threshold: if so, perform real-time memory allocation for multiple virtual machines and multi-task processes of the domain controller, and execute step 24); if not, do not perform real-time memory allocation for multiple virtual machines and multi-task processes of the domain controller, and execute step 23);

[0061] 23) Allocate memory according to the first adaptation rule: if the first adaptation rule cannot find a suitable free memory block, execute step 24); if the first adaptation rule can find a suitable free memory block, execute step 25);

[0062] 24) Inputting the state information of memory and computing tasks into the real-time memory allocation model of the multi-virtual machine and multi-process architecture of the automobile domain controller, outputting the Q values ​​of various real-time memory allocations, and selecting the memory allocation rule corresponding to the maximum Q value for memory allocation;

[0063] 25) After the central domain controller computing system completes memory allocation according to the allocation rules given by the real-time memory allocation model, the status information of the memory and computing tasks changes, and the reward value is calculated.

[0064] Step 3: Determine whether there is a computing task to be executed: if so, run the memory real-time allocation model and execute step 2; if not, do not run the memory real-time allocation model.

[0065] Furthermore, in step 24), for the intelligent agent deployed on the host machine 0 When the physical memory cannot meet the memory requirements of the currently running computing tasks and the computing tasks started at the current time step, the memory management mechanism on the host machine follows the Agent 0 Given Allocate physical memory to the virtual machine; for the agent deployed on the virtual machine m m , when virtual machine m has a new computing task requesting memory allocation at time step, virtual machine m follows Agent m Given Allocate the virtual memory of the virtual machine to the new computing process; for the virtual machine that has no new computing tasks generated in the current time step or the host machine that has no physical memory allocation requirements, the best action given by the corresponding agent No actual memory allocation is triggered.

[0066] Further, in step 25), after obtaining the reward value, the data {S t ,A t ,R t ,S t+1} is stored in the experience replay pool of the corresponding agent, and the data in the experience replay pool is uploaded to the cloud server for retraining of the memory real-time allocation model.

[0067] The beneficial effects of the present invention are:

[0068] The present invention discloses a method for real-time memory allocation for a multi-virtual machine and a multi-process architecture of an automobile domain controller. The method converts the real-time memory allocation process for the multi-virtual machine and the multi-process architecture in the automobile domain controller into a Markov decision process, and uses a reinforcement learning deep Q network to fit a value function Q(S, A), so as to construct a real-time memory allocation model. Based on the constructed real-time memory allocation model, real-time memory allocation for the multi-virtual machine and the multi-task process of the domain controller is performed, including: static and dynamic classification of computing tasks, pre-sorting of computing tasks; judgment of the degree of memory fragmentation, input of memory and computing task status information, and real-time memory allocation rules output by the execution model. The method can ensure that the real-time memory allocation model runs when necessary, reduces automobile energy consumption, and reduces the degree of memory fragmentation by flexibly switching a variety of memory allocation rules, thereby achieving real-time response and adaptive real-time memory allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0070] Figure 1 A flowchart of a method for real-time memory allocation of a multi-virtual machine and multi-process architecture of an automobile domain controller of the present invention;

[0071] Figure 2 It is the three-level memory mapping relationship and memory hierarchy of host machine, virtual machine and computing task;

[0072] Figure 3 This is the structural design diagram of the Q-value neural network;

[0073] Figure 4 This is the training principle diagram of the Q-value neural network;

[0074] Figure 5 A diagram of the steps to allocate memory in real time. DETAILED DESCRIPTION

[0075] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0076] like Figure 1 As shown, the real-time memory allocation method of the multi-virtual machine and multi-process architecture of the automobile domain controller of this embodiment includes the following steps:

[0077] Step 1: Deploy the real-time memory allocation model of the multi-virtual machine and multi-process architecture of the automotive domain controller to the vehicle domain controller.

[0078] In this embodiment, a method for constructing a real-time memory allocation model of a multi-virtual machine and multi-process architecture of a vehicle domain controller includes the following steps.

[0079] 11) The real-time memory allocation process for multi-virtual machines and multi-process architecture in the automotive domain controller is transformed into a Markov decision process. At time step t, the automotive domain controller operating environment determines the actions A of multiple agents. t Memory is allocated in real time, and the environment state is determined by S t Change to S t+1 , giving the reward value R for this action t And feedback to the agent, so that it can adjust its allocation strategy π in real time. The real-time memory allocation model analyzes and collects the state information of the computing tasks and memory every time a new computing task is generated, and then takes appropriate allocation actions according to these state information and allocation strategies. The transformation of the Markov decision process requires the definition of the real-time memory allocation process of this instance, and the construction of the Markov decision model (S, A, π, R, γ); where: S represents the state space that can characterize the domain controller computing tasks and physical memory state information in the real-time memory allocation process; A represents the action space composed of a variety of specific allocation rules when the automobile domain controller resource management mechanism allocates memory; π represents the allocation strategy of the domain controller resource management mechanism, which is based on multi-agent deep reinforcement learning training; R represents the reward function, which needs to reflect the goal of the allocation model, that is, to minimize the total computing delay of all on-board computing tasks of the car, especially the highly sensitive delay computing tasks; γ∈(0,1) represents the reward discount rate, which reflects the ability of the task scheduling agent to foresee the future. The larger it is, the more long-term rewards it will get.

[0080] 12) Define the state space S, action space A, reward function R and allocation strategy π:

[0081] (1) The state space S is defined as:

[0082] S={S TASK +S MEMORY}

[0083] Where: S TASK represents the computing task state information matrix; S MEMORY Represents the memory state information matrix.

[0084] like Figure 2 As shown, the memory hierarchy of this embodiment is memory as a whole, memory block, and memory page. The memory page is the smallest unit representing memory status information. Figure 3 As shown, the memory state is three-dimensional. block Indicates the number of memory blocks divided by the resource management mechanism of the automotive domain controller, and the number of memory pages contained in a memory block:

[0085] Numpage =Size block / Size page

[0086] Where: Num page Indicates the number of memory pages contained in a memory block; Size block Indicates the memory size of a memory block; Size page Indicates the memory size of a memory page.

[0087] The third dimension is the amount of state information that the memory needs to represent. Due to the three-level memory management mechanism of the automotive central domain controller, the information that the memory needs to represent is That is, in this embodiment, the memory state information matrix is ​​expressed as:

[0088]

[0089] in: and They respectively represent the virtual machine code and computing task code to which the j-th memory page of the ith memory block belongs; these two code values ​​are generally two-digit numbers, corresponding to the virtual machine sequence number and computing task sequence number, respectively. For example, the code of the third virtual machine is 03. The maximum value is the number of virtual machines Num vm , Indicates that the memory does not participate in the calculation of the task. The maximum value is the code The maximum number of tasks that a virtual machine can perform at the same time Indicates that the host machine has not allocated the memory to the virtual machine.

[0090] like Figure 2 As shown, and Respectively represent the memory block and memory page code of the memory in its corresponding virtual machine, and its maximum value is the same as Num block With Num page Similarly, it is related to the memory page and memory block sizes specified in the virtual machine. hour, and Must be equal to 00.

[0091] Calculate the task status S TASK It is necessary not only to characterize the status information of the current computing task, but also to characterize the information of the computing tasks that can be foreseen in the future. This embodiment reserves k rows of empty slots for each computing task to characterize the future computing tasks predicted based on the computing task. It is also three-dimensional, and the three dimensions correspond to Num vm , The amount of information that needs to be represented in the computational task That is, in this embodiment, the task status information matrix S is calculated TASK It is expressed as:

[0092]

[0093] in: It indicates the time from the mth virtual machine to the moment when it processes its nth computing task. Indicates that the computing task has been running for a period of time. Indicates that the calculation task will take some time to start; Indicates the remaining time required to complete the calculation task; P mn Indicates the priority level of the task; Indicates the task code that has a dependency relationship with this task; since tasks between different virtual machines also have dependencies, is the virtual machine code corresponding to the dependent task. Since tasks between different virtual machines also have dependencies, It is a four-digit number containing the virtual machine code; Indicates the maximum delay allowed for processing this task.

[0094] (2) Action Space A

[0095] The action space A is a vector. The host machine allocates memory between virtual machines according to the output value of the corresponding agent combined with A. Each virtual machine allocates memory between computing tasks on the virtual machine according to the output value of the corresponding agent combined with A. Specifically, in this embodiment, the action space A is defined as:

[0096] A={rule 1 ,rule 2 ,rule 3 ,rule 4 ,rule 5 ,rule 6}

[0097] The action space A contains 6 rules, among which: 1 Indicates that if the first adaptation rule condition is not met, the best rule for the fragment is executed; rule 2 Indicates the first rule of the fragment; rule 3 Indicates the best fit rule; rule 4 Indicates that the best fit rule is considered first, and then the fragment first rule is considered; rule 5 Indicates that the best adaptation rule is considered first; rule 6Indicates that the fragment first rule is considered first.

[0098] First fit rule: When a single continuous memory is satisfied, traverse the free area of ​​the memory, find the first free block that is large enough, and allocate it to the requesting virtual machine or process;

[0099] Optimal fragmentation rule: When the first fit rule is not met, the largest possible memory fragment is allocated to the requested virtual machine or process;

[0100] First fragment rule: When the first adaptation rule cannot be met, the memory is traversed and the first few memory fragments that meet the requirements are allocated to the requesting virtual machine or process;

[0101] Best fit rule: When a single continuous memory is satisfied, traverse the free area of ​​the memory to find the free block with the most suitable size, otherwise perform the best fragmentation rule.

[0102] (3) Reward function R

[0103] Different reward functions are designed according to different real-time memory allocation goals. The allocation goal of this implementation example is to reduce the total computing delay of all on-board computing tasks of the car as much as possible, especially the total computing delay of highly sensitive delay computing characters. When action A makes the memory allocation closer to this goal, the reward value calculated by the reward function should be larger. In this embodiment, the reward function R is defined as:

[0104]

[0105] in: When the process memory is allocated at time step t, the environment state S t Change to S t+1 The reward value calculated by the agent of the mth virtual machine through the reward function; L represents the number of tasks completed by the mth virtual machine during the period from time step t to time step t+1; Indicates the time taken to complete the lth computing task on the mth virtual machine; represents the maximum delay time of the lth computing task completed by the mth virtual machine from time step t to time step t+1; P ml Indicates the priority level of the lth computing task completed by the mth virtual machine from time step t to time step t+1. The larger the priority level value, the higher the priority of the computing task;

[0106] (4) The allocation strategy π is defined as: t What action to take next? t A strategy to maximize cumulative rewards;

[0107] Optimizing the allocation strategy π(A k |Sk ) is used to make the action value function Q π (S t ,A t ) is minimal, and the action value function Q π (S t ,A t ) is equal to in state S t Take action A t The expected value of the cumulative discounted reward is expressed as:

[0108] Q π (S t ,A t )=E π {R t +γQ π (S t+1 ,A t+1 )}

[0109] Among them, E π represents the expected value of the cumulative discounted reward obtained under the allocation strategy π.

[0110] 13) Train the deep Q network to fit the value function Q(S,A).

[0111] Since the state space of real-time memory allocation for multiple virtual machines and multi-task processes of a domain controller is very large, take the real-time memory allocation for multi-task processes of a single operating system as an example. Assume that at the beginning of real-time memory allocation, 10 computing tasks without dependencies start to calculate in parallel, and each initial computing task has 5 computing tasks with linear dependencies waiting to start computing and memory allocation. Each computing task has 6 allocation rules to choose from, and the possibility of the state matrix is ​​(10×6) 5 . In this implementation example, the actual memory allocation is the real-time allocation of memory between the host machine and multiple virtual machines, and the computing tasks between different virtual machines may have dependencies, and the completion time of the computing tasks between each virtual machine must be asynchronous, so the possibility of the state matrix in the actual environment must be more. Since the Q value table of traditional Q-learning cannot store such a number of Q(S, A), it is necessary to introduce a deep learning neural network system to replace the Q value table. Since the action space of this implementation example is composed of memory allocation rules and the output actions of the neural network are discrete, the deep neural network structure of the intelligent body of this example uses a deep Q network (Deep Q-Network, DQN) to fit the value function Q(S, A). Specifically, in this embodiment, with the Q network outputting the maximum value as the goal, the best action A selected at time step t t for:

[0112]

[0113] Where: Q(S,A) represents the value function of the deep Q network.

[0114] Since the input of the neural network is the state information S composed of S TASK and S MEMORY Since the state information S is composed of values ​​of different magnitudes, normalization is used to process the input to avoid the disappearance and explosion of the neural network gradient. TASK and S MEMORY is a three-dimensional matrix, so the structure of the neural network needs to include convolutional layers, pooling layers, and tiling layers. Figure 3 As shown, S TASK and S MEMORY After passing through the convolution layer, pooling layer, flattening layer, normalization processing, and full connection layer on each branch, the output is Q(S,A).

[0115] After completing the definition of the core elements such as state, action, and reward in the Markov decision process and the structural design of the deep neural network, the construction of the memory implementation allocation model has been basically completed. Next, the intelligent agent with the neural network as the main body is deployed on the host and virtual machine of the automotive domain controller computing system, and then the model training based on multi-agent deep reinforcement learning is carried out. Specifically, in this embodiment, the training steps of the deep Q network are as follows.

[0116] 131) Initialize the algorithm hyperparameters of the Q network, including learning rate, discount factor, exploration rate, experience replay pool size, batch size, number of training steps, hyperparameters of the Adam optimizer, random seed, and target network update frequency; initialize the weight parameters of the main body of each intelligent agent. Figure 4 As shown, the environment state s at time step t = 0 is set according to the random seed 0 ; Among them: Agent 0 Deployed on the host computer of the automotive domain controller computing system, the intelligent agent m Deployed on virtual machine m. In addition, during the training of the real-time memory allocation model, the dynamic changes of the computing task type during its execution are simulated through the random seeds of the algorithm hyperparameters to ensure that the real-time memory allocation model can cope with the dynamic factors of changes in the computing task type.

[0117] 132) According to the set exploration rate (the exploration rate decays as the time step increases), the agents on the host and each virtual machine randomly generate the neural network of the current training step; obtain the best action of the host and each virtual machine at the current time step t And interact with the environment to get the state s t+1And feedback reward value The data generated Store to Agent 0 In the experience replay pool, the data generated Store to Agent m Experience replay pool; among them: Represents the host agent 0 The best action to output, Agent m The best action to output; Represents the environment to give feedback to the Agent 0 The reward value of Represents the reward value fed back by the environment to the virtual machine m agent.

[0118] 133) Determine whether the data in the experience replay pool of any intelligent agent has reached the set amount; if so, execute step 134); if not, set t=t+1 and execute step 132).

[0119] 134) For the agents whose experience replay pool data reaches the set amount, randomly sample the batch amount of data from the corresponding experience replay pool data to the Q network and the target network. The Q network uses the state-action pair data (S t ,A t ) predicts the corresponding Q value; the target network uses the state S t+1 The data calculates the corresponding maximum Q value in various actions.

[0120] 135) Parameter update:

[0121] Use the loss function L to reversely calculate the loss function gradient of each parameter θ in the deep Q network To update the weight parameters of each node in the Q network. Determine whether the number of random Batch samplings after the last parameter update on the target network is equal to the preset number H: If so, update the parameters of the target network; if not, do not update the parameters of the target network.

[0122] In this embodiment, the parameters of the deep Q network approximator are updated by instant differential learning. The principle is:

[0123]

[0124] Among them, α represents the learning rate.

[0125] Furthermore, Q(S,A,θ) represents the action value function fitted by the deep neural network, and the loss function L is defined as:

[0126]

[0127] Among them, θ z represents the zth parameter in the deep Q network; Indicates that in state S t When taking action A t Then extract the reward R from the domain controller t and in state S t+1 The expected value of the cumulative discounted reward when taking the highest Q-value action; y i represents the target value, and:

[0128]

[0129] in, Indicates that the reward R is extracted from the domain controller t and in state S t+1 The expected value of the cumulative discounted reward when taking the highest Q-value action.

[0130] 136) Determine whether the time step reaches the set maximum time step: if so, the deep Q network training is completed; if not, set t = t + 1, and execute step 142).

[0131] In the preferred implementation of this embodiment, the preliminary training process of the real-time memory allocation model is completed by the intelligent connected vehicle manufacturer, and the training is completed through the human-vehicle-road digital twin simulator. The intelligent connected vehicle records the real-time operation data during driving, and then transmits it to the cloud for retraining, so as to update the real-time memory allocation model of the multi-virtual machine and multi-process task of the domain controller of the intelligent connected vehicle described in this embodiment. After the training is completed, the car uses the real-time memory allocation model of this embodiment to allocate memory, such as Figure 5 shown.

[0132] Step 2: Real-time memory allocation for multiple virtual machines and multi-task processes on the domain controller

[0133] 21) Task static instruction pre-ordering and dynamic ordering during execution:

[0134] Classify computing tasks into divisible tasks and indivisible tasks, and split divisible tasks into indivisible tasks; divide all indivisible tasks into independent tasks and dependent tasks; sort computing tasks according to priority levels; for computing tasks with the same priority level, give priority to computing tasks with a small maximum allowable delay; for dependent tasks, sort them according to the dependency relationship; during the execution of computing tasks, the computing system performs real-time dynamic sorting of computing tasks;

[0135] Specifically, in this embodiment, at the beginning of the execution task generation, the computing tasks are divided into two dimensions through static instructions (control flow analysis, instruction feature analysis, data dependency analysis) and the computing tasks are assigned priority levels. One dimension is separable tasks and indivisible tasks, and the other dimension is independent and dependent. For separable tasks, they are split into indivisible tasks. Then these indivisible computing tasks are divided into independent and dependent types. Then the computing tasks are sorted according to the priority level. For computing tasks with the same priority level, computing tasks with small maximum allowable delays are given priority. For dependent tasks, they are sorted according to the dependency relationship. During the execution of the computing tasks, the computing system analyzes the type of computing tasks in real time by observing the state of the program at runtime (data flow monitoring, resource usage monitoring, task decomposition detection, task behavior detection), the execution order of tasks, input and output data flows, etc., thereby dynamically adjusting the sorting of computing tasks.

[0136] 22) Determine whether the number of free memory blocks is greater than or equal to the set number threshold: if so, perform real-time memory allocation for multiple virtual machines and multi-task processes of the domain controller, and execute step 24); if not, do not perform real-time memory allocation for multiple virtual machines and multi-task processes of the domain controller, and execute step 23).

[0137] 23) Allocate memory according to the first adaptation rule: if the first adaptation rule cannot find a suitable free memory block, execute step 24); if the first adaptation rule can find a suitable free memory block, execute step 25);

[0138] 24) Inputting the status information of memory and computing tasks into the real-time memory allocation model of the multi-virtual machine and multi-process architecture of the automotive domain controller, outputting the Q values ​​of various real-time memory allocations, and selecting the memory allocation rule corresponding to the maximum Q value for memory allocation.

[0139] Specifically, for the intelligent agent deployed on the host machine 0 When the physical memory cannot meet the memory requirements of the currently running computing tasks and the computing tasks started at the current time step, the memory management mechanism on the host machine follows the Agent 0 Given Allocate physical memory to the virtual machine; for the agent deployed on the virtual machine m m , when virtual machine m has a new computing task requesting memory allocation at time step, virtual machine m follows Agent m Given Allocate the virtual memory of the virtual machine to the new computing process; for the virtual machine that has no new computing tasks generated in the current time step or the host machine that has no physical memory allocation requirements, the best action given by the corresponding agent No actual memory allocation is triggered.

[0140] 25) After the central domain controller computing system completes memory allocation according to the allocation rules given by the real-time memory allocation model, the state information of the memory and computing tasks changes, and the reward value is calculated. After obtaining the reward value, the data {S t ,A t ,R t ,S t+1} is stored in the experience replay pool of the corresponding agent, and the data in the experience replay pool is uploaded to the cloud server for retraining of the memory real-time allocation model.

[0141] Step 3: Determine whether there is a computing task to be executed: if so, run the memory real-time allocation model and execute step 2; if not, do not run the memory real-time allocation model.

[0142] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A real-time memory allocation model for a multi-virtual machine and multi-process architecture of an automotive domain controller, characterized by: The construction method includes the following steps: 11) The real-time memory allocation process for multi-virtual machines and multi-process architectures in automotive domain controllers is transformed into a Markov decision process, and a Markov decision model (S, A, π, R, γ) is constructed; where: S represents the state space that can characterize the domain controller computing tasks and physical memory state information in the real-time memory allocation process; A represents the action space composed of multiple specific allocation rules when the automotive domain controller resource management mechanism allocates memory; π represents the allocation strategy of the domain controller resource management mechanism; R represents the reward function; γ∈(0,1) represents the reward discount rate; 12) Define the state space S, action space A, reward function R and allocation strategy π: The state space S is defined as: S={S TASK +S MEMORY } Where: S TASK represents the computing task state information matrix; S MEMORY represents the memory state information matrix; and: in: and They respectively represent the virtual machine code and computing task code to which the j-th memory page of the i-th memory block belongs; and The codes respectively represent the memory block and memory page of the memory in its corresponding virtual machine; in: It indicates the time from the mth virtual machine to the moment when it processes its nth computing task. Indicates that the computing task has been running for a period of time. Indicates that the calculation task will take some time to start; Indicates the remaining time required to complete the calculation task; P mn Indicates the priority level of the task; Indicates the task code that has a dependency relationship with this task; since tasks between different virtual machines also have dependencies, The virtual machine code corresponding to the dependent task; Indicates the maximum delay allowed for processing the task; The action space A is defined as: A={rule1,rule2,rule3,rule4,rule5,rule6} Among them: rule1 means executing the best rule of fragments when the first adaptation rule is not met; rule2 means the first rule of fragments; rule3 means the best adaptation rule; rule4 means first considering the best adaptation rule, then the first rule of fragments; rule5 means first considering satisfying the best adaptation rule; rule6 means first considering the first rule of fragments; First fit rule: When a single continuous memory is satisfied, traverse the free area of ​​the memory, find the first free block that is large enough, and allocate it to the requesting virtual machine or process; Optimal fragmentation rule: When the first fit rule is not met, the largest possible memory fragment is allocated to the requested virtual machine or process; First fragment rule: When the first adaptation rule cannot be met, the memory is traversed and the first few memory fragments that meet the requirements are allocated to the requesting virtual machine or process; Best fit rule: When a single continuous memory is satisfied, traverse the free area of ​​the memory to find the free block with the most suitable size, otherwise, perform the best fit rule for fragmentation; The reward function R is defined as: in: When the process memory is allocated at time step t, the environment state S t Change to S t+1 The reward value calculated by the agent of the mth virtual machine through the reward function; L represents the number of tasks completed by the mth virtual machine during the period from time step t to time step t+1; Indicates the time taken to complete the lth computing task on the mth virtual machine; represents the maximum delay time of the lth computing task completed by the mth virtual machine from time step t to time step t+1; P ml Indicates the priority level of the lth computing task completed by the mth virtual machine from time step t to time step t+1. The larger the priority level value, the higher the priority of the computing task; The allocation strategy π is defined as: in state S t What action to take next? t A strategy to maximize cumulative rewards; Optimizing the allocation strategy π(A k |S k ) is used to make the action value function Q π (S t ,A t ) is minimal, and the action value function Q π (S t ,A t ) is equal to in state S t Take action A when t The expected value of the cumulative discounted reward is expressed as: Q π (S t ,A t )=E π {R t +γQ π (S t+1 ,A t+1 )} Among them, E π represents the expected value of the cumulative discounted reward obtained under the allocation strategy π; 13) Train the deep Q network to fit the value function Q(S,A).

2. The real-time memory allocation model of the multi-virtual machine and multi-process architecture of the automotive domain controller according to claim 1 is characterized by: In step 13), the training steps of the deep Q network are: 131) Initialize the algorithm hyperparameters of the Q network, including learning rate, discount factor, exploration rate, experience replay pool size, batch size, number of training steps, hyperparameters of the Adam optimizer, random seed, and target network update frequency; initialize the weight parameters of the main body of each intelligent agent, where: the intelligent agent Agnet0 is deployed on the host machine of the automotive domain controller computing system, and the intelligent agent Agent m Deployed on virtual machine m; 132) According to the set exploration rate, the agents on the host machine and each virtual machine randomly generate the neural network of the current training step; obtain the best action of the host machine and each virtual machine at the current time step t And interact with the environment to get the state s t+1 And feedback reward value The data generated Stored in Agnet0's experience replay pool, the generated data Store to Agnet m Experience replay pool; among them: Represents the best action output by the host Agent0. Agent m The best action to output; Represents the reward value fed back to Agent0 by the environment; Represents the reward value fed back to the virtual machine m agent by the environment; 133) Determine whether the data in the experience replay pool of any agent has reached the set amount; if so, execute step 134); if not, set t=t+1 and execute step 132); 134) For the agents whose experience replay pool data reaches the set amount, randomly sample the batch amount of data from the corresponding experience replay pool data to the Q network and the target network; the Q network uses the state-action pair data (S t ,A t ) predicts the corresponding Q value; the target network uses the state S t+1 The data calculates the corresponding maximum Q value in various actions; 135) Parameter update: Use the loss function L to reversely calculate the loss function gradient of each parameter θ in the deep Q network To update the weight parameters of each node in the Q network; Determine whether the number of random Batch sampling after the last parameter update on the target network is equal to the preset number H: if so, update the parameters of the target network; if not, do not update the parameters of the target network; 136) Determine whether the time step reaches the set maximum time step: if so, the deep Q network training is completed; if not, set t = t + 1, and execute step 142).

3. The real-time memory allocation model of the multi-virtual machine and multi-process architecture of the automotive domain controller according to claim 2 is characterized by: In step 132), the optimal action A is selected at time step t with the maximum output of the Q network as the goal. t for: Where: Q(S,A) represents the value function of the deep Q network.

4. The real-time memory allocation model of the multi-virtual machine and multi-process architecture of the automotive domain controller according to claim 2 is characterized by: In step 135), the parameters of the deep Q network approximator are updated by instant differential learning, the principle is: Among them, α represents the learning rate.

5. The real-time memory allocation model of the multi-virtual machine and multi-process architecture of the automotive domain controller according to claim 2 is characterized by: Let Q(S,A,θ) represent the action value function fitted by the deep neural network, and the loss function L is defined as: Among them, θ z represents the zth parameter in the deep Q network; Indicates that in state S t When taking action A t Then extract the reward R from the domain controller t and in state S t+1 The expected value of the cumulative discounted reward when taking the highest Q-value action; y i represents the target value, and: in, Indicates that the reward R is extracted from the domain controller t and in state S t+1 The expected value of the cumulative discounted reward when taking the highest Q-value action.

6. A method for real-time memory allocation of a multi-virtual machine and multi-process architecture of an automobile domain controller, characterized in that: The steps include: Step 1: deploying the real-time memory allocation model of the multi-virtual machine and multi-process architecture of the automobile domain controller as described in any one of claims 1 to 5 to the vehicle domain controller; Step 2: Real-time memory allocation for multiple virtual machines and multi-task processes on the domain controller 21) Task static instruction pre-ordering and dynamic ordering during execution: Classify computing tasks into divisible tasks and indivisible tasks, split divisible tasks into indivisible tasks; divide all indivisible tasks into independent tasks and dependent tasks; Sort computing tasks by priority level; for computing tasks with the same priority level, give priority to computing tasks with smaller maximum allowable delay; for dependent tasks, sort them by dependency relationship; During the execution of computing tasks, the computing system dynamically sorts the computing tasks in real time; 22) Determine whether the number of free memory blocks is greater than or equal to a set number threshold: if so, perform real-time memory allocation for multiple virtual machines and multi-task processes of the domain controller, and execute step 24); if not, do not perform real-time memory allocation for multiple virtual machines and multi-task processes of the domain controller, and execute step 23); 23) Allocate memory according to the first adaptation rule: if the first adaptation rule cannot find a suitable free memory block, execute step 24); if the first adaptation rule can find a suitable free memory block, execute step 25); 24) Inputting the state information of memory and computing tasks into the real-time memory allocation model of the multi-virtual machine and multi-process architecture of the automobile domain controller, outputting the Q values ​​of various real-time memory allocations, and selecting the memory allocation rule corresponding to the maximum Q value for memory allocation; 25) After the central domain controller computing system completes memory allocation according to the allocation rules given by the real-time memory allocation model, the state information of the memory and computing tasks changes, and the reward value is calculated; Step 3: Determine whether there is a computing task to be executed: if so, run the memory real-time allocation model and execute step 2; if not, do not run the memory real-time allocation model.

7. The real-time memory allocation method for a multi-virtual machine and multi-process architecture of a vehicle domain controller according to claim 6 is characterized by: In step 24), for the agent Agent0 deployed on the host machine, when the physical memory cannot meet the memory requirements of the currently running computing tasks and the computing tasks started at the current time step, the memory management mechanism on the host machine is based on the memory requirements given by Agent0. Allocate physical memory to the virtual machine; for the agent deployed on the virtual machine m m , when virtual machine m has a new computing task requesting memory allocation at time step, virtual machine m follows Agent m Given Allocate the virtual memory of the virtual machine to the new computing process; for the virtual machine that has no new computing tasks generated in the current time step or the host machine that has no physical memory allocation requirements, the best action given by the corresponding agent No actual memory allocation is triggered.

8. The real-time memory allocation method for a multi-virtual machine and multi-process architecture of a vehicle domain controller according to claim 6 is characterized by: In the step 25), after obtaining the reward value, the data {S t ,A t ,R t ,S t+1 } is stored in the experience replay pool of the corresponding agent, and the data in the experience replay pool is uploaded to the cloud server for retraining of the memory real-time allocation model.

Images