Industrial automation control method and system based on virtual engine

Abstract

The invention discloses an industrial automation control method and system based on a virtual engine, relates to the technical field of virtual machine migration, and is used for solving the problem of how to quickly evaluate and select a proper migration target in a plurality of virtual machines and production lines. The state of the migration target is comprehensively evaluated according to an Ac algorithm, and whether virtual machine migration needs to be conducted or not is predicted; executing a decision of an Ac algorithm, performing virtual machine migration or keeping existing configuration, and ensuring reasonable resource allocation and meeting production requirements; the resource load and the stability of the production line can be comprehensively considered in the production environment, so that the system is more accurate and reasonable during migration decision making, and resource overload and unstable migration operation are effectively avoided. Through the intelligent control of the migration tendency, the system can significantly improve the resource utilization efficiency and the overall stability, enhance the adaptability to a complex dynamic environment, and achieve more efficient resource scheduling and optimization under variable production requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual machine migration. More specifically, the present invention relates to an industrial automation control method and system based on a virtual engine. Background Art

[0002] With the promotion of Industry 4.0 and intelligent manufacturing, enterprises' requirements for production efficiency and resource management are constantly increasing. The application of virtualization technology provides enterprises with a more flexible resource allocation method. However, in actual operation, especially during the addition and withdrawal of production lines, the effective migration of resources still faces many challenges.

[0003] In modern industrial production, the addition or withdrawal of production lines is often accompanied by adjustments to resource allocation. As a key resource in the production environment, the migration strategy of virtual machines needs to be dynamically adjusted according to changes in production line requirements. Traditional virtual machine migration decision-making methods are often based on static rules and cannot effectively cope with frequent changes in production requirements and resource pressure. When adding a new production line, existing virtual machines need to be migrated to the new environment, and when a production line is withdrawn, it is necessary to ensure that the resources in use can be smoothly transferred to other production lines. This process requires accurate timing judgment to avoid affecting production. Therefore, in multiple virtual machines and production lines, how to quickly evaluate and select a suitable migration target is a complex decision-making problem.

[0004] In view of the above problems, the present invention proposes a solution. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide an industrial automation control method and system based on a virtual engine to solve the problems raised in the above background art.

[0006] To achieve the above object, the present invention provides the following technical solutions: An industrial automation control method based on a virtual engine, comprising the following steps: When a production line is added or withdrawn, obtain the working condition state information of each production line; Comprehensively evaluate the state of the migration target according to the Actor-Critic algorithm, and predict whether virtual machine migration is required; Execute the decision of the Actor-Critic algorithm to perform virtual machine migration or maintain the existing configuration to ensure reasonable resource allocation and meet production requirements; The system continuously monitors the state of the migrated production line, updates through the learning of the Actor-Critic algorithm, and updates the working condition state information of each production line.

[0007] In a preferred embodiment, the working condition state information of each production line includes resource utilization rate, virtual machine migration frequency, production line performance volatility value, and migration target load prediction value.

[0008] In a preferred embodiment, input the current state space S into the neural network to obtain the state value function V(s); Calculate the time difference error δ, and use the time difference error δ to update the neural network parameters θ of the state value function; Use policy gradient to update the policy parameter ϕ; Repeat the above steps until convergence or the maximum number of iterations is reached; According to the current policy π ϕ (a∣s) to select whether to perform migration.

[0009] In a preferred embodiment, in the decision-making of executing the Actor-Critic algorithm, the success value reward c in the reward function R is further optimized. The success value reward c is set as a dynamic variable, which gradually changes with the working condition state information of the production line; the improved success value reward c is jointly determined according to the virtual machine migration frequency of the production line and the migration target load prediction value.

[0010] In a preferred embodiment, if there are multiple production lines available for migration selection, the optimal production line is selected as the first migration target according to the working condition state information of the production line: Comprehensively determine the first migration target according to the resource utilization rate of the production line and the production line performance volatility value. When the first migration production line is screened and determined, only the migration strategy for the first migration production line is generated at this time.

[0011] In a preferred embodiment, when the first migration production line is screened and determined, the first migration target coefficient is used as an important variable in the state space of the Actor-Critic algorithm and incorporated into the state space S.

[0012] An industrial automation control system based on a virtual engine further includes a data acquisition module, an Actor-Critic algorithm module, and a migration target screening module, and the modules are signal-connected to each other; The data acquisition module is used to acquire the working condition state information of each production line to better formulate the migration strategy; The Actor-Critic algorithm module is used to formulate the migration strategy according to the working condition state information of each production line; The Actor-Critic algorithm module further includes a dynamic adjustment unit, which is used to dynamically adjust the success value reward in the reward function R to facilitate better adjustment of the migration strategy tendency according to the actual situation; The migration target screening module is used to screen the optimal migration production line and make the migration strategy decision.

[0013] Technical effects and advantages of an industrial automation control method and system based on a virtual engine according to the present invention: The present invention can flexibly select whether to perform virtual machine migration according to the addition or withdrawal of production lines, improving the intelligent level of resource scheduling. And the system is guided to perform intelligent virtual machine migration by dynamically adjusting the reward score. The Actor is responsible for generating the migration strategy and continuously optimizing the decision according to the feedback given by the Critic (calculated through TD Error). The Critic uses a neural network to fit the state value function V(s) and guides the long-term migration strategy of the system by updating the estimation of the state value. This method can help the system achieve resource optimization in a dynamic environment, improving the migration success rate and resource allocation efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flowchart of an industrial automation control method based on a virtual engine according to the present invention; Figure 2 It is a schematic structural diagram of an industrial automation control system based on a virtual engine according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0016] An industrial automation control method and system based on a virtual engine according to the present invention proposes a dynamic virtual machine migration strategy based on the Actor-Critic algorithm for the problem of virtual machine migration when a production line is added or withdrawn. This solution first calculates whether each production line is suitable as the target of virtual machine migration through the reward score coefficient (RSC). At the same time, the migration tendency is adjusted by combining the dynamic success value reward, so as to achieve intelligent migration under different resource load conditions.

[0017] After determining the primary migration target, the Actor-Critic algorithm is used for decision-making. The demand state of the production line and the reward score coefficient are comprehensively analyzed to determine whether to perform virtual machine migration. The algorithm determines the migration action through the policy network (Actor) and evaluates the effect of the migration action by the value network (Critic). The reward function is adjusted through the dynamic reward score coefficient, making the migration behavior more in line with the actual production requirements and avoiding excessive pressure on production lines with high loads or frequent migrations.

[0018] Example 1, as Figure 1As shown in the figure, an industrial automation control method based on a virtual engine according to the present invention includes the following steps: Step S1: When a production line is newly added or withdrawn, obtain the working condition state information of each production line; Step S2: Comprehensively evaluate the state of the migration target according to the Actor-Critic algorithm, and predict whether virtual machine migration is required; Step S3: Execute the decision of the Actor-Critic algorithm to perform virtual machine migration or maintain the existing configuration to ensure reasonable resource allocation and meet production requirements; Step S4: The system continuously monitors the state of the production line after migration, updates through the learning of the Actor-Critic algorithm, and updates the working condition state information of each production line.

[0019] Specifically, in Step S1, the working condition state information of each production line includes resource utilization rate, virtual machine migration frequency, production line performance volatility value, and migration target load prediction value.

[0020] The resource utilization rate reflects the current resource load of the production line, including CPU, memory, storage, etc., and can be the weighted average of resources such as CPU and memory. A production line with a high resource utilization rate usually has a high load and relatively tight resources, so it is not suitable to be a migration target. On the contrary, a production line with a low resource utilization rate has more idle resources and is suitable for receiving migrated virtual machines.

[0021] If a certain production line has a high resource utilization rate, it means that the computing, storage and other resources of this production line are close to saturation. Continuing to migrate virtual machines to this production line may cause resource overload and affect the overall performance. Therefore, when the system selects the primary migration target, it will avoid selecting production lines with a high resource utilization rate. A production line with a low resource utilization rate indicates that there are more available resources on this production line and it can withstand more virtual machine migrations. Therefore, a production line with a low resource utilization rate is more likely to become a migration target.

[0022] The virtual machine migration frequency represents the number of virtual machines migrated in or out of a certain production line per unit time. A production line with a high migration frequency participates in multiple virtual machine migrations in a short period of time. Frequent migration operations will consume network bandwidth, affect system performance, and bring additional resource management overhead. Therefore, a production line with a high migration frequency is usually not suitable as a new migration target.

[0023] The production lines with a higher migration frequency have participated in virtual machine migrations multiple times within a unit of time. Further increasing the migration operations may cause an overburden and affect the stability of the production line. The system will assign these production lines a higher reward score coefficient and reduce their priority as migration targets. The production lines with a lower migration frequency participate in migrations less frequently, and the resource scheduling is relatively stable, making them suitable for receiving new virtual machines. Therefore, the production lines with a lower migration frequency are more likely to be selected by the system in the selection of the primary migration target.

[0024] The production line performance volatility value represents the degree of change in the resource utilization rate of the production line over a period of time, that is, the stability or fluctuation range of the resource usage load. A high production line performance volatility value means that the resource usage of the production line often fluctuates significantly, while a low volatility indicates relatively stable resource usage. This indicator is very important for evaluating the adaptability and stability of the migration target. During the virtual machine migration process, the system needs to ensure that the target production line to which the migration is made can stably bear the newly added resource load.

[0025] The production line performance volatility value is usually measured by statistical methods, such as variance or standard deviation. These indicators can quantify the fluctuations in resource usage over a period of time. Specifically, in this embodiment, the production line performance volatility value σi can be calculated as the standard deviation of the resource utilization rate (such as CPU, memory, etc.) of production line i.

[0026] The prediction of the migration target load value means that before performing the virtual machine migration operation, the system estimates the load situation of the target production line after the migration operation is completed. This prediction helps the system judge the impact of the migration operation on the resources of the target production line, so as to decide whether to perform the migration and the priority of the migration operation. When selecting a migration target, the system not only needs to consider the current resource usage situation, but also needs to evaluate whether the target production line can bear the new load after the migration is completed.

[0027] In order to estimate the resource utilization situation of the target production line after the migration is completed, the calculation formula for the predicted migration target load value P can be as follows: P = Ucurrent,target + Estimated Migration Load; In the formula, Ucurrent,target is the resource utilization rate of the current target production line, which can be the weighted average of resources such as CPU and memory; Estimated Migration Load is the increase in load brought by the migration operation, usually estimated according to the resource requirements of the migrated virtual machine. For example, if the migrated virtual machine requires 20% of the CPU and 10% of the memory, these requirements will be added to the current resource utilization rate of the target production line.

[0028] Furthermore, the Actor-Critic algorithm is a policy-based reinforcement learning algorithm. It selects actions through a policy function (Actor) and evaluates the quality of states through a value function (Critic). Combining this algorithm, it predicts whether to perform virtual machine migration based on the working condition state information of the production line. The specific steps are as follows: Define the state space S; The state space S contains all important variables that describe the current state of the migration target (production line), such as: The average migration time cost Ti of the production line: That is, the average migration time of production line i within a certain past period, reflecting the execution efficiency of the production line migration. A production line with a higher migration time cost may drag down the overall migration speed of the system.

[0029] The production line task type Ki: That is, the type of production line task (such as batch processing tasks, real-time tasks, etc.) to evaluate whether the migration is appropriate. Some task types have high resource requirements or are not suitable for frequent migration, which can be used as a reference for the migration decision; Define the action space A; The action space includes two choices: a = 1: Execute the migration operation; a = 0: Do not execute the migration.

[0030] Set the reward function R; The reward function R is used to feedback the effect of the migration operation, taking into account the number of migrations and the production line stability. Here, the reward function is designed as: ; In the formula, c is the success value reward for successful migration; d is the penalty for migration failure or insufficient resources.

[0031] Calculate the state value function V(s); The state value function V(s) represents the future expected cumulative reward of the system under the state space S.

[0032] The calculation expression can be: V(s)=[R t +γR t+1 +γ 2 R t+1 +...|s t =s]; In the formula, R t is the immediate reward function, s t is the current state space, and γ is the discount factor used to decay future rewards.

[0033] To solve the state value function V(s), in this embodiment, it is approximated through a neural network. The network input is set as the state s, and the output is the estimated V(s).

[0034] In the neural network structure, the input layer is the state space S, which contains all important variables describing the current state of the transfer target (production line); the hidden layer is used to extract state features; the output layer is the state value function V(s), representing the total future return in the current state.

[0035] Calculate the time difference error and update the state value function using the time difference error; The update of the state value function is based on the time difference error (TD Error), which is used to measure the gap between the currently estimated state value V(s) and the sum of future rewards. The calculation expression of the time difference error δ is: δ = R + γV(s’) - V(s); R is the immediate reward function at the current time step, that is, R t ; V(s’) is the value estimate of the next state s’ after performing the operation.

[0036] Update the neural network parameter θ through the time difference error δ, and optimize V(s) to more accurately reflect the future expected return. The specific update process is as follows: θ ← θ + α * δ * ∇θV(s); where θ is the neural network parameter, α is the first learning rate, which controls the update step size; ∇θV(s) is the gradient of the state value function with respect to the parameter θ; the neural network parameter θ is used to control the weights of the neural network, thereby affecting the estimation accuracy of the state value V(s).

[0037] Calculate the policy function π ϕ (a|s); The Actor selects the operation a through the policy function π ϕ (a|s). This policy represents the probability of performing the action a in the state space S. The policy gradient is used to update the parameters to maximize the probability of performing the action a in the current state.

[0038] Specifically, use the policy gradient method to update the parameters ϕ of the Actor. The update formula is: ϕ ← ϕ + β * δ * ∇ ϕ logπ ϕ V(s); where β is the second learning rate, and ∇ ϕ logπ ϕ V(s) is the gradient of the policy function with respect to the parameter ϕ, which is used to guide the policy update direction; δ is used to determine the update amplitude.

[0039] Estimate the action value and calculate the advantage function; In the Actor-Critic framework, the advantage function A(s, a) represents the advantage of the current action a relative to the average policy in that state. The definition of the advantage function is: A(s, a) = Q(s, a) - V(s); where Q(s, a) is the action-value function, representing the cumulative reward for executing action a in state s; in the present invention, the temporal difference error is approximated as the advantage function A(s, a) to guide the policy update of the Actor.

[0040] In summary, the steps of the Actor-Critic algorithm of the present invention are as follows: Input the current state space S into the neural network to obtain the state-value function V(s); Calculate the temporal difference error δ, and use the temporal difference error δ to update the neural network parameters θ of the state-value function; Update the policy parameter ϕ using the policy gradient; Repeat the above steps until convergence or the maximum number of iterations is reached; According to the current policy π ϕ (a|s) to select whether to perform migration.

[0041] Furthermore, in the Actor-Critic algorithm, the success value reward c in the reward function R is further optimized, that is, the success value reward c is set as a dynamic variable, which gradually changes with the production line working condition state information, so as to affect the system's migration tendency according to the production line working condition state, making the judgment of the necessity of migration more accurate.

[0042] Specifically, the improved success value reward c is jointly determined according to the production line virtual machine migration frequency and the predicted value of the migration target load. To avoid resource overload, when the predicted value of the load of the migration target is high, the success value reward c should be smaller to reduce the migration tendency and avoid adding more load to the production line that is already close to full load, thereby reducing the risk of resource overload. To improve the balance of resource allocation, the success value decreases as the predicted load value increases, and the system will be more inclined to select the target production line with a lower load for migration operations, which helps to achieve load balancing of resource allocation. If migration operations are carried out frequently, the consumption of system resources and operation costs increase. By reducing the success value reward c, the system can reduce migration operations at high frequencies, thereby avoiding resource waste, and high migration frequencies may bring additional system overheads (such as network traffic and CPU usage), while reducing the success value reward can reduce the occurrence of frequent migration operations and help the system maintain stability under high load conditions.

[0043] Therefore, the calculation expression of the dynamic success value reward can be: c = c 0 / (1 + w 1 P + w 2 M); where c 0Base success value; w 1 and w 2 are respectively the preset proportionality coefficients of the migration target load prediction value and the production line virtual machine migration frequency, and M is the production line virtual machine migration frequency.

[0044] Thus, the adjustment of the success value by comprehensively considering the migration frequency and the load prediction value enables the system to manage virtual machine migration operations more intelligently and flexibly in the production environment. Through this solution, the system can effectively control the migration tendency, optimize resource allocation, enhance system stability, and improve the rationality and intelligence of migration decisions, ultimately achieving better resource scheduling and management in complex and dynamic environments.

[0045] The Actor-Critic algorithm of the present invention guides the system to perform intelligent virtual machine migration by dynamically adjusting the reward score. The Actor is responsible for generating the migration strategy and continuously optimizing the decision according to the feedback given by the Critic (calculated through TD Error). The Critic uses a neural network to fit the state value function V(s) and guides the long-term migration strategy of the system by updating the estimation of the state value. This method can help the system achieve resource optimization in a dynamic environment and improve the migration success rate and resource allocation efficiency of the system.

[0046] Embodiment 2: In Embodiment 1 of the present invention, it is described in detail how to guide the system to perform intelligent virtual machine migration by dynamically adjusting the reward score. However, in the actual production process, when there is an addition or withdrawal of a production line, there are usually multiple production lines that can be selected for migration. In Embodiment 1, the production line selection is not carried out for this situation, resulting in the need to generate migration strategies for multiple production lines, increasing the overall migration selection time cost of the system. Embodiment 2 further optimizes this.

[0047] In step S2, if there are multiple production lines available for migration selection, the optimal production line is selected as the first migration target according to the production line working condition state information.

[0048] Specifically, the first migration target is determined according to the resource utilization rate and the production line performance volatility value of the production line.

[0049] Mark the resource utilization rate as U, which represents the current resource usage situation of the production line (such as the occupancy rate of CPU or memory). The higher this value, the more resources the production line occupies, and the greater the migration burden may be. Mark the production line performance volatility as σ, which represents the degree of fluctuation of the production line resource usage, usually measured by the standard deviation of the resource usage rate. The higher the volatility, the more unstable the resource demand of the production line, and it may not be suitable for high-frequency migration operations.

[0050] That is, the first migration target coefficient of each production line can be calculated through the resource utilization rate and the production line performance volatility value. Specifically, the calculation expression can be: E = r1 U + r 2 σ; where E is the first migration target coefficient, r 1 and r 2 are respectively the preset proportionality coefficients of the resource utilization rate and the production line performance volatility value.

[0051] Furthermore, when the first migration production line is determined by screening, it indicates that the migration states of other production lines are all inferior to this production line. At this time, only the migration strategy generation for the first migration production line needs to be performed.

[0052] Meanwhile, the state of the first migration production line determined each time may be different. Different states of the first migration production line will affect the decision of the migration strategy. Therefore, in this embodiment, the first migration target coefficient is also used as an important variable in the state space and incorporated into the state space S, that is, the state space S also includes the first migration target coefficient E.

[0053] It should be noted that the preset proportionality coefficients of the present invention are not limited here and can be specifically set according to the actual situation. For example, the weight value can be optimized based on the particle swarm algorithm to make it more in line with the corresponding fitness function. This process is a conventional existing technology and will not be elaborated here.

[0054] Embodiment 3, as Figure 2 shown, the present invention further includes an industrial automation control system based on a virtual engine for implementing the above method, which may include the following modules: A data acquisition module for acquiring the working condition state information of each production line to better formulate a migration strategy; An Actor-Critic algorithm module for formulating a migration strategy according to the working condition state information of each production line; Among them, the Actor-Critic algorithm module further includes a dynamic adjustment unit for dynamically adjusting the success value reward in the reward function R to facilitate better adjustment of the migration strategy tendency in line with the actual situation; A migration target screening module for screening the optimal migration production line and making a migration strategy decision.

[0055] The above formulas are all dimensionless and take their numerical calculations. The formulas are obtained by collecting a large amount of data for software simulation to obtain a formula closest to the actual situation. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0056] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product.

[0057] Those of ordinary skill in the art can realize that the modules and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application of the technical solution and the invention constraints. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0058] In addition, in each embodiment of this application, the functional modules can be integrated into a processing module, or each module can exist physically alone, or two or more modules can be integrated into one module.

[0059] As mentioned above, the above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0060] Finally: The above is only the preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An industrial automation control method based on a virtual engine, characterized in that: The steps include: When a production line is added or withdrawn, obtain the operating status information of each production line; Comprehensively evaluate the status of the migration target based on the Actor-Critic algorithm to predict whether virtual machine migration is needed; Execute the decision of the Actor-Critic algorithm to migrate virtual machines or maintain the existing configuration to ensure that resources are reasonably allocated and meet production needs; Continuously monitor the status of the production lines after migration, update the learning of the Actor-Critic algorithm, and update the working status information of each production line.

2. The industrial automation control method based on a virtual engine according to claim 1, characterized in that: The operating status information of each production line includes resource utilization, virtual machine migration frequency, production line performance volatility value, and migration target load prediction value.

3. The industrial automation control method based on a virtual engine according to claim 1, characterized in that: The Actor-Critic algorithm specifically includes the following steps: Set the state space S; Input the current state space S into the neural network to obtain the state value function V(s); Calculate the time difference error δ, and use the time difference error δ to update the neural network parameter θ of the state value function; Update the policy parameters ϕ using policy gradients; Repeat the above steps until convergence or the maximum number of iterations is reached; According to the current strategy π ϕ (a|s) Select whether to migrate.

4. The industrial automation control method based on a virtual engine according to claim 3 is characterized in that ； In the decision-making of the Actor-Critic algorithm, the success value reward c in the reward function R is further optimized. The success value reward c is set as a dynamic variable, which gradually changes with the production line working status information; the improved success value reward c is jointly determined based on the production line virtual machine migration frequency and the migration target load prediction value.

5. The industrial automation control method based on a virtual engine according to claim 2, characterized in that: If there are multiple production lines available for migration, select the best production line as the first migration target based on the production line operating status information: The first migration target is determined comprehensively based on the resource utilization rate of the production line and the volatility value of the production line performance. After the first migration production line is screened and determined, the migration strategy is generated only for the first migration production line.

6. The industrial automation control method based on a virtual engine according to claim 5, characterized in that: After the first migration production line is screened and determined, the first migration target coefficient is used as an important variable in the state space of the Actor-Critic algorithm and incorporated into the state space S.

7. An industrial automation control system based on a virtual engine, used to implement an industrial automation control method based on a virtual engine as claimed in any one of claims 1 to 6, characterized in that: It also includes a data acquisition module, an Actor-Critic algorithm module, and a migration target screening module, and signal connections between the modules; Data acquisition module, used to obtain the working status information of each production line in order to better formulate migration strategies; Actor-Critic algorithm module, used to formulate migration strategies based on the working status information of each production line; The Actor-Critic algorithm module also includes a dynamic adjustment unit for dynamically adjusting the success value reward in the reward function R, so as to better adjust the migration strategy tendency according to the actual situation; The migration target screening module is used to screen the optimal migration production line and make migration strategy decisions.

Images