Industrial automation control method and system based on virtual engine

Abstract

The present invention discloses an industrial automation control method and system based on a virtual engine, which relates to the field of virtual machine migration technology and is used to solve the problem of how to quickly evaluate and select appropriate migration targets among multiple virtual machines and production lines; comprehensively evaluate the status of the migration target according to the Ac algorithm to predict whether virtual machine migration is needed; execute the decision of the Ac algorithm to migrate the virtual machine or maintain the existing configuration to ensure reasonable allocation of resources and meet production needs; and be able to comprehensively consider the resource load and the stability of the production line in a production environment, so that the system is more accurate and reasonable when making migration decisions, effectively avoiding resource overload and unstable migration operations. Through intelligent control of migration tendencies, the system can significantly improve resource utilization efficiency and overall stability, enhance adaptability to complex dynamic environments, and achieve more efficient resource scheduling and optimization under changing production needs.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual machine migration, and more particularly, to an industrial automation control method and system based on a virtual engine. Background Art

[0002] Driven by Industry 4.0 and smart manufacturing, enterprises are increasingly demanding higher levels of production efficiency and resource management. While virtualization technology offers enterprises a more flexible approach to resource allocation, effective resource migration still presents numerous challenges in practice, particularly when adding or removing production lines.

[0003] In modern industrial production, the addition or removal of production lines is often accompanied by adjustments to resource allocation. Virtual machines, as critical resources in production environments, require dynamic adjustments to their migration strategies based on changing production line requirements. Traditional virtual machine migration decision-making methods are often based on static rules and cannot effectively address frequently changing production demands and resource pressures. When adding a new production line, existing virtual machines need to be migrated to the new environment, and when a production line is removed, it is necessary to ensure that the resources currently in use can be smoothly transferred to other production lines. This process requires precise timing to avoid impacting production. Therefore, how to quickly evaluate and select the appropriate migration target among multiple virtual machines and production lines is a complex decision-making problem.

[0004] In view of the above problems, the present invention proposes a solution. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide an industrial automation control method and system based on a virtual engine to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] An industrial automation control method based on a virtual engine comprises the following steps:

[0008] When a production line is added or withdrawn, obtain the working status information of each production line;

[0009] Comprehensively evaluate the status of the migration target using the Actor-Critic algorithm to predict whether virtual machine migration is necessary.

[0010] Execute the decisions of the Actor-Critic algorithm to migrate virtual machines or maintain the existing configuration to ensure reasonable resource allocation and meet production needs;

[0011] The system continuously monitors the status of the production lines after migration, updates the learning of the Actor-Critic algorithm, and updates the working status information of each production line.

[0012] In a preferred embodiment, the operating status information of each production line includes resource utilization, virtual machine migration frequency, production line performance volatility value, and migration target load prediction value.

[0013] In a preferred embodiment, the current state space S is input into the neural network to obtain the state value function V(s);

[0014] Calculate the time difference error δ, and use the time difference error δ to update the neural network parameter θ of the state value function;

[0015] Update the policy parameters ϕ using policy gradients;

[0016] Repeat the above steps until convergence or the maximum number of iterations is reached;

[0017] According to the current strategy π ϕ (a|s) Select whether to migrate.

[0018] In a preferred embodiment, when executing the decision-making of the Actor-Critic algorithm, the success value reward c in the reward function R is further optimized, and the success value reward c is set as a dynamic variable, which gradually changes with the production line working status information; the improved success value reward c is jointly determined based on the production line virtual machine migration frequency and the migration target load prediction value.

[0019] In a preferred embodiment, if there are multiple production lines available for migration, the optimal production line is selected as the first migration target based on the production line operating status information:

[0020] The first migration target is determined based on the resource utilization rate and production line performance volatility of the production line. After the first migration production line is screened and determined, the migration strategy is generated only for the first migration production line.

[0021] In a preferred embodiment, after the first migration production line is screened and determined, the first migration target coefficient is incorporated into the state space S as an important variable in the state space of the Actor-Critic algorithm.

[0022] An industrial automation control system based on a virtual engine, further comprising a data acquisition module, an Actor-Critic algorithm module and a migration target screening module, wherein the modules are signal-connected;

[0023] The data acquisition module is used to obtain the working status information of each production line to better formulate the migration strategy;

[0024] Actor-Critic algorithm module, used to formulate migration strategies based on the working status information of each production line;

[0025] The Actor-Critic algorithm module also includes a dynamic adjustment unit for dynamically adjusting the success value reward in the reward function R, so as to better adjust the migration strategy tendency according to the actual situation;

[0026] The migration target screening module is used to select the optimal migration production line and make migration strategy decisions.

[0027] The technical effects and advantages of the industrial automation control method and system based on the virtual engine of the present invention are as follows:

[0028] The present invention can flexibly choose whether to migrate virtual machines based on the addition or exit of production lines, thereby improving the intelligence level of resource scheduling. It also guides the system to perform intelligent virtual machine migration by dynamically adjusting reward points. The Actor is responsible for generating migration strategies and continuously optimizing decisions based on the feedback provided by the Critic (calculated through TD Error). The Critic uses a neural network to fit the state value function V(s) and guides the system's long-term migration strategy by updating the estimate of the state value. This method can help the system achieve resource optimization in a dynamic environment, improving the system's migration success rate and resource allocation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of an industrial automation control method based on a virtual engine according to the present invention;

[0030] Figure 2 This is a structural diagram of an industrial automation control system based on a virtual engine in the present invention. DETAILED DESCRIPTION

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0032] This invention presents a virtualization engine-based industrial automation control method and system. Addressing the virtual machine migration problem when adding or removing production lines, a dynamic virtual machine migration strategy based on the actor-critic algorithm is proposed. This strategy first calculates each production line's suitability as a virtual machine migration target using a reward score coefficient (RSC). It then uses dynamic success value rewards to adjust migration preferences, enabling intelligent migration under varying resource load conditions.

[0033] After determining the primary migration target, the Actor-Critic algorithm makes a decision, comprehensively analyzing the production line's demand status and reward coefficients to determine whether to migrate the virtual machines. The algorithm uses the policy network (Actor) to determine the migration action, and the value network (Critic) to evaluate the migration's effectiveness. The reward function dynamically adjusts the reward coefficient to ensure that migration behavior better meets actual production needs, avoiding excessive pressure on production lines with high loads or frequent migrations.

[0034] Example 1, as Figure 1 As shown, the present invention provides an industrial automation control method based on a virtual engine, comprising the following steps:

[0035] Step S1, when a production line is added or exited, obtain the working status information of each production line;

[0036] Step S2: Comprehensively evaluate the state of the migration target based on the Actor-Critic algorithm to predict whether virtual machine migration is required.

[0037] Step S3: Execute the decision of the Actor-Critic algorithm to migrate virtual machines or maintain the existing configuration to ensure that resources are reasonably allocated and meet production needs;

[0038] In step S4, the system continuously monitors the status of the production lines after migration, updates the working status information of each production line through learning and updating of the Actor-Critic algorithm.

[0039] Specifically, in step S1, the working status information of each production line includes resource utilization, virtual machine migration frequency, production line performance volatility value, and migration target load prediction value.

[0040] Resource utilization reflects the current resource load of a production line, including CPU, memory, and storage. It can be a weighted average of resources like CPU and memory. Production lines with high resource utilization are typically heavily loaded and resource-constrained, making them unsuitable migration targets. Conversely, production lines with low resource utilization have more idle resources and are therefore suitable candidates for VM migration.

[0041] If a production line's resource utilization is high, it indicates that its compute, storage, and other resources are nearing saturation. Further migrating virtual machines to this production line may cause resource overload and impact overall performance. Therefore, when selecting a primary migration target, the system avoids production lines with high resource utilization. Production lines with low resource utilization have more available resources and can handle more virtual machine migrations. Therefore, production lines with low resource utilization are more likely to become migration targets.

[0042] The VM migration frequency indicates the number of VMs that migrate in or out of a production line per unit time. A production line with a high migration frequency involves multiple VM migrations within a short period of time. Frequent migrations consume network bandwidth, impact system performance, and incur additional resource management overhead. Therefore, production lines with a high migration frequency are generally not suitable as new migration targets.

[0043] Production lines with a high migration frequency already participate in multiple VM migrations per unit time. Further migration operations could overload the system and impact production line stability. The system assigns these production lines a higher bonus coefficient, lowering their priority as migration targets. Production lines with a low migration frequency participate less frequently, have more stable resource scheduling, and are therefore suitable for receiving new VMs. Therefore, production lines with a low migration frequency are more likely to be selected as the primary migration target.

[0044] The line performance volatility value indicates the degree of change in resource utilization over a period of time, specifically the stability or fluctuation of resource usage load. A high line performance volatility value indicates frequent and significant fluctuations in resource usage, while a low volatility value indicates relatively stable resource usage. This metric is crucial for assessing the adaptability and stability of the migration target. During virtual machine migration, the system must ensure that the target production line can stably handle the increased resource load.

[0045] Production line performance volatility is typically measured using statistical methods, such as variance or standard deviation. These metrics can quantify fluctuations in resource usage over time. Specifically, in this embodiment, the production line performance volatility value σi can be calculated as the standard deviation of resource utilization (e.g., CPU, memory, etc.) for production line i.

[0046] Before executing a virtual machine migration, the system estimates the target production line's load after the migration is complete. This prediction helps the system determine the impact of the migration on the target production line's resources, deciding whether to proceed and prioritizing the migration. When selecting a migration target, the system not only considers current resource usage but also assesses whether the target production line can withstand the new load after the migration is complete.

[0047] To estimate the resource utilization of the target production line after the migration is completed, the migration target load prediction value P can be calculated as follows:

[0048] P=Ucurrent,target+Estimated Migration Load;

[0049] Where Ucurrent,target is the current resource utilization of the target production line, which can be a weighted average of resources such as CPU and memory. Estimated Migration Load is the load increase caused by the migration operation, which is usually estimated based on the resource requirements of the migrated VM. For example, if the migrated VM requires 20% CPU and 10% memory, these requirements will be added to the current resource utilization of the target production line.

[0050] Furthermore, the Actor-Critic algorithm is a policy-based reinforcement learning algorithm that uses a policy function (Actor) to select actions and a value function (Critic) to evaluate the quality of the state. This algorithm can be used to predict whether to migrate virtual machines based on the production line's operating status. Specifically, it involves the following steps:

[0051] Define the state space S;

[0052] The state space S contains all important variables that describe the current state of the migration target (production line), such as:

[0053] Average migration time cost Ti for a production line: This is the average migration time for production line i over a certain period of time, reflecting the efficiency of production line migration. Production lines with higher migration time costs may slow down the overall migration speed of the system.

[0054] Production line task type Ki: This refers to the type of production line task (e.g., batch processing, real-time processing, etc.) to assess whether migration is appropriate. Some task types require high resources or are not suitable for frequent migration. This can serve as a reference for migration decisions.

[0055] Define the action space A;

[0056] The action space consists of two options:

[0057] a=1: execute migration operation; a=0: do not execute migration.

[0058] Set the reward function R;

[0059] The reward function R is used to feedback the effect of the migration operation, taking into account the number of migrations and the stability of the production line. Here the reward function is designed as:

[0060] ;

[0061] Where c is the success reward for successful migration; d is the penalty for migration failure or insufficient resources.

[0062] Calculate the state value function V(s);

[0063] The state value function V(s) represents the future expected cumulative reward of the system in the state space S.

[0064] The calculation expression can be: V(s)=[R t +γR t+1 +γ 2 R t+1 +...|s t =s]; where R t is the immediate reward function, s t is the current state space, and γ is the discount factor used to decay future rewards.

[0065] In order to solve the state value function V(s), this embodiment uses a neural network to approximate it. The network input is set to be the state s, and the output is the estimated V(s).

[0066] In the neural network structure, the input layer is the state space S, which contains all important variables describing the current state of the migration target (production line); the hidden layer is used to extract state features; and the output layer is the state value function V(s), which represents the total future return under the current state.

[0067] Calculate the time difference error and use the time difference error to update the state value function;

[0068] The update of the state value function is based on the time difference error (TD Error), which is used to measure the difference between the current estimated state value V(s) and the sum of future rewards. The calculation expression of the time difference error δ is: δ=R+γV(s')-V(s); R is the instantaneous reward function of the current time step, that is, R t ; V(s') is the estimated value of the next state s' after the operation is performed.

[0069] The neural network parameters θ are updated through the time difference error δ, and V(s) is optimized to more accurately reflect the expected future returns. The specific update process is as follows: ; In the formula, θ is the neural network parameter, α is the first learning rate, which controls the update step size; ∇θV(s) is the gradient of the state value function with respect to the parameter θ; the neural network parameter θ is used to control the weight of the neural network, thereby affecting the estimation accuracy of the state value V(s).

[0070] Calculate the policy function π ϕ (a|s);

[0071] Actor passes the strategy function π ϕ (a|s) selects action a. The policy represents the probability of executing action a in the state space S. The policy gradient is used to update the parameters to maximize the probability of executing action a in the current state.

[0072] Specifically, the policy gradient method is used to update the Actor's parameter ϕ. The update formula is: ; Where β is the second learning rate, ∇ ϕ logπ ϕ V(s) is the gradient of the policy function with respect to the parameter ϕ, which is used to guide the direction of policy update; δ is used to determine the magnitude of the update.

[0073] Estimate action values ​​and calculate advantage functions;

[0074] In the Actor-Critic framework, the advantage function A(s, a) represents the advantage of the current action a relative to the average policy in that state. The advantage function is defined as: A(s, a) = Q(s, a) - V(s); where Q(s, a) is the action-value function, representing the cumulative reward for executing action a in state s. In this paper, the time difference error is approximated as the advantage function A(s, a) to guide the actor's policy update.

[0075] In summary, the steps of the Actor-Critic algorithm of the present invention are:

[0076] Input the current state space S into the neural network to obtain the state value function V(s);

[0077] Calculate the time difference error δ, and use the time difference error δ to update the neural network parameter θ of the state value function;

[0078] Update the policy parameters ϕ using policy gradients;

[0079] Repeat the above steps until convergence or the maximum number of iterations is reached;

[0080] According to the current strategy π ϕ (a|s) Select whether to migrate.

[0081] Furthermore, in the Actor-Critic algorithm, the success value reward c in the reward function R is further optimized. That is, the success value reward c is set as a dynamic variable. As the production line operating status information gradually changes, the migration tendency of the system is affected by the production line operating status, which makes the judgment on the necessity of migration more accurate.

[0082] Specifically, the improved success reward c is determined based on the migration frequency of the production line's virtual machines and the predicted load of the migration target. To avoid resource overload, the success reward c should be smaller when the predicted load of the migration target is higher. This reduces the tendency to migrate and avoids adding more load to a production line that is already nearing full capacity, thereby reducing the risk of resource overload. To improve resource balance, the success reward decreases as the predicted load increases. The system will be more inclined to select the target production line with lower load for migration operations, helping to achieve load balancing in resource allocation. If migration operations are performed frequently, system resource consumption and operating costs increase. By reducing the success reward c, the system can reduce high-frequency migration operations, thereby avoiding resource waste. High migration frequency may incur additional system overhead (such as network traffic and CPU usage). However, reducing the success reward can reduce frequent migration operations, helping the system maintain stability under high load.

[0083] Therefore, the calculation expression for the dynamic success value reward can be: c=c0 / (1+w1P+w2M); where c0 is the basic success value; w1 and w2 are the preset proportional coefficients of the migration target load prediction value and the production line virtual machine migration frequency, respectively, and M is the production line virtual machine migration frequency.

[0084] By combining migration frequency and load prediction to adjust success rates, the system can more intelligently and flexibly manage virtual machine migration operations in production environments. This approach effectively controls migration trends, optimizes resource allocation, enhances system stability, and improves the rationality and intelligence of migration decisions, ultimately achieving better resource scheduling and management in complex and dynamic environments.

[0085] The actor-critic algorithm of this invention guides the system for intelligent virtual machine migration by dynamically adjusting reward points. The actor is responsible for generating migration strategies and continuously optimizing them based on the critic's feedback (calculated through TD Error). The critic uses a neural network to fit the state-value function V(s) and guides the system's long-term migration strategy by updating its state-value estimate. This method helps the system achieve resource optimization in a dynamic environment, improving the system's migration success rate and resource allocation efficiency.

[0086] In Example 2, the present invention detailed in Example 1 how to guide the system to intelligently migrate virtual machines by dynamically adjusting reward points. However, in actual production, when a production line is added or removed, multiple production lines are often available for migration. Example 1 does not include a production line selection for this situation, resulting in the need to generate migration strategies for multiple production lines, increasing the overall migration selection time and cost of the system. This Example 2 further optimizes this issue.

[0087] In step S2, if there are multiple production lines available for migration, the optimal production line is selected as the first migration target based on the production line operating status information.

[0088] Specifically, the first migration target is determined according to the resource utilization rate of the production line and the volatility value of the production line performance.

[0089] Resource utilization, labeled U, indicates the current resource usage of the production line (such as CPU or memory usage). A higher value indicates greater resource usage and potentially a greater migration burden. Line performance volatility, labeled σ, indicates the degree of fluctuation in line resource usage, typically measured as the standard deviation of resource utilization. Higher volatility indicates more volatile resource requirements, making it less suitable for high-frequency migrations.

[0090] The first migration target coefficient of each production line can be calculated by using the resource utilization rate and the production line performance volatility value. Specifically, the calculation expression can be: E=r1U+r2σ; where E is the first migration target coefficient, r1 and r2 are the preset proportional coefficients of the resource utilization rate and the production line performance volatility value, respectively.

[0091] Furthermore, after the first migration production line is screened and determined, it indicates that the migration status of other production lines is not as good as this production line. At this time, it is only necessary to generate a migration strategy for the first migration production line.

[0092] At the same time, the status of the first migration production line may be different each time it is determined. Different first migration production line statuses will affect the decision of the migration strategy. Therefore, this embodiment also incorporates the first migration target coefficient as an important variable in the state space into the state space S, that is, the state space S also includes the first migration target coefficient E.

[0093] It should be noted that the preset proportional coefficients of the present invention are not limited here and can be specifically set according to actual conditions. For example, the weight value can be optimized based on the particle swarm algorithm to make it more consistent with the corresponding fitness function. This process is an existing conventional technology and will not be elaborated here.

[0094] Example 3, as Figure 2 As shown, the present invention also includes an industrial automation control system based on a virtual engine, which is used to implement the above method and may include the following modules:

[0095] The data acquisition module is used to obtain the working status information of each production line to better formulate the migration strategy;

[0096] Actor-Critic algorithm module, used to formulate migration strategies based on the working status information of each production line;

[0097] The Actor-Critic algorithm module also includes a dynamic adjustment unit, which is used to dynamically adjust the success value reward in the reward function R, so as to better adjust the migration strategy tendency according to the actual situation.

[0098] The migration target screening module is used to select the optimal migration production line and make migration strategy decisions.

[0099] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0100] The above embodiments may be implemented in whole or in part through software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0101] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application of the technical solution and the invention constraints. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0102] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0103] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0104] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An industrial automation control method based on a virtual engine, characterized in that: The steps include: When a production line is added or withdrawn, obtain the working status information of each production line; Comprehensively evaluate the status of the migration target using the Actor-Critic algorithm to predict whether virtual machine migration is necessary. Execute the decisions of the Actor-Critic algorithm to migrate virtual machines or maintain the existing configuration to ensure reasonable resource allocation and meet production needs; Continuously monitor the status of the production lines after migration, update the working status information of each production line through learning and updating of the Actor-Critic algorithm; The Actor-Critic algorithm specifically includes the following steps: Input the current state space S into the neural network to obtain the state value function V(s); Calculate the time difference error δ, and use the time difference error δ to update the neural network parameter θ of the state value function; Update the policy parameters ϕ using policy gradients; Repeat the above steps until convergence or the maximum number of iterations is reached; According to the current strategy π ϕ (a|s) Select whether to migrate; When executing the decision-making of the Actor-Critic algorithm, the success value reward c in the reward function R is further optimized. The success value reward c is set as a dynamic variable, which gradually changes with the production line operating status information. The improved dynamic variable success value c is jointly determined based on the production line virtual machine migration frequency and the migration target load prediction value.

2. The industrial automation control method based on a virtual engine according to claim 1, characterized in that: The working status information of each production line includes resource utilization, virtual machine migration frequency, production line performance volatility value, and migration target load prediction value.

3. The industrial automation control method based on a virtual engine according to claim 2, characterized in that: If there are multiple production lines available for migration, select the optimal production line as the first migration target based on the production line operating status information: The first migration target is determined based on the resource utilization rate and production line performance volatility of the production line. After the first migration production line is screened and determined, the migration strategy is generated only for the first migration production line.

4. The industrial automation control method based on a virtual engine according to claim 3, characterized in that: After the first migration production line is screened and determined, the first migration target coefficient is used as an important variable in the state space of the Actor-Critic algorithm and incorporated into the state space S.

5. An industrial automation control system based on a virtual engine, used to implement the industrial automation control method based on a virtual engine according to any one of claims 1 to 4, characterized in that: It also includes a data acquisition module, an Actor-Critic algorithm module, and a migration target screening module, with signal connections between the modules; The data acquisition module is used to obtain the working status information of each production line to better formulate the migration strategy; Actor-Critic algorithm module, used to formulate migration strategies based on the working status information of each production line; The Actor-Critic algorithm module also includes a dynamic adjustment unit for dynamically adjusting the success value reward in the reward score R, so as to better adjust the migration strategy tendency according to the actual situation; The migration target screening module is used to select the optimal migration production line and make migration strategy decisions.

Images