A multi-objective dynamic scheduling optimization method from a machine perspective

Abstract

The present invention discloses a multi-objective dynamic scheduling optimization method from a machine perspective, including: based on a multi-layer neural network model, using multi-task learning to obtain a meta-state representation of the current order processing situation; based on the meta-state representation, training an intelligent agent to obtain a value function; using a multi-objective optimization method to solve the value function, and based on the solution result, completing the multi-objective dynamic scheduling optimization of production scheduling. The present invention effectively improves the universality of the system through a combination of multiple types of intelligent agents, quickly responds to new production environments, and assists managers to formulate corresponding scheduling plans based on the weight ratios of different objectives.

Description

Technical Field

[0001] The present invention belongs to the technical field of production scheduling, and in particular relates to a multi-objective dynamic scheduling optimization method from a machine perspective. Background Art

[0002] Scheduling is a key activity that shortens production cycles, improves resource utilization, and reduces production costs by rationally arranging and optimizing production resources. It plays an important role in production systems. It can be seen that dynamic scheduling methods based on artificial intelligence, data-driven, and in complex environments have become an important part of intelligent manufacturing.

[0003] Different from static production scheduling where the machine type, number of workpieces, and the time required for workpiece processing are fixed, various emergencies may occur in dynamic production scheduling, affecting key indicators such as manufacturing cycle and cost. For example, machine failure, order adjustment, manual operation, etc. will lead to uncertainty in the production process. When dealing with dynamic production scheduling problems, offline scheduling algorithms including swarm intelligence and evolutionary algorithms are unable to effectively evaluate the pros and cons of scheduling plans under uncertainty, while having difficulty adjusting scheduling plans in real time according to the production process caused by uncertainty.

[0004] In the warehouse of an e-commerce company, there are multiple robots and logistics carts, which are responsible for executing order processing tasks. If multi-objective dynamic scheduling is implemented in this company to improve production efficiency and enhance corporate competitiveness, it is necessary to face the unique attributes of each order, such as urgency, package size and weight. Secondly, the number of robots and logistics carts in the warehouse is limited, and the execution efficiency in performing different tasks varies, and there are energy limitations. Therefore, this realistic scenario requires that the multi-objective dynamic scheduling optimization method can maintain high universality in a constantly changing environment. However, the main problems of reinforcement learning include: high training cost and difficulty in knowledge transfer, which restricts the application of scheduling methods based on reinforcement learning in complex environments. Summary of the invention

[0005] The present invention mainly aims at the problem that the dynamic scheduling method based on reinforcement learning has not yet solved the difficulty of migrating scheduling strategies after changes in the production environment. A multi-objective dynamic scheduling optimization method from a machine perspective is proposed, which effectively improves the universality of the system by combining multiple types of intelligent agents; quickly responds to new production environments, and assists managers to formulate corresponding scheduling plans based on the weight ratios of different objectives.

[0006] To achieve the above object, the present invention provides a multi-objective dynamic scheduling optimization method from a machine perspective, comprising:

[0007] Based on a multi-layer neural network model, multi-task learning is used to obtain the meta-state representation of the current order processing status;

[0008] Based on the meta-state representation, training the production equipment to obtain a value function;

[0009] The value function is solved by adopting a multi-objective optimization method, and based on the solution result, the multi-objective dynamic scheduling optimization of production scheduling is completed.

[0010] Optionally, the multi-layer neural network model includes: an input layer, a hidden layer and an output layer;

[0011] The input layer is:

[0012] out0=(s t ,a t )

[0013] Among them, s t It indicates the state, a t represents the action, and out0 represents the feature vector corresponding to the state;

[0014] The hidden layer is:

[0015]

[0016] Among them, Relu represents the activation function. represents the output vector of the i-1th layer, θ i It represents the weight of the i-th layer;

[0017] The output layer is:

[0018]

[0019] Among them, out4 represents the final output result. It represents the output vector of the third layer.

[0020] Optionally, the agent is trained to obtain a value function including:

[0021] Construct mathematical models of production equipment environments;

[0022] Design environmental rewards;

[0023] Based on the environmental rewards, the mathematical model is solved according to a preset loss function to obtain the value function.

[0024] Optionally, designing the environmental reward includes:

[0025] In performing the task i At time t, rewards are given according to the state and action of production equipment order scheduling. If the order scheduling is not completed, the reward r i,t; If t = T, the order scheduling has been completed, then the reward Among them, T represents the time when the scheduling is completed. It represents the objective function An exponential function is used to adjust the scale of the reward to better reflect the importance or urgency of task objectives.

[0026] Optionally, the preset loss function is:

[0027]

[0028] in, Indicates in status Take action a t The action value function, r i,t+1 Indicates in status Take action a t And transfer to the state The immediate reward received afterwards, γ∈[0,1] represents the discount factor, Indicates that according to the status Choose the next action a t+1 strategy, Loss represents loss.

[0029] Optionally, using a multi-objective optimization method to solve the value function includes:

[0030] Multi-objective optimization is used to generate the Pareto frontier, and the multi-objective optimization problem is defined as a preset expression;

[0031] Substitute the value function into the preset expression and solve it.

[0032] Optionally, the preset expression is:

[0033]

[0034] in, represents the decision sequence, and st represents the decision sequence Constraints, It means that each decision a i must belong to a set of allowed actions, P represents the problem to be optimized, l represents the length of the decision sequence, and a T It represents the eigenvalue of the last decision element in the sequence.

[0035] Optionally, the preset expression after the value function is introduced is:

[0036]

[0037] Among them, P′ represents the problem that needs to be optimized.

[0038] The present invention has the following beneficial effects:

[0039] The present invention proposes a multi-objective dynamic scheduling optimization method from a machine perspective, which effectively improves the universality of the system by combining multiple types of intelligent agents; quickly responds to new production environments, and assists managers to formulate corresponding scheduling plans based on the weight ratios of different objectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 It is a technical route of a multi-objective dynamic scheduling optimization method from a machine perspective in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0044] When the goal changes due to the adjustment of management philosophy, the agent trained according to the original goal will lose its usability. At present, most of the dynamic scheduling research based on reinforcement learning is based on fixed goals, that is, the goal is to minimize the average delay or average cost; when the goal changes, for example, from efficiency priority to energy efficiency priority, it is difficult to adjust the existing strategy.

[0045] In the field of dynamic production scheduling, the high training cost and difficult knowledge transfer of reinforcement learning methods restrict the application of scheduling methods based on reinforcement learning in complex environments. In response to these problems, this embodiment proposes a multi-objective dynamic scheduling optimization method from a machine perspective, which effectively improves the universality of the system through the combination of multiple types of intelligent agents; quickly responds to new production environments, and assists managers to formulate corresponding scheduling plans based on the weight ratios of different objectives.

[0046] This embodiment constructs meta-state representation based on multi-task learning. It proposes a potential target pool for multi-dimensional evaluation indicators, combines multi-objective optimization technology, and studies the optimal scheduling strategy under different concepts. It proposes a dynamic multi-agent scheduling method based on multiple optimization objectives. Its technical roadmap is as follows: Figure 1 shown.

[0047] The evaluation indicators of the scheduling strategy under different dimensions are designed by artificial experts and used as the agent's goal. The meta-state representation under different goals is further extracted based on multi-task learning. The value function is then obtained through training. Finally, the multi-objective optimization method is used to solve the value function and output the Pareto frontier under the current state, that is, the optimal action set under different goals. The specific contents are as follows:

[0048] 1. It is proposed to use multi-task learning to extract state meta-representation: The present invention uses multi-task learning on the multi-layer neural network model to obtain the meta-representation of the current order processing situation, thereby accelerating the learning and migration process of scheduling strategies under multi-objective conditions. Each task is a single-objective optimization, and the weights between tasks are equal. At this time, the meta-representation of the current state can be obtained by minimizing the weighted loss.

[0049] 2. Extracting the meta-representation of the current order processing status through a multi-layer neural network model, wherein the multi-layer neural network model includes: an input layer, a hidden layer, and an output layer;

[0050] The input layer is:

[0051] out0=(s t ,a t )

[0052] Among them, s t Indicates the status, a t represents the action, out0 represents the feature vector corresponding to the state pair; in the field of production scheduling, s t It indicates the production status, equipment status, order requirements, etc. t It indicates the adjustment of production plan, equipment operation mode, etc., and out0 indicates the utilization rate of equipment, energy utilization rate, production time, etc.

[0053] The hidden layer is:

[0054]

[0055] Among them, Relu represents the activation function. represents the output vector of the i-1th layer, θ i It represents the weight of the i-th layer;

[0056] The output layer is:

[0057]

[0058] Among them, out4 represents the final output result. It represents the output vector of the third layer.

[0059] 3. The production equipment is trained to obtain the value function. On the production line, the state of the production equipment at time t is In Status Take action a t The reward obtained is r i,t , a t ∈A, then the corresponding environment is defined as Where S represents the set of conditions of the production equipment, and A represents the set of actions to be taken.

[0060] 3.1 Mathematical model of the construction environment: In the current environment, the task set is In the task i The state of the production equipment is Take the corresponding action as a t , generating a series of goals Any task i The corresponding target is in

[0061] 3.2 Designing environmental rewards: When performing task t i At time t, rewards are given according to the state and action of production equipment order scheduling. If the order scheduling is not completed, the reward r i,t ; If t = T, the order scheduling has been completed, then the reward

[0062] 3.3 Calculate the loss according to the formula:

[0063]

[0064] in, Indicates in status Take action (a t )’s action value function, r i,t+1 Indicates in status Take action (a t ) and transfer to the state The immediate reward received afterwards, γ∈[0,1] represents the discount factor, Indicates that according to the status Select the next action (a t+1 ) strategy.

[0065] 4. Use multi-objective optimization method value function to solve

[0066] Proposed to use multi-objective optimization to generate Pareto frontier: when the value function Current Status When is known, action a can be regarded as a variable, and the Pareto frontier can be searched through swarm intelligence optimization technology. Let the number of objectives be l ≥ 1. At this time, the multi-objective optimization problem about actions can be formally described as:

[0067]

[0068] in, represents the decision sequence, and st(subject to) represents the decision sequence Constraints, It means that each decision a i must belong to a set of allowed actions, decision sequences It can be obtained through online scheduling of reinforcement learning. From the specific implementation 3.2, it can be seen that the environmental reward is negatively correlated with the target value, so the value function It is also negatively correlated with the target value. If the value function It is known that the online multi-objective optimization problem based on value function can be further described as:

[0069]

[0070] Through online scheduling based on reinforcement learning, we can solve dynamic scheduling problems in complex environments represented by adjustments to production equipment structure and changes in enterprise demand, improve the migration of scheduling strategies across factories and enterprises, reduce the training overhead of scheduling strategies, and realize the reuse of scheduling strategies after adjustments to the enterprise's own needs.

[0071] A smart factory involves multiple production links and production equipment to produce different types of products. It was originally efficiency-oriented and adopted a single-objective production scheduling optimization method, that is, pursuing minimization of production costs and maximization of production efficiency. However, due to changes in social concepts and the increasing prominence of corporate social responsibility, the company decided to adjust its strategy to more comprehensively consider the environment, employee care and social responsibility; it decided to make adjustments in many aspects, including minimizing production cycles, maximizing equipment utilization, minimizing energy consumption, etc. The traditional intensive chemistry method has high training costs, difficult knowledge transfer, and difficult adjustment of existing strategies. Using this method, a multi-layer neural network model is used to extract meta-state representations under different objectives, and the value function is obtained through training to accelerate the strategy learning and transfer process under multi-objective conditions. Finally, the value function is solved by using a multi-objective optimization method to output the optimal action set under different objectives.

[0072] On the production line, each production equipment is regarded as an intelligent agent from the perspective of the machine, and meta-state representations are extracted from each intelligent agent in the workshop. These representations include information such as the working status of the equipment, production progress, and processing requirements of the current order. By observing the processing status of the current order, equipment status, and production progress, the intelligent agent learns multiple tasks, including production time, equipment utilization, energy consumption, etc., and adjusts the production plan and equipment operation mode of the corresponding intelligent agent to meet the optimal action set under different goals.

[0073] When the business philosophy changes, each goal corresponds to a task, and the weights of tasks are equal. By adding, deleting or changing task modules, the entire system can be avoided from being retrained. This allows the system to quickly adapt to new optimization goals and reduces the pressure of retraining caused by changes in business philosophy.

[0074] This embodiment uses multi-objective optimization as a method and a new method to achieve balanced scheduling under multiple orientations. This embodiment compares multi-objective optimization and multi-task learning to weigh the impact of different evaluation indicators on scheduling strategies and improve the difficulty of migrating scheduling strategies under multiple objectives.

[0075] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multi-objective dynamic scheduling optimization method from a machine perspective, characterized in that: include: Based on a multi-layer neural network model, multi-task learning is used to obtain the meta-state representation of the current order processing status; Based on the meta-state representation, training the production equipment to obtain a value function; Training production equipment to obtain value functions includes: Construct mathematical models of production equipment environments; Design environmental rewards; Based on the environmental reward, solving the mathematical model according to a preset loss function to obtain the value function; Designing the environmental rewards includes: In performing the task i At time t, rewards are given according to the state and action of production equipment order scheduling. If the order scheduling is not completed, the reward r i,t ; If t = T, the order scheduling has been completed, then the reward Among them, T represents the time when the scheduling is completed. It represents the objective function An exponential function is used to adjust the scale of rewards to better reflect the importance or urgency of task objectives; The preset loss function is: in, Indicates in status Take action a t The action value function, r i,t+1 Indicates in status Take action a t And transfer to the state The immediate reward received afterwards, γ∈[0,1] represents the discount factor, Indicates that according to the status Choose the next action a t+1 strategy, Loss means loss; A multi-objective optimization method is used to solve the value function, and based on the solution result, a multi-objective dynamic scheduling optimization of production scheduling is completed; The multi-layer neural network model includes: an input layer, a hidden layer and an output layer; The input layer is: out0=(s t ,a t ) Among them, s t It indicates the state, a t represents the action, and out0 represents the feature vector corresponding to the state; The hidden layer is: Among them, Relu represents the activation function. represents the output vector of the i-1th layer, θ i It represents the weight of the i-th layer; The output layer is: Among them, out4 represents the final output result. It represents the output vector of the third layer; Using a multi-objective optimization method, solving the value function includes: Multi-objective optimization is used to generate the Pareto frontier, and the multi-objective optimization problem is defined as a preset expression; Substitute the value function into the preset expression and solve it; The preset expression is: in, represents the decision sequence, and st represents the decision sequence Constraints, It means that each decision a i must belong to a set of allowed actions, P represents the problem to be optimized, l represents the length of the decision sequence, and a T It represents the characteristic value of the last decision element in the sequence; The preset expression after the value function is introduced is: Among them, P ′ Indicates a problem that needs to be optimized.

Images