Collaborative optimization method, system and device for multiple service processes and medium

Abstract

The invention discloses a collaborative optimization method, system and device for multiple business processes and a medium, and the method comprises the steps: obtaining a first state of a plurality of business processes, assigning the first state to a current state, processing the current state through a deep reinforcement learning algorithm, obtaining a first action, and enabling one first action to represent one business process, respectively optimizing each first action by utilizing a moth fire suppression algorithm to obtain a second action of each first action and the fitness of the second action; processing the second action by utilizing a deep reinforcement learning algorithm to obtain a next state and a first reward, updating parameters of the deep reinforcement learning algorithm according to the current state, the first reward, the fitness and the next state, taking the next state as the current state, and returning to execute the first action of processing the current state by utilizing the deep reinforcement learning algorithm, and outputting an optimization scheme in which the next state meeting the preset condition is the multi-business process until the preset condition is met. The method can realize global optimization of a plurality of business processes, and can be applied to the technical field of multi-process scheduling.

Description

Technical Field

[0001] The present invention relates to the field of multi-process scheduling technology, and in particular to a collaborative optimization method, system, device and medium for multiple business processes. Background Art

[0002] When urgent production orders appear, it is necessary to adjust not only the production scheduling plan, but also the procurement plan, transportation plan, warehousing storage plan, etc. Existing optimization methods usually only perform local optimization on a single business process, while the business processes involved in the product life cycle are closely related. The optimization of a single business process cannot meet the needs of efficient integration and collaborative optimization control of cross-system multi-business processes such as supply chain management, production scheduling, route planning, and warehousing planning. Considering the characteristics of cross-system business processes being complex and diverse, poor interoperability, and complex collaborative interaction processes, only the local optimization of a single business process can be achieved, and the global optimization cannot be achieved. Summary of the invention

[0003] In order to solve one of the above problems, the purpose of the present invention is to provide a collaborative optimization method, system, device and medium for multiple business processes, which can realize collaborative optimization between multiple business processes and achieve global optimization.

[0004] In one aspect, the present invention provides a method for collaborative optimization of multiple business processes, comprising the following steps:

[0005] Obtaining first states of several business processes, assigning the first states to current states, and processing the current states using an agent of a deep reinforcement learning algorithm to obtain a first action set; wherein one of the first actions represents a business process; and the agent of the deep reinforcement learning algorithm is obtained by improving an attention mechanism;

[0006] Using the moth-to-flame algorithm to optimize each of the first actions respectively, to obtain a second action corresponding to each of the first actions and the fitness of the second action;

[0007] Applying the second action to the environment of the deep reinforcement learning algorithm to obtain a next state and a first reward; updating parameters of the deep reinforcement learning algorithm according to the current state, the first reward, the fitness of the second action, and the next state;

[0008] Taking the next state as the current state, returning to execute the intelligent agent using the deep reinforcement learning algorithm to process the current state, obtaining the first action, until the preset conditions are met, and outputting the current state that meets the preset conditions as an optimization plan for multiple business processes.

[0009] Optionally, the using the moth-to-flame algorithm to optimize each of the first actions respectively to obtain the second action corresponding to each of the first actions and the fitness of the second action specifically includes:

[0010] Processing the first action by using chaotic mapping and reverse learning to generate an initial moth population, and obtaining a first generation of flames according to the initial moth population;

[0011] Iteration is performed according to the Brownian motion mechanism, the elite strategy, the initial moth population and the first generation of flames to obtain the second action and the fitness of the second action.

[0012] Optionally, the using of chaotic mapping and reverse learning to process the first action to generate an initial moth population, and obtaining a first generation of flames according to the initial moth population specifically includes:

[0013] Using the chaotic mapping to perform a mapping operation on the first action to obtain a moth population after mapping;

[0014] Using the reverse learning, performing random operations on the mapped moth population to obtain the initial moth population;

[0015] The fitness of the initial moth population is calculated, and the initial moth population is screened according to the fitness of the initial moth population to obtain the first generation of flames.

[0016] Optionally, the iterating according to the Brownian motion mechanism, the elite strategy, the initial moth population and the first generation flame to obtain the second action and the fitness of the second action specifically includes:

[0017] Assign the initial moth population to the current moth population, and the first generation flame to the current flame;

[0018] Using the Brownian motion mechanism to update the current moth population to obtain the next generation moth population;

[0019] Calculating the fitness of the next generation moth population, sorting the next generation moth population according to the fitness of the next generation moth population to obtain the sorted next generation moth population, and screening the sorted next generation moth population using the elite strategy to obtain the next generation flame;

[0020] Determine whether the next generation flame meets the cutoff condition. If the next generation flame does not meet the cutoff condition, use the next generation moth population as the contemporary moth population and the next generation flame as the contemporary flame, return to execute the Brownian motion mechanism to update the contemporary moth population to obtain the next generation moth population, until the obtained next generation flame meets the cutoff condition, output the next generation flame that meets the cutoff condition as the second action, and the fitness of the next generation moth population is used as the fitness of the second action.

[0021] Optionally, the updating of the contemporary moth population by using the Brownian motion mechanism to obtain the next generation moth population specifically includes:

[0022] Calculating the scale factor of the Brownian motion mechanism using a preset activation function;

[0023] Substituting the scale factor, random number and the current moth population into the Brownian motion mechanism, the next generation moth population is obtained.

[0024] Optionally, the step of sorting the next generation moth population according to the fitness of the next generation moth population to obtain the sorted next generation moth population specifically includes:

[0025] Calculating the crowding distance of each moth according to the fitness of the next generation moth population; wherein the moth population includes a plurality of moths, and each moth represents a solution to a business process;

[0026] The next generation moth population is sorted according to the crowding distance to obtain a sorted next generation moth population.

[0027] Optionally, updating the parameters of the deep reinforcement learning algorithm according to the current state, the first reward, the fitness of the second action, and the next state specifically includes:

[0028] Performing a weighted summation of the first reward and the fitness of the second action according to a preset ratio to obtain a second reward;

[0029] Substituting the second reward, the current state, and the next state into the deep reinforcement learning algorithm to obtain a first loss function set;

[0030] The current state is horizontally flipped to obtain a first state, the first state is input into a deep reinforcement learning algorithm to obtain a second state set, a second loss function set is obtained according to the second state set and the first loss function set, and the parameters of the deep reinforcement learning algorithm are updated according to the second loss function set and a back propagation algorithm.

[0031] In another aspect, the present invention provides a collaborative optimization device for multiple business processes, comprising:

[0032] at least one processor;

[0033] at least one memory for storing at least one program;

[0034] When the at least one program is executed by the at least one processor, the at least one processor implements the above-described method.

[0035] In another aspect, the present invention provides a computer-readable storage medium storing a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to perform the above-mentioned method.

[0036] On the other hand, the present invention provides a collaborative optimization system for multiple business processes, including a data sensing and acquisition device and a computer device connected to the data sensing and acquisition device, wherein the data sensing and acquisition device is installed in the business process; wherein:

[0037] The data sensing and acquisition device is used to collect the first status of a plurality of business processes and upload the first status to the computer device;

[0038] The computer device comprises:

[0039] at least one processor;

[0040] at least one memory for storing at least one program;

[0041] When the at least one program is executed by the at least one processor, the at least one processor implements the above-described method.

[0042] The implementation of the present invention includes the following beneficial effects: the present invention obtains the first states of several business processes, assigns the first state to the current state, uses the intelligent agent of the deep reinforcement learning algorithm to process the current state, and obtains a first action set; wherein, a first action represents a business process, and the intelligent agent of the deep reinforcement learning algorithm is obtained by improving the attention mechanism, and uses the moth-to-fire algorithm to optimize each first action respectively, obtains the second action corresponding to each first action and the fitness of the second action, and applies the second action to the environment of the deep reinforcement learning algorithm to obtain the next state and the first reward; updates the parameters of the deep reinforcement learning algorithm according to the current state, the first reward, the fitness of the second action and the next state, and uses the next state as the current state. state, returns to execute the first action of calculating the current state using the deep reinforcement learning algorithm, until the preset conditions are met, and outputs the next state that meets the preset conditions as an optimization solution for multiple business processes; the present invention integrates multiple business processes involved in the production process as the intelligent body of the deep reinforcement learning algorithm, obtains the action of the deep reinforcement learning algorithm, and then uses the moth-to-fire algorithm to optimize each business process (that is, each action of the deep reinforcement learning algorithm), and then involves the fitness of the moth-to-fire algorithm in the reward of the deep reinforcement learning algorithm, and substitutes the single-process business optimized by the moth-to-fire algorithm into the environment of the deep reinforcement learning algorithm to obtain the optimal solution for multiple business processes, which can realize collaborative optimization between multiple different business processes and achieve global optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of the steps of a collaborative optimization method for multiple business processes provided by the present invention;

[0044] Figure 2 It is an iterative flow chart of a collaborative optimization method for multiple business processes provided by the present invention;

[0045] Figure 3 It is a simplified deep reinforcement learning model provided by the present invention;

[0046] Figure 4 is a flowchart of steps for obtaining a second action and the fitness of the second action provided by the present invention;

[0047] Figure 5 It is a flow chart of steps for obtaining an initial moth population and a first generation of flames provided by the present invention;

[0048] Figure 6 It is an iterative flow chart of a moth-to-flame algorithm provided by the present invention;

[0049] Figure 7 It is a specific step flow chart of a second action and the fitness of the second action provided by the present invention;

[0050] Figure 8 It is a flowchart of the steps of updating the parameters of a deep reinforcement learning algorithm provided by the present invention;

[0051] Fig. 9 It is a structural diagram of a collaborative optimization device for multiple business processes provided by the present invention;

[0052] Fig.10 It is a structural diagram of a collaborative optimization system for multiple business processes provided by the present invention. DETAILED DESCRIPTION

[0053] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0054] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0055] When the following description relates to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. In the description of the present application, it is necessary to understand that the terms "first / S100", "second / S200", "third / S300", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0056] The problem that reinforcement learning algorithms study is how agents learn certain strategies in an environment to maximize long-term rewards. The mathematical basis and modeling tool of reinforcement learning is the Markov decision process. Deep reinforcement learning combines the advantages of deep neural networks in modeling complex problems and solves sequential decision problems based on the principle of maximizing rewards.

[0057] like Figure 1 and Figure 2 As shown, Figure 1It is a step flow chart of the collaborative optimization method of multiple business processes. Figure 2 It is an iterative flowchart of a collaborative optimization method for multiple business processes. Figure 2 Where Env represents the environment of the deep reinforcement learning algorithm, and Actor represents the agent of the deep reinforcement learning algorithm. The present invention provides a collaborative optimization method for multiple business processes, including the following steps:

[0058] S100. Obtain first states of several business processes, assign the first states to a current state, process the current state using a deep reinforcement learning algorithm, and obtain a first action.

[0059] Wherein, a first action represents a business process.

[0060] The business process may include but is not limited to supply chain management, procurement planning, production scheduling, transportation route planning, and warehousing planning involved in the production process.

[0061] Supply chain management and procurement planning include but are not limited to supplier selection, procurement quantity, procurement price, procurement goods, procurement quality, and the corresponding procurement volume for each supplier.

[0062] Production scheduling may include but is not limited to the output volume, processing hours, production line load and production cost of each production line on the mixed-flow assembly line.

[0063] Transport route planning may include but is not limited to the number of transport vehicles, the load capacity of each transport vehicle, the route of each transport vehicle, the transport time and the transport cost, etc.

[0064] Warehousing planning may include but is not limited to storage locations, storage capacity, storage locations for incoming goods, and storage costs, etc.

[0065] Specifically, the parameters of the deep reinforcement learning algorithm are initialized, and the current data of each business process is obtained as the first state. The first state includes several first actions. A first action represents a business process. For example, in supply chain management and procurement planning, a first action can represent the current supplier selection, procurement quantity, procurement price, procurement goods, procurement quality, and procurement volume corresponding to each supplier of the business process.

[0066] The intelligent agent of the deep reinforcement learning algorithm is obtained by improving the attention mechanism. The improvement method is to add the attention mechanism to the policy network of the existing deep reinforcement learning algorithm. The existing deep reinforcement learning algorithm directly inputs the current state s into the policy network to obtain the first action a1. In the present invention, the feature representation of the current state s is first extracted by using the attention mechanism, and the feature representation is input into the policy network. The feature representation extraction process can be shown in formulas (1)-(4):

[0067] Q = s * ω_Q + b_Q (1)

[0068] K = s * ω_K + b_K (2)

[0069] V = s * ω_V + b_V (3)

[0070]

[0071] Among them, ω_Q, ω_K, ω_V are weight parameters to be learned, b_Q, b_K, b_V are bias parameters to be learned. Q, K and V are the three intermediate feature representations of s, s1′ is the feature representation of the current state s, K T is the transposed matrix of K, d k is the dimension of K.

[0072] The acquired first state is assigned to the current state s, and after obtaining the feature representation s1′ of the current state s, the policy network processes the feature representation s1′. The specific process of obtaining the first action a1 is the same as the processing process of the policy network of the deep reinforcement learning algorithm in the prior art. The difference is that, in the present invention, multiple business processes are input into the intelligent agent of the deep reinforcement learning algorithm.

[0073] When using the deep reinforcement learning algorithm to process the current state and obtain the first action, multi-objective functions can be set according to quality, cost, efficiency, energy consumption, etc., and constraints can be set based on the working environment, time, etc. to obtain the optimal solution.

[0074] The deep reinforcement learning algorithm may be, but is not limited to, a proximal policy optimization (PPO) algorithm or an actor-critic algorithm.

[0075] S200: Optimize each of the first actions respectively by using a moth-to-flame algorithm to obtain a second action corresponding to each of the first actions and the fitness of the second action.

[0076] Wherein, in this step, the moth-to-flame algorithm may also be a particle swarm algorithm or a genetic algorithm.

[0077] In the moth-to-flame algorithm, each moth in each generation of the moth population represents a business process, and the flame represents the optimal solution in a business process.

[0078] When using the moth-to-flame algorithm to solve the optimal solution of a single business process to obtain the second action and the fitness of the second action, multiple objective functions can be set for a single business process with the best quality, lowest cost, highest efficiency, and lowest energy consumption.

[0079] Specifically, the parameters of the multi-objective moth-to-fire optimization algorithm are initialized, and the first action is processed by using chaotic mapping combined with reverse learning to obtain the initial moth population of the moth-to-fire algorithm; the fitness of the initial moth population is calculated, and the moths in the moth population are sorted according to the fitness of the initial moth population to generate the first generation of flames.

[0080] According to the initial moth population, the first generation of flames, the constrained Brownian motion mechanism and the elite strategy, several iterations are performed, and the flame obtained in the last iteration is used as the second action, and the fitness of the moth population obtained in the last iteration is used as the fitness of the second action.

[0081] S300. Process the second action using the deep reinforcement learning algorithm to obtain a next state and a first reward; update the parameters of the deep reinforcement learning algorithm according to the current state, the first reward, the fitness of the second action, and the next state.

[0082] The content in the next state is the same as that in the first state, except that the value of each business process is different.

[0083] Specifically, Figure 3 As shown, Figure 3 It is a simplified deep reinforcement learning model. In the figure, S0 represents the state, including the current state s and the next state s′, a represents the action, including the first action and the second action. The second action is applied in the deep reinforcement learning environment of the deep reinforcement learning algorithm to obtain the next state and the first reward of multiple business processes. After substituting the second action into the deep reinforcement learning algorithm, its specific process is the same as the processing process of the deep reinforcement learning algorithm in the prior art.

[0084] The current state, the first reward, the fitness of the second action, and the next state are stored in the experience pool as a tuple.

[0085] The first reward r1 and the fitness of the second action r2 are weighted and summed according to the preset ratio to obtain the second reward. The probability ratio r of the deep reinforcement learning algorithm is calculated based on the current state s and the next state s′. t (θ), according to the second reward r and the probability ratio r t(θ), the current state s and the next state s′ are substituted into the first loss function set of the deep reinforcement learning algorithm, the current state is horizontally flipped to obtain the first state, the first state is input into the deep reinforcement learning algorithm to obtain the second state set, and the second loss function set is obtained according to the second state set and the first loss function set. The parameters of the deep reinforcement learning algorithm are updated according to the second loss function set and the back propagation algorithm.

[0086] In the next iteration, the next state is used as the current state of the deep reinforcement learning algorithm, the current state and the total reward r are input into the agent, and step S100 is executed.

[0087] S400, taking the next state as the current state, returning to execute the first action of processing the current state using the deep reinforcement learning algorithm until a preset condition is met, and outputting the next state that meets the preset condition as an optimization solution for multiple business processes.

[0088] The preset conditions may be, but are not limited to, the preset number of iterations or the satisfaction of multi-objective optimization conditions. The multi-objective optimization conditions may be, but are not limited to, the minimum cost, the shortest time, etc.

[0089] Specifically, when the number of iterations reaches a preset number or satisfies the multi-objective optimization conditions, the iteration ends and the next state that satisfies the preset conditions is output as the optimization solution for the multi-business processes.

[0090] If the preset number of times is not reached or the multi-objective optimization conditions are not met, the next state is taken as the current state, and the process returns to S100 to process the current state using the deep reinforcement learning algorithm, and the first action is obtained and executed until the next state that meets the conditions is obtained, and the next state that meets the preset conditions is output as the optimization plan for the multi-business process.

[0091] The optimization plan for each business process is issued to the corresponding business process respectively, so that the corresponding business process can be produced according to the optimization plan.

[0092] It is worth noting that, in the above iteration process, the first action and the second action both represent the amount of change in each iteration process.

[0093] In some embodiments, Figure 4 As shown, Figure 4 The invention is a flowchart of steps for obtaining a second action and the fitness of the second action. In step S200, the moth-to-flame algorithm is used to optimize each of the first actions respectively to obtain the second action corresponding to each of the first actions and the fitness of the second action, which specifically includes:

[0094] S210 , processing the first action by using chaotic mapping and reverse learning to generate an initial moth population, and obtaining a first generation of flames according to the initial moth population.

[0095] In some embodiments, Figure 5 and Figure 6 As shown, Figure 5 is a flow chart of the steps to get the initial moth population and the first generation of flames, Figure 6 It is an iterative flow chart of the moth-to-flame algorithm, and step S210 specifically includes the following steps:

[0096] S211. Use the chaotic mapping to perform a mapping operation on the first action to obtain a mapped moth population.

[0097] Specifically, the Kent chaotic mapping is first used to map the first action to obtain the moth population after mapping. For each first action, it is substituted into the Kent chaotic mapping formula to obtain the moth population x after mapping. i+1 , as shown in formula (5):

[0098]

[0099] Among them, x i is the first action before updating, and γ is a random number between 0 and 1.

[0100] Through this step, the diversity of the moth population can be improved.

[0101] S212: Perform random operations on the mapped moth population using the reverse learning to obtain the initial moth population.

[0102] Specifically, the mapped moth population and the random number rand1 are substituted into the preset reverse learning formula to obtain the initial moth population, as shown in formula (6):

[0103] x i+2 =Ub1+Lb1-rand1*x i+1 (6)

[0104] Among them, Ub1 is the upper bound of the moth population (variable), Lb1 is the lower bound of the moth population (variable), and rand1 is a random number between 0 and 1.

[0105] S213, calculating the fitness of the initial moth population, and screening the initial moth population according to the fitness of the initial moth population to obtain the first generation of flames.

[0106] Specifically, the fitness of the initial moth population is calculated, and the moths in the initial moth population whose fitness is greater than a preset value are retained as the first generation flames, or the N moths with the largest fitness are used as the first generation flames.

[0107] S220, iterating according to the Brownian motion mechanism, the elite strategy, the initial moth population, and the first generation of flames to obtain the second action and the fitness of the second action.

[0108] Specifically, Figure 7 As shown, Figure 7 The second action and the fitness of the second action are concrete steps of the flow chart, and step S220 specifically includes the following steps:

[0109] S221. Assign the initial moth population to the current moth population, and assign the first generation of flames to the current flames.

[0110] Specifically, the initial moth population is assigned to the contemporary moth population, and the first generation of flames is assigned to the contemporary flames, and the iteration begins.

[0111] S222. Use the Brownian motion mechanism to update the current moth population and obtain the next generation of moth population.

[0112] Specifically, step S222a is first performed to calculate the scale factor k of the Brownian motion mechanism using a preset activation function, as shown in formulas (7)-(8):

[0113]

[0114] Among them, Ub2 is the upper bound of k, Lb2 is the lower bound of k, sigmoid(x) is the activation function,

[0115] Execute step S222b, substitute the scale factor, random number and current moth population into the Brownian motion mechanism to obtain the next generation moth population, as shown in formula (9):

[0116]

[0117] Among them, x k is the contemporary moth population, x k+1 is the next generation of moth population, and rand2 is a random number between 0 and 1.

[0118] Using the Brownian motion mechanism to update the moth population can reduce the situation where the moth-to-fire algorithm quickly falls into locality during iteration and fails to obtain the optimal solution for a single business process, thereby improving the accuracy of single business process optimization.

[0119] S223, calculating the fitness of the next generation moth population, sorting the next generation moth population according to the fitness of the next generation moth population to obtain the sorted next generation moth population, and using the elite strategy to screen the sorted next generation moth population to obtain the next generation flame.

[0120] Specifically, the fitness of each moth in the next generation of moth population is calculated, and the crowding distance of each moth is calculated according to the fitness of the moth. Taking the i-th moth as an example, the maximum fitness f in the next generation of moth population is determined. j max , the minimum fitness f j min , the fitness of the i-1th moth f j i-1 and the fitness of the i+1th moth f j i+1 , according to the maximum fitness f j max , the minimum fitness f j min , the fitness of the i-1th moth f j i-1 and the fitness of the i+1th moth f j i+1 Calculate the crowding distance of the i-th moth as shown in formula (10):

[0121]

[0122] in, represents the crowding distance of the i-th moth to the j-th objective function.

[0123] The next generation of moth populations is sorted according to the crowding distance, and moths with larger crowding distances are preferentially selected to join the next generation of flames.

[0124] S224, determining whether the next generation flame meets the cutoff condition, if the next generation flame does not meet the cutoff condition, taking the next generation moth population as the contemporary moth population, and the next generation flame as the contemporary flame, returning to execute the Brownian motion mechanism to update the contemporary moth population to obtain the next generation moth population, until the obtained next generation flame meets the cutoff condition, outputting the next generation flame that meets the cutoff condition as the second action, and the fitness of the next generation moth population as the fitness of the second action.

[0125] Through the aforementioned steps, the present invention uses chaotic mapping, reverse learning, Brownian motion mechanism and elite strategy to improve the traditional multi-objective moth-to-flame optimization algorithm, and the improved algorithm can better solve the multi-objective optimization problem.

[0126] In some embodiments, Figure 8 As shown, Figure 8The method is a flowchart of the steps of updating the parameters of a deep reinforcement learning algorithm. In step S300, the parameters of the deep reinforcement learning algorithm are updated according to the current state, the first reward, the fitness of the second action and the next state, specifically including:

[0127] S310: Perform a weighted summation of the first reward and the fitness of the second action according to a preset ratio to obtain a second reward.

[0128] Specifically, when updating the parameters of the deep reinforcement learning algorithm, the total reward of the deep reinforcement learning algorithm consists of two parts. The first part is to reward the operation of all business processes: when each business process runs smoothly, the agent is positively rewarded, and when there are delivery delays, insufficient production resources, etc. between business processes, the agent is punished, which is the first reward obtained above. The second part is given by the single business process optimization plan executed by the multi-objective moth-to-flame optimization algorithm, and the effect of executing the business optimization plan in the current state is evaluated, that is, the fitness of the second action. The ultimate goal of the reward is to guide the agent to execute the corresponding optimization plan under different states and achieve global coordinated optimization.

[0129] In this embodiment, the first reward and the fitness of the second action can be weighted and summed according to a preset ratio to obtain the second reward r, as shown in formula (11):

[0130] r=c1*r1+c2*r2 (11)

[0131] Among them, c1 and c2 are custom weight coefficients, r1 is the first reward, and r2 is the fitness of the second action.

[0132] S320. Substitute the second reward, the current state, and the next state into the deep reinforcement learning algorithm to obtain a first loss function set.

[0133] Among them, the first loss function set includes the first policy network loss function and the first value network loss function of the deep reinforcement learning algorithm itself.

[0134] Specifically, substituting the second reward, the current state, and the next state into the deep reinforcement learning algorithm to obtain the first loss function set is the same as the existing algorithm and will not be repeated here.

[0135] S330. Flip the current state horizontally to obtain a first state, input the first state into a deep reinforcement learning algorithm to obtain a second state set, obtain a second loss function set based on the second state set and the first loss function set, and update the parameters of the deep reinforcement learning algorithm based on the second loss function set and a back propagation algorithm.

[0136] The second state set includes the output of the policy network and the output of the value network in the first state.

[0137] The second loss function set includes a second policy network loss function and a second value network loss function.

[0138] Specifically, the current state is horizontally flipped to obtain the first state. Horizontal flipping can be interpreted as swapping the first element of each row in the current state with the last element, the second element with the n-1th element, and so on. The first state is input into the policy network and value network of the deep reinforcement learning algorithm respectively to obtain the output of the policy network and the output of the value network. The second policy network loss function and the second value network loss function are calculated based on the output of the policy network, the output of the value network, the first policy network loss function and the first value network loss function. As shown in formulas (12)-(13):

[0139]

[0140]

[0141] Among them, L p (θ) is the first policy network loss function; L v (θ) is the first value network loss function; are all preset weight coefficients, indicating the proportion of image loss; π θ (a|s) is the output of the policy network under the original current state s; π θ (a m |s m ) represents the output of the policy network in the mirror environment (first state); |A(s)| represents the size of the action space; V θ (s) represents the output of the value network in the original current state; V θm (s m ) represents the output of the value network in the mirror environment (first state); L ′ p (θ) represents the loss function of the second strategy network; L v ′(θ) represents the second value network loss function.

[0142] The second policy network loss function and back propagation algorithm are used to update the parameters of the policy network, and the second value network loss function and back propagation algorithm are used to update the parameters of the value network.

[0143] In step S300, the supplier management solution is optimized using the moth-to-flame algorithm:

[0144] Aiming at the problem of supplier selection and order quantity allocation, a mathematical model of multi-objective optimization problem is established with quality, cost and delivery time as objectives, and a multi-objective moth-to-flame optimization algorithm is used to solve it.

[0145] Use the moth-to-flame algorithm to optimize the layout planning scheme:

[0146] Aiming at the production scheduling problem of mixed-flow assembly line, a dual-objective optimization model with the goals of minimizing the final assembly time and minimizing the product switching setup time is established, and the multi-objective moth-to-flame optimization algorithm is used to solve it.

[0147] Using the moth-to-fire algorithm to optimize the transport vehicle path planning scheme:

[0148] Transport vehicle path planning belongs to the vehicle routing problem (VRP). Its main goal is to arrange a certain number of vehicles to start from the distribution center and provide goods or services to different targets along a specific route to minimize the total driving distance or total transportation cost.

[0149] For the vehicle routing problem, considering the three objectives of shortest distance, shortest time and least cost in the vehicle distribution route, a mathematical model of multi-objective optimization problem is established, and the multi-objective moth-to-flame optimization algorithm is used to solve it.

[0150] Using the moth-to-flame algorithm to optimize storage planning:

[0151] Aiming at the problem of storage location allocation, the storage environment is modeled by reinforcement learning, and a reinforcement learning interactive environment that follows the Last In First Out (LIFO) constraint and hinders the relocation of goods is constructed. The goal is to minimize non-productive movement and the shortest movement path, and a deep reinforcement learning algorithm is used to solve it.

[0152] The implementation of the present invention includes the following beneficial effects: the present invention integrates multiple business processes involved in the production process as the intelligent body of the deep reinforcement learning algorithm, obtains the actions of the deep reinforcement learning algorithm, and then uses the moth-to-fire algorithm to optimize each business process (i.e., each action of the deep reinforcement learning algorithm) separately, and then involves the fitness of the moth-to-fire algorithm in the reward of the deep reinforcement learning algorithm, and substitutes the single-process business optimized by the moth-to-fire algorithm into the deep reinforcement learning algorithm to obtain the optimal solution for multiple business processes, which can realize collaborative optimization between multiple different business processes and achieve global optimization.

[0153] The present invention can improve the diversity of moth populations through chaotic mapping and reverse learning. By adopting the Brownian motion mechanism to update the moth population, it can reduce the situation where the moth-to-fire algorithm quickly falls into locality during iteration and cannot obtain the optimal solution for a single business process, thereby improving the optimization accuracy of a single business process.

[0154] The present invention uses chaos mapping, reverse learning, Brownian motion mechanism and elite strategy to improve the traditional multi-objective moth-to-flame optimization algorithm. The improved algorithm can better solve the multi-objective optimization problem to obtain a better optimization solution.

[0155] like Fig. 9 As shown, Fig. 9 The present invention is a structural diagram of a collaborative optimization device for multiple business processes. The present invention also provides a collaborative optimization device for multiple business processes, including:

[0156] at least one processor;

[0157] at least one memory for storing at least one program;

[0158] When the at least one program is executed by the at least one processor, the at least one processor implements the above-described method.

[0159] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0160] In addition, the embodiment of the present application also discloses a computer program product or a computer program, which is stored in a computer-readable storage medium. The processor of the computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the above method.

[0161] The present invention also provides a computer-readable storage medium, which stores a program executable by a processor, and the program executable by the processor is used to implement the above method when executed by the processor. Similarly, the contents of the above method embodiment are all applicable to the present storage medium embodiment, and the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method embodiment.

[0162] It is understood that all or some steps and systems in the disclosed method above can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0163] like Fig.10 As shown, Fig.10 The present invention also provides a collaborative optimization system for multiple business processes, including a data sensing and acquisition device and a computer device connected to the data sensing and acquisition device, wherein the data sensing and acquisition device is installed in the business process; wherein,

[0164] The data sensing and acquisition device is used to collect the first status of a plurality of business processes and upload the first status to the computer device;

[0165] The computer device comprises:

[0166] at least one processor;

[0167] at least one memory for storing at least one program;

[0168] When the at least one program is executed by the at least one processor, the at least one processor implements the above-described method.

[0169] Specifically, for data sensing and acquisition equipment, it is mainly realized through sensors or cameras, and for computer equipment, it can be different types of electronic devices, including but not limited to desktop computers, laptops and other terminals.

[0170] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0171] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0172] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application. In the present invention, the same or substantially equivalent methods between different embodiments are applicable to other embodiments.

Claims

1. A collaborative optimization method for multiple business processes, characterized in that: The following steps are involved: Obtaining first states of several business processes, assigning the first states to the current state, and processing the current state using an agent of a deep reinforcement learning algorithm to obtain a first action set; wherein one first action represents one business process; and the agent of the deep reinforcement learning algorithm is obtained by improving the attention mechanism; Using the moth-to-flame algorithm to optimize each of the first actions respectively, to obtain a second action corresponding to each of the first actions and the fitness of the second action; Applying the second action to the environment of the deep reinforcement learning algorithm to obtain a next state and a first reward; updating parameters of the deep reinforcement learning algorithm according to the current state, the first reward, the fitness of the second action, and the next state; Taking the next state as the current state, returning to execute the intelligent agent using the deep reinforcement learning algorithm to process the current state, obtaining a first action set, until the preset conditions are met, and outputting the current state that meets the preset conditions as an optimization solution for multiple business processes.

2. The method according to claim 1, characterized in that The using the moth-to-flame algorithm to optimize each of the first actions respectively to obtain the second action corresponding to each of the first actions and the fitness of the second action specifically includes: Processing the first action by using chaotic mapping and reverse learning to generate an initial moth population, and obtaining a first generation of flames according to the initial moth population; Iteration is performed according to the Brownian motion mechanism, the elite strategy, the initial moth population and the first generation of flames to obtain the second action and the fitness of the second action.

3. The method according to claim 2, characterized in that The method of processing the first action by using chaotic mapping and reverse learning to generate an initial moth population, and obtaining a first generation of flames according to the initial moth population, specifically includes: Using the chaotic mapping to perform a mapping operation on the first action to obtain a moth population after mapping; Using the reverse learning, performing random operations on the mapped moth population to obtain the initial moth population; The fitness of the initial moth population is calculated, and the initial moth population is screened according to the fitness of the initial moth population to obtain the first generation of flames.

4. The method according to claim 2, characterized in that: The iterating according to the Brownian motion mechanism, the elite strategy, the initial moth population and the first generation of flames to obtain the second action and the fitness of the second action specifically includes: Assign the initial moth population to the current moth population, and the first generation flame to the current flame; Using the Brownian motion mechanism to update the current moth population to obtain the next generation moth population; Calculating the fitness of the next generation moth population, sorting the next generation moth population according to the fitness of the next generation moth population to obtain the sorted next generation moth population, and screening the sorted next generation moth population using the elite strategy to obtain the next generation flame; Determine whether the next generation flame meets the cutoff condition. If the next generation flame does not meet the cutoff condition, use the next generation moth population as the contemporary moth population and the next generation flame as the contemporary flame, return to execute the Brownian motion mechanism to update the contemporary moth population to obtain the next generation moth population, until the obtained next generation flame meets the cutoff condition, output the next generation flame that meets the cutoff condition as the second action, and the fitness of the next generation moth population is used as the fitness of the second action.

5. The method according to claim 4, characterized in that The method of updating the contemporary moth population by using the Brownian motion mechanism to obtain the next generation moth population specifically includes: Calculating the scale factor of the Brownian motion mechanism using a preset activation function; Substituting the scale factor, random number and the current moth population into the Brownian motion mechanism, the next generation moth population is obtained.

6. The method according to claim 4, characterized in that The step of sorting the next generation moth population according to the fitness of the next generation moth population to obtain the sorted next generation moth population specifically includes: Calculating the crowding distance of each moth according to the fitness of the next generation moth population; wherein the moth population includes a plurality of moths, and each moth represents a solution to a business process; The next generation moth population is sorted according to the crowding distance to obtain a sorted next generation moth population.

7. The method according to any one of claims 1 to 6, characterized in that: The updating of the parameters of the deep reinforcement learning algorithm according to the current state, the first reward, the fitness of the second action and the next state specifically includes: Performing a weighted summation of the first reward and the fitness of the second action according to a preset ratio to obtain a second reward; Substituting the second reward, the current state, and the next state into the deep reinforcement learning algorithm to obtain a first loss function set; The current state is horizontally flipped to obtain a first state, the first state is input into a deep reinforcement learning algorithm to obtain a second state set, a second loss function set is obtained according to the second state set and the first loss function set, and the parameters of the deep reinforcement learning algorithm are updated according to the second loss function set and a back propagation algorithm.

8. A collaborative optimization device for multiple business processes, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 7 when executed by the processor.

10. A collaborative optimization system for multiple business processes, characterized in that: It includes a data sensing and acquisition device and a computer device connected to the data sensing and acquisition device, wherein the data sensing and acquisition device is installed in a business process; wherein, The data sensing and acquisition device is used to collect the first status of a plurality of business processes and upload the first status to the computer device; The computer device comprises: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

Images