A multi-service process collaborative optimization method, system, device and medium

Abstract

The application discloses a kind of multi-service process collaborative optimization method, system, device and medium, comprising: obtaining the first state of several service processes and assigning to current state, using deep reinforcement learning algorithm to process current state, obtain first action, a first action represents a service process, using moth to optimize each first action respectively, obtain the second action of each first action and the fitness of second action;Using deep reinforcement learning algorithm to process second action, obtain next state and first reward, according to current state, first reward, fitness and next state update the parameters of deep reinforcement learning algorithm, following next state is current state, return to execute the first action of using deep reinforcement learning algorithm to process current state, until meeting preset condition, output next state meeting preset condition as the optimization scheme of multi-service process.The application can realize the global optimum of multiple service processes, and can be applied to multi-process scheduling technical field.

Description

Technical Field

[0001] This invention relates to the field of multi-process scheduling technology, and in particular to a collaborative optimization method, system, device and medium for multiple business processes. Background Technology

[0002] When urgent production orders arise, it's necessary to adjust not only production scheduling but also procurement, transportation, and warehousing plans. Existing optimization methods typically only optimize a single business process locally. However, the various business processes involved in the product lifecycle are closely intertwined. Optimizing a single business process cannot meet the needs for efficient integration and collaborative optimization control of multiple cross-system business processes, such as supply chain management, production scheduling, route planning, and warehousing planning. Considering the complexity, diversity, poor interoperability, and intricate collaborative interaction processes of cross-system business processes, only local optimization of a single business process can be achieved, not global optimization. Summary of the Invention

[0003] To address one of the aforementioned problems, the present invention aims to provide a method, system, apparatus, and medium for collaborative optimization of multiple business processes, which can achieve collaborative optimization among multiple business processes to reach global optimality.

[0004] On the one hand, this invention provides a collaborative optimization method for multiple business processes, comprising the following steps:

[0005] A first state of several business processes is obtained, and the first state is assigned to the current state. The current state is processed by an agent using a deep reinforcement learning algorithm to obtain a first action set. Each first action represents a business process. The agent of the deep reinforcement learning algorithm is obtained by improving the attention mechanism.

[0006] The first action is optimized using the moth-to-a-flame algorithm to obtain the second action and the fitness of the second action corresponding to each first action.

[0007] The second action is applied to the environment of the deep reinforcement learning algorithm to obtain the next state and the first reward; the parameters of the deep reinforcement learning algorithm are updated according to the current state, the first reward, the fitness of the second action and the next state.

[0008] Using the next state as the current state, the agent that uses a deep reinforcement learning algorithm is returned to process the current state to obtain the first action, until a preset condition is met, and the current state that meets the preset condition is output as an optimization scheme for the multi-business process.

[0009] Optionally, the step of using the moth-to-a-flame algorithm to optimize each of the first actions to obtain the second action and the fitness of each of the first actions specifically includes:

[0010] The first action is processed using chaotic mapping and reverse learning to generate an initial moth population, and the first generation of flames is obtained based on the initial moth population.

[0011] The second action and its fitness are obtained by iterating based on the Brownian motion mechanism, the elite strategy, the initial moth population, and the first generation of flames.

[0012] Optionally, the step of using chaotic mapping and reverse learning to process the first action, generating an initial moth population, and obtaining the first-generation flame based on the initial moth population specifically includes:

[0013] The first action is mapped using the chaotic mapping to obtain the mapped moth population.

[0014] The initial moth population is obtained by performing random operations on the mapped moth population using the reverse learning;

[0015] The fitness of the initial moth population is calculated, and the initial moth population is screened based on the fitness of the initial moth population to obtain the first generation of flames.

[0016] Optionally, the step of iterating based on Brownian motion mechanism, elite strategy, the initial moth population, and the first generation of flames to obtain the second action and the fitness of the second action specifically includes:

[0017] The initial moth population is assigned to the current moth population, and the first generation flame is assigned to the current flame.

[0018] The current moth population is updated using the Brownian motion mechanism to obtain the next generation of moth population;

[0019] Calculate the fitness of the next generation of moths, sort the next generation of moths according to the fitness, obtain the sorted next generation of moths, and use the elite strategy to screen the sorted next generation of moths to obtain the next generation of flames.

[0020] Determine whether the next-generation flame meets the cutoff condition. If the next-generation flame does not meet the cutoff condition, use the next-generation moth population as the current moth population and the next-generation flame as the current flame. Return to execute the Brownian motion mechanism to update the current moth population and obtain the next-generation moth population. Continue until the obtained next-generation flame meets the cutoff condition. Output the next-generation flame that meets the cutoff condition as the second action, and use the fitness of the next-generation moth population as the fitness of the second action.

[0021] Optionally, updating the current moth population using the Brownian motion mechanism to obtain the next generation of moth population specifically includes:

[0022] The scale factor of the Brownian motion mechanism is calculated using a preset activation function;

[0023] Substituting the scale factor, random number, and contemporary moth population into the Brownian motion mechanism yields the next generation moth population.

[0024] Optionally, the step of sorting the next generation of moths according to their fitness to obtain a sorted next generation of moths specifically includes:

[0025] The crowding distance of each moth is calculated based on the fitness of the next generation of moth population; wherein, the moth population includes several moths, and each moth represents a solution to a business process.

[0026] The next generation of moths is sorted according to the crowding distance to obtain the sorted next generation of moths.

[0027] Optionally, updating the parameters of the deep reinforcement learning algorithm based on the current state, the first reward, the fitness of the second action, and the next state specifically includes:

[0028] The fitness of the first reward and the second action are weighted and summed according to a preset ratio to obtain the second reward;

[0029] Substituting the second reward, the current state, and the next state into the deep reinforcement learning algorithm yields the first loss function set;

[0030] The current state is horizontally flipped to obtain a first state. The first state is input into a deep reinforcement learning algorithm to obtain a second state set. A second loss function set is obtained based on the second state set and the first loss function set. The parameters of the deep reinforcement learning algorithm are updated based on the second loss function set and the backpropagation algorithm.

[0031] On the other hand, the present invention provides a collaborative optimization device for multiple business processes, comprising:

[0032] At least one processor;

[0033] At least one memory for storing at least one program;

[0034] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0035] On the other hand, the present invention provides a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the methods described above.

[0036] On the other hand, the present invention provides a collaborative optimization system for multiple business processes, including a data sensing and acquisition device and a computer device connected to the data sensing and acquisition device, wherein the data sensing and acquisition device is installed in the business process; wherein,

[0037] The data sensing and acquisition device is used to collect the first state of several business processes and upload the first state to the computer device.

[0038] The computer device includes:

[0039] At least one processor;

[0040] At least one memory for storing at least one program;

[0041] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0042] Implementing this invention provides the following beneficial effects: This invention acquires the first states of several business processes, assigns these first states to the current state, and utilizes a deep reinforcement learning algorithm's agent to process the current state, obtaining a first action set. Each first action represents a business process. The deep reinforcement learning algorithm's agent is obtained through an improved attention mechanism. The "moth to a flame" algorithm is used to optimize each first action, obtaining the corresponding second action and its fitness. The second action is then applied to the environment of the deep reinforcement learning algorithm to obtain the next state and a first reward. The parameters of the deep reinforcement learning algorithm are updated based on the current state, the first reward, the fitness of the second action, and the next state, with the next state serving as the current state. The invention integrates multiple business processes involved in the production process as an intelligent agent of the deep reinforcement learning algorithm to obtain the actions of the deep reinforcement learning algorithm. Then, it uses the "moth to a flame" algorithm to optimize each business process (i.e., each action of the deep reinforcement learning algorithm) separately. The fitness of the "moth to a flame" algorithm is then incorporated into the reward of the deep reinforcement learning algorithm. The single-process business optimized by the "moth to a flame" algorithm is then substituted into the environment of the deep reinforcement learning algorithm to obtain the optimal solution for multiple business processes. This can achieve collaborative optimization between multiple different business processes and achieve global optimization. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the steps of a collaborative optimization method for multiple business processes provided by the present invention.

[0044] Figure 2 This is an iterative flowchart of a collaborative optimization method for multiple business processes provided by the present invention;

[0045] Figure 3 This invention provides a simplified deep reinforcement learning model;

[0046] Figure 4 This is a flowchart of the steps for obtaining the fitness of the second action and the second action provided by the present invention;

[0047] Figure 5 This invention provides a flowchart of the steps for obtaining an initial moth population and a first-generation flame.

[0048] Figure 6 This is an iterative flowchart of a moth-to-a-flame algorithm provided by the present invention;

[0049] Figure 7 This is a flowchart illustrating the specific steps of a second action and the adaptability of the second action provided by the present invention.

[0050] Figure 8 This is a flowchart illustrating the steps involved in updating the parameters of a deep reinforcement learning algorithm, as provided by this invention.

[0051] Figure 9 This is a structural diagram of a multi-business process collaborative optimization device provided by the present invention;

[0052] Figure 10 This is a structural diagram of a collaborative optimization system for multiple business processes provided by the present invention. Detailed Implementation

[0053] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0054] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0055] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first / S100," "second / S200," "third / S300," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0056] The problem that reinforcement learning algorithms study is how agents learn a policy within an environment to maximize long-term rewards. The mathematical foundation and modeling tool of reinforcement learning is the Markov Decision Process. Deep reinforcement learning combines the advantages of deep neural networks in modeling complex problems, solving sequential decision-making problems based on the principle of maximizing rewards.

[0057] like Figure 1 and Figure 2 As shown, Figure 1This is a flowchart illustrating the steps of a collaborative optimization method for multiple business processes. Figure 2 This is an iterative flowchart of a collaborative optimization method for multiple business processes. Figure 2 In this paper, Env represents the environment of a deep reinforcement learning algorithm, and Actor represents the intelligent agent of the deep reinforcement learning algorithm. This invention provides a collaborative optimization method for multiple business processes, including the following steps:

[0058] S100. Obtain the first state of several business processes, assign the first state to the current state, process the current state using a deep reinforcement learning algorithm, and obtain the first action.

[0059] In this context, a first action represents a business process.

[0060] The business processes can be, but are not limited to, supply chain management, procurement planning, production scheduling, transportation route planning, and warehousing planning involved in the production process.

[0061] Supply chain management and procurement planning include, but are not limited to, supplier selection, procurement quantity, procurement price, procurement goods, procurement quality, and the procurement quantity corresponding to each supplier.

[0062] Production scheduling can include, but is not limited to, the output, processing time, production line load, and production cost of each production line on a mixed-flow assembly line.

[0063] Transportation route planning may include, but is not limited to, the number of transport vehicles, the load capacity of each transport vehicle, the route of each transport vehicle, transportation time, and transportation costs.

[0064] Warehouse planning can include, but is not limited to, storage locations, storage capacity, location of incoming goods, and warehousing costs.

[0065] Specifically, the parameters of the deep reinforcement learning algorithm are initialized, and the current data of each business process is obtained as the first state. The first state includes several first actions. Each first action represents a business process. For example, in supply chain management and procurement planning, a first action can represent the current supplier selection, procurement quantity, procurement price, procurement goods, procurement quality, and the procurement quantity corresponding to each supplier in the business process.

[0066] The agent of the deep reinforcement learning algorithm is obtained by improving the attention mechanism. The improvement method is to add the attention mechanism to the policy network of the existing deep reinforcement learning algorithm. The existing deep reinforcement learning algorithm directly inputs the current state s into the policy network to obtain the first action a1. In this invention, the attention mechanism is first used to extract the feature representation of the current state s, and the feature representation is input into the policy network. The feature representation extraction process can be shown in formulas (1)-(4):

[0067] Q = s * ω_Q + b_Q (1)

[0068] K = s * ω_K + b_K (2)

[0069] V = s * ω_V + b_V (3)

[0070]

[0071] Where ω_Q, ω_K, and ω_V are the weight parameters to be learned, and b_Q, b_K, and b_V are the bias parameters to be learned. Q, K, and V are three intermediate feature representations of s, s1′ is the feature representation of the current state s, and K... T It is the transpose of K, d k It is the dimension of K.

[0072] After the first state is obtained and assigned to the current state s, the feature representation s1′ of the current state s is obtained. The policy network processes the feature representation s1′ to obtain the first action a1. The specific process is the same as the processing process of the policy network in the deep reinforcement learning algorithm in the prior art. The difference is that in this invention, the agent inputting the deep reinforcement learning algorithm consists of multiple business processes.

[0073] When using deep reinforcement learning algorithms to process the current state and obtain the first action, multiple objective functions can be set based on factors such as quality, cost, efficiency, and energy consumption, and constraints such as the working environment and time can be set to obtain the optimal solution.

[0074] Deep reinforcement learning algorithms can be, but are not limited to, proximal policy optimization (PPO) algorithms or actor-critic algorithms.

[0075] S200. Optimize each of the first actions using the moth-to-a-flame algorithm to obtain the second action and the fitness of each of the first actions.

[0076] In this step, the moth-to-a-flame algorithm can also be a particle swarm algorithm or a genetic algorithm.

[0077] In the moth-to-flame algorithm, each generation of moths represents a business process, and the flame represents the optimal solution in a business process.

[0078] When using the moth-to-a-flame algorithm to find the optimal solution for a single business process and obtain the fitness of the second action, multiple objective functions can be set for the single business process with the best quality, lowest cost, highest efficiency, and lowest energy consumption.

[0079] Specifically, the parameters of the multi-target moth-to-fire optimization algorithm are initialized, and the first action is processed using chaotic mapping combined with reverse learning to obtain the initial moth population for the moth-to-fire algorithm. The fitness of the initial moth population is calculated, and the moths in the initial moth population are sorted according to the fitness of the initial moth population to generate the first generation of flames.

[0080] Based on the initial moth population, the first-generation flame, the constrained Brownian motion mechanism, and the elite strategy, several iterations are performed. The flame obtained in the last iteration is used as the second action, and the fitness of the moth population obtained in the last iteration is used as the fitness of the second action.

[0081] S300: Process the second action using the deep reinforcement learning algorithm to obtain the next state and the first reward; update the parameters of the deep reinforcement learning algorithm according to the current state, the first reward, the fitness of the second action, and the next state.

[0082] The content in the next state is the same as that in the first state, except that the values ​​for each business process are different.

[0083] Specifically, such as Figure 3 As shown, Figure 3 It is a simplified deep reinforcement learning model. In the figure, S0 represents the state, including the current state s and the next state s′, and a represents the action, including the first action and the second action. The second action is applied in the deep reinforcement learning environment of the deep reinforcement learning algorithm to obtain the next state and the first reward of multiple business processes. After the second action is substituted into the deep reinforcement learning algorithm, its specific process is the same as the processing process of the deep reinforcement learning algorithm in the existing technology.

[0084] Store the current state, first reward, fitness of the second action, and next state as a tuple in the experience pool.

[0085] The first reward r1 and the fitness of the second action r2 are weighted and summed according to a preset ratio to obtain the second reward. The probability ratio r of the deep reinforcement learning algorithm is calculated based on the current state s and the next state s′. t (θ), based on the second reward r and the probability ratio r tSubstituting (θ), the current state s, and the next state s′ into the first loss function set of the deep reinforcement learning algorithm, the current state is horizontally flipped to obtain the first state. The first state is then input into the deep reinforcement learning algorithm to obtain the second state set. The second loss function set is obtained based on the second state set and the first loss function set. The parameters of the deep reinforcement learning algorithm are updated based on the second loss function set and the backpropagation algorithm.

[0086] In the next iteration, the next state is used as the current state of the deep reinforcement learning algorithm. The current state and the total reward r are input into the agent, and step S100 is executed.

[0087] S400. Using the next state as the current state, return to execute the first action of processing the current state using a deep reinforcement learning algorithm until a preset condition is met, and output the next state that meets the preset condition as an optimization scheme for the multi-service process.

[0088] Preset conditions can be, but are not limited to, a preset number of iterations or satisfying multi-objective optimization conditions. Multi-objective optimization conditions can be, but are not limited to, minimum cost, minimum time, etc.

[0089] Specifically, when the number of iterations reaches a preset number or the multi-objective optimization conditions are met, the iteration ends, and the next state that meets the preset conditions is output as the optimization scheme for the multi-business process.

[0090] If the preset number of iterations is not reached or the multi-objective optimization conditions are not met, the next state is taken as the current state. The process starts from S100, which uses a deep reinforcement learning algorithm to process the current state and obtain the first action. This continues until the next state that meets the conditions is obtained. The next state that meets the preset conditions is then output as the optimization scheme for the multi-business process.

[0091] The optimization plan for each business process is distributed to the corresponding business process so that the corresponding business process can produce according to the optimization plan.

[0092] It is worth noting that in the above iteration process, both the first action and the second action represent the amount that changes in each iteration.

[0093] In some embodiments, such as Figure 4 As shown, Figure 4 This is a flowchart illustrating the steps for obtaining the fitness of a second action and a second action. In step S200, the moth-to-a-flame algorithm is used to optimize each of the first actions to obtain the second action and the fitness of the second action corresponding to each of the first actions. Specifically, this includes:

[0094] S210. The first action is processed using chaotic mapping and reverse learning to generate an initial moth population, and the first generation of flames is obtained based on the initial moth population.

[0095] In some embodiments, such as Figure 5 and Figure 6 As shown, Figure 5 This is a flowchart outlining the steps to obtain an initial moth population and the first generation of flames. Figure 6 This is an iterative flowchart of a moth-to-a-flame algorithm. Step S210 specifically includes the following steps:

[0096] S211. The first action is mapped using the chaotic mapping to obtain the mapped moth population.

[0097] Specifically, the first action is mapped using Kent's chaotic mapping to obtain the mapped moth population. For each first action, it is substituted into the Kent's chaotic mapping formula to obtain the mapped moth population x. i+1 As shown in formula (5):

[0098]

[0099] Where, x i This is the first action before the update, where γ is a random number between 0 and 1.

[0100] This step can increase the diversity of moth populations.

[0101] S212. Using the reverse learning, perform random operations on the mapped moth population to obtain the initial moth population.

[0102] Specifically, the mapped moth population and the random number rand1 are substituted into the preset reverse learning formula to obtain the initial moth population, as shown in formula (6):

[0103] x i+2 =Ub1+Lb1-rand1*x i+1 (6)

[0104] Where Ub1 is the upper bound of the moth population (variable), Lb1 is the lower bound of the moth population (variable), and rand1 is a random number between 0 and 1.

[0105] S213. Calculate the fitness of the initial moth population, and select the initial moth population according to the fitness of the initial moth population to obtain the first generation of flames.

[0106] Specifically, the fitness of the initial moth population is calculated, and moths with fitness greater than a preset value in the initial moth population are retained as the first generation of flames, or the N moths with the highest fitness are selected as the first generation of flames.

[0107] S220. Based on the Brownian motion mechanism, elite strategy, the initial moth population and the first generation of flames, iterate to obtain the second action and the fitness of the second action.

[0108] Specifically, such as Figure 7 As shown, Figure 7 This is a flowchart illustrating the specific steps of a second action and the fitness of that second action. Step S220 specifically includes the following steps:

[0109] S221. Assign the initial moth population to the current moth population, and assign the first generation flame to the current flame.

[0110] Specifically, the initial moth population is assigned to the current moth population, and the first-generation flame is assigned to the current flame, thus entering the iteration process.

[0111] S222. Utilize the Brownian motion mechanism to update the current generation of moth populations and obtain the next generation of moth populations.

[0112] Specifically, first execute step S222a, and use the preset activation function to calculate the scale factor k of the Brownian motion mechanism, as shown in formulas (7)-(8):

[0113]

[0114] Where Ub2 is the upper bound of k, Lb2 is the lower bound of k, and sigmoid(x) is the activation function.

[0115] Perform step S222b, substituting the scale factor, random number, and current moth population into the Brownian motion mechanism to obtain the next generation moth population, as shown in formula (9):

[0116]

[0117] Where, x k It is a contemporary moth population, x k+1 It is the next generation of moth population, and rand2 is a random number between 0 and 1.

[0118] By using the Brownian motion mechanism to update the moth population, the moth-to-fire algorithm can be reduced from quickly getting stuck in local conditions during iteration, thus failing to obtain the optimal solution for a single business process and improving the accuracy of optimizing a single business process.

[0119] S223. Calculate the fitness of the next generation moth population, sort the next generation moth population according to the fitness of the next generation moth population, obtain the sorted next generation moth population, and use the elite strategy to screen the sorted next generation moth population to obtain the next generation flame.

[0120] Specifically, the fitness of each moth in the next generation of moth population is calculated, and the crowding distance of each moth is calculated based on the fitness. Taking the i-th moth as an example, the maximum fitness f in the next generation of moth population is determined. j max Minimum fitness f j min The fitness f of the (i-1)th moth j i-1 The fitness f of the (i+1)th moth j i+1 Based on the maximum fitness f j max Minimum fitness f j min The fitness f of the (i-1)th moth j i-1 The fitness f of the (i+1)th moth j i+1 The crowding distance of the i-th moth is calculated as shown in formula (10):

[0121]

[0122] in, Let represent the crowding distance of the i-th moth to the j-th objective function.

[0123] The next generation of moths was sorted according to their crowding distance, with priority given to moths with larger crowding distances to be added to the next generation of flames.

[0124] S224. Determine whether the next generation flame meets the cutoff condition. If the next generation flame does not meet the cutoff condition, use the next generation moth population as the current moth population and the next generation flame as the current flame. Return to execute the Brownian motion mechanism to update the current moth population and obtain the next generation moth population. Continue until the obtained next generation flame meets the cutoff condition. Output the next generation flame that meets the cutoff condition as the second action, and use the fitness of the next generation moth population as the fitness of the second action.

[0125] Through the aforementioned steps, this invention improves the traditional multi-objective moth-to-flame optimization algorithm by using chaotic mapping, back learning, Brownian motion mechanism, and elite strategy. The improved algorithm can better solve multi-objective optimization problems.

[0126] In some embodiments, such as Figure 8 As shown, Figure 8This is a flowchart illustrating the steps for updating the parameters of a deep reinforcement learning algorithm. Step S300, which involves updating the parameters of the deep reinforcement learning algorithm based on the current state, the first reward, the fitness of the second action, and the next state, specifically includes:

[0127] S310. The fitness of the first reward and the second action are weighted and summed according to a preset ratio to obtain the second reward.

[0128] Specifically, when updating the parameters of the deep reinforcement learning algorithm, the total reward consists of two parts. The first part rewards the agent based on the performance of all business processes: when each business process runs smoothly, the agent is positively rewarded; when delivery delays or insufficient production resources occur between business processes, the agent is penalized—this is the aforementioned first reward. The second part is given by the optimization scheme of a single business process executed by the multi-objective moth-to-a-flame optimization algorithm, evaluating the effectiveness of executing this optimization scheme in the current state—this is the fitness of the second action. The ultimate goal of the reward is to guide the agent to execute corresponding optimization schemes in different states, achieving global coordinated optimization.

[0129] In this embodiment, the fitness of the first reward and the second action can be weighted and summed according to a preset ratio to obtain the second reward r, as shown in formula (11):

[0130] r = c1*r1 + c2*r2 (11)

[0131] Among them, c1 and c2 are custom weight coefficients, r1 is the first reward, and r2 is the fitness of the second action.

[0132] S320. Substitute the second reward, the current state, and the next state into the deep reinforcement learning algorithm to obtain the first loss function set.

[0133] The first set of loss functions includes the first policy network loss function and the first value network loss function of the deep reinforcement learning algorithm itself.

[0134] Specifically, the second reward, the current state, and the next state are substituted into the deep reinforcement learning algorithm to obtain the first loss function set, which is the same as that of the existing algorithm, and will not be repeated here.

[0135] S330. The current state is horizontally flipped to obtain a first state. The first state is input into the deep reinforcement learning algorithm to obtain a second state set. A second loss function set is obtained based on the second state set and the first loss function set. The parameters of the deep reinforcement learning algorithm are updated based on the second loss function set and the backpropagation algorithm.

[0136] The second state set includes the outputs of the policy network and the value network in the first state.

[0137] The second set of loss functions includes the second policy network loss function and the second value network loss function.

[0138] Specifically, the current state is horizontally flipped to obtain the first state. Horizontal flipping can be interpreted as swapping the first and last elements of each row in the current state, swapping the second element with the (n-1)th element, and so on. The first state is then input into the policy network and value network of the deep reinforcement learning algorithm to obtain the outputs of the policy network and value network, respectively. The second policy network loss function and the second value network loss function are calculated based on the outputs of the policy network, the value network, the first policy network loss function, and the first value network loss function. As shown in formulas (12)-(13):

[0139]

[0140]

[0141] Among them, L p (θ) is the first-policy network loss function; L v (θ) is the first value network loss function; These are all preset weighting coefficients, representing the proportion of mirror image loss; π θ (a|s) is the output of the policy network under the original current state s; π θ (a m |s m ) represents the output of the policy network in the mirror environment (first state); |A(s)| represents the size of the action space; V θ (s) represents the output of the value network in the original current state; V θm (s m ) represents the output of the value network in the mirrored environment (first state); L ′ p (θ) represents the loss function of the second policy network; L v ′(θ) represents the second value network loss function.

[0142] The parameters of the policy network are updated using the second policy network loss function and the backpropagation algorithm, and the parameters of the value network are updated using the second value network loss function and the backpropagation algorithm.

[0143] In step S300, the supplier management scheme is optimized using the moth-to-a-flame algorithm:

[0144] To address the problem of supplier selection and order quantity allocation, a mathematical model of a multi-objective optimization problem is established with quality, cost, and delivery time as objectives, and the multi-objective "moth to a flame" optimization algorithm is used to solve it.

[0145] Optimize the layout planning scheme using the moth-to-a-flame algorithm:

[0146] To address the scheduling problem in mixed-flow assembly lines, a dual-objective optimization model is established with the goals of minimizing final assembly time and minimizing product changeover setup time. This model is then solved using a multi-objective "moth to a flame" optimization algorithm.

[0147] Optimizing transport vehicle route planning using the moth-to-a-flame algorithm:

[0148] Transportation vehicle routing, a type of vehicle routing problem (VRP), aims to arrange a certain number of vehicles to depart from a distribution center and deliver goods or services to different destinations along specific routes in order to minimize the total travel distance or total transportation cost.

[0149] To address the vehicle routing problem, considering the three objectives of minimizing distance, time, and cost in the vehicle delivery route, a mathematical model for a multi-objective optimization problem is established, and the multi-objective "moth to a flame" optimization algorithm is used to solve it.

[0150] Optimize storage planning scheme using the moth-to-a-flame algorithm:

[0151] To address the storage location allocation problem, a reinforcement learning model is applied to the storage environment, constructing a reinforcement learning interaction environment that follows Last-In-First-Out (LIFO) constraints and hinders cargo relocation strategies. The objectives are to minimize unproductive movements and the shortest movement path, and a deep reinforcement learning algorithm is used to solve this problem.

[0152] Implementing this invention has the following beneficial effects: This invention integrates multiple business processes involved in the production process as an intelligent agent of a deep reinforcement learning algorithm to obtain the actions of the deep reinforcement learning algorithm. Then, the "moth to a flame" algorithm is used to optimize each business process (i.e., each action of the deep reinforcement learning algorithm) separately. The fitness of the "moth to a flame" algorithm is then incorporated into the reward of the deep reinforcement learning algorithm. By substituting the single-process business optimized by the "moth to a flame" algorithm into the deep reinforcement learning algorithm, the optimal solution for multiple business processes is obtained. This enables collaborative optimization among multiple different business processes to achieve global optimization.

[0153] This invention can improve the diversity of moth populations through chaotic mapping and reverse learning. By using the Brownian motion mechanism to update the moth population, it can reduce the situation where the moth-to-fire algorithm quickly gets stuck in local conditions during iteration and fails to obtain the optimal solution for a single business process, thereby improving the accuracy of optimizing a single business process.

[0154] This invention improves the traditional multi-objective moth-to-flame optimization algorithm by using chaotic mapping, back learning, Brownian motion mechanism and elite strategy. The improved algorithm can better solve multi-objective optimization problems and obtain better optimization solutions.

[0155] like Figure 9 As shown, Figure 9 This invention provides a structural diagram of a multi-business process collaborative optimization device, comprising:

[0156] At least one processor;

[0157] At least one memory for storing at least one program;

[0158] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0159] It is evident that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0160] Furthermore, embodiments of this application also disclose a computer program product or computer program stored in a computer-readable storage medium. A processor of a computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the methods described above.

[0161] The present invention also provides a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to implement the above-described method. Similarly, the content of the above method embodiments is applicable to this storage medium embodiment, and the specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0162] It is understood that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0163] like Figure 10 As shown, Figure 10 This invention provides a structural diagram of a multi-business process collaborative optimization system. The system includes a data sensing and acquisition device and a computer device connected to the data sensing and acquisition device. The data sensing and acquisition device is installed within the business process.

[0164] The data sensing and acquisition device is used to collect the first state of several business processes and upload the first state to the computer device.

[0165] The computer device includes:

[0166] At least one processor;

[0167] At least one memory for storing at least one program;

[0168] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0169] Specifically, data sensing and acquisition devices are mainly achieved through sensors or cameras, while computer devices can be different types of electronic devices, including but not limited to desktop computers, laptops and other terminals.

[0170] It is evident that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0171] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0172] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application. In the present invention, the same or substantially equivalent methods between different embodiments are applicable to other embodiments.

Claims

1. A collaborative optimization method for multiple business processes, characterized in that, Includes the following steps: A first state of several business processes is obtained, and the first state is assigned to the current state. The current state is processed by an agent using a deep reinforcement learning algorithm to obtain a first action set. Here, one first action represents one business process. The agent of the deep reinforcement learning algorithm is obtained by improving the attention mechanism. The first action is optimized using the moth-to-a-flame algorithm to obtain the second action and the fitness of the second action corresponding to each first action. The second action is applied to the environment of the deep reinforcement learning algorithm to obtain the next state and the first reward; the parameters of the deep reinforcement learning algorithm are updated according to the current state, the first reward, the fitness of the second action and the next state. Using the next state as the current state, the agent that uses a deep reinforcement learning algorithm is returned to process the current state to obtain the first action set until a preset condition is met. The current state that meets the preset condition is then output as an optimization scheme for the multi-business process. The process of optimizing each of the first actions using the moth-to-a-flame algorithm to obtain the second action and the fitness of each second action specifically includes: The first action is processed using chaotic mapping and reverse learning to generate an initial moth population, and the first generation of flames is obtained based on the initial moth population. The second action and its fitness are obtained by iterating based on the Brownian motion mechanism, the elite strategy, the initial moth population, and the first generation of flames. The process of using chaotic mapping and reverse learning to process the first action to generate an initial moth population, and obtaining the first generation of flames based on the initial moth population, specifically includes: The first action is mapped using the chaotic mapping to obtain the mapped moth population. The initial moth population is obtained by performing random operations on the mapped moth population using the reverse learning; The fitness of the initial moth population is calculated, and the initial moth population is screened based on the fitness of the initial moth population to obtain the first generation of flames; The process of iterating based on Brownian motion mechanisms, elite strategies, the initial moth population, and the first generation of flames to obtain the second action and its fitness specifically includes: The initial moth population is assigned to the current moth population, and the first generation flame is assigned to the current flame. The current moth population is updated using the Brownian motion mechanism to obtain the next generation of moth population; Calculate the fitness of the next generation of moths, sort the next generation of moths according to the fitness, obtain the sorted next generation of moths, and use the elite strategy to screen the sorted next generation of moths to obtain the next generation of flames. Determine whether the next-generation flame meets the cutoff condition. If the next-generation flame does not meet the cutoff condition, use the next-generation moth population as the current moth population and the next-generation flame as the current flame. Return to execute the Brownian motion mechanism to update the current moth population and obtain the next-generation moth population. Continue until the obtained next-generation flame meets the cutoff condition. Output the next-generation flame that meets the cutoff condition as the second action, and use the fitness of the next-generation moth population as the fitness of the second action.

2. The method according to claim 1, characterized in that, The process of updating the current moth population using the Brownian motion mechanism to obtain the next generation of moth population specifically includes: The scale factor of the Brownian motion mechanism is calculated using a preset activation function; Substituting the scale factor, random number, and contemporary moth population into the Brownian motion mechanism yields the next generation moth population.

3. The method according to claim 1, characterized in that, The step of ranking the next generation of moths according to their fitness to obtain the ranked next generation of moths specifically includes: The crowding distance of each moth is calculated based on the fitness of the next generation of moth population; wherein, the moth population includes several moths, and each moth represents a solution to a business process. The next generation of moths is sorted according to the crowding distance to obtain the sorted next generation of moths.

4. The method according to any one of claims 1-3, characterized in that, The step of updating the parameters of the deep reinforcement learning algorithm based on the current state, the first reward, the fitness of the second action, and the next state specifically includes: The fitness of the first reward and the second action are weighted and summed according to a preset ratio to obtain the second reward; Substituting the second reward, the current state, and the next state into the deep reinforcement learning algorithm yields the first loss function set; The current state is horizontally flipped to obtain a first state. The first state is input into a deep reinforcement learning algorithm to obtain a second state set. A second loss function set is obtained based on the second state set and the first loss function set. The parameters of the deep reinforcement learning algorithm are updated based on the second loss function set and the backpropagation algorithm.

5. A multi-business process collaborative optimization device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1-3.

6. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-3.

7. A collaborative optimization system for multiple business processes, characterized in that, This includes data sensing and acquisition devices and computer equipment connected to the data sensing and acquisition devices, wherein the data sensing and acquisition devices are installed in the business process; wherein, The data sensing and acquisition device is used to collect the first state of several business processes and upload the first state to the computer device. The computer device includes: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1-3.

Images